Sep 16 19:05:21.152838 ip-10-0-139-148 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 19:05:21.599715 ip-10-0-139-148 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:05:21.599715 ip-10-0-139-148 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 19:05:21.599715 ip-10-0-139-148 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:05:21.599715 ip-10-0-139-148 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 19:05:21.599715 ip-10-0-139-148 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:05:21.602453 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.602373    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 19:05:21.607188 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607174    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:05:21.607188 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607188    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607192    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607195    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607200    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607203    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607207    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607210    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607212    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607215    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607217    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607228    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607231    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607235    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607238    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607240    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607243    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607246    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607248    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607251    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:05:21.607250 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607253    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607256    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607259    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607262    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607265    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607268    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607270    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607273    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607276    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607279    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607281    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607284    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607286    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607289    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607291    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607293    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607296    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607298    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607301    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:05:21.607721 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607303    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607305    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607308    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607310    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607313    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607315    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607318    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607320    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607323    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607325    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607329    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607333    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607335    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607338    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607340    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607345    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607348    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607351    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607354    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607357    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:05:21.608218 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607361    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607363    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607366    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607369    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607372    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607374    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607377    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607380    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607383    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607385    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607388    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607391    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607394    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607396    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607399    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607401    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607404    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607406    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607409    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607411    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:05:21.608720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607414    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607416    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607419    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607421    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607424    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607426    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607429    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607432    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607434    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607437    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607439    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607442    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607445    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607447    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607451    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607455    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607457    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607460    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607462    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:05:21.609203 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.607465    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608135    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608142    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608146    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608149    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608152    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608155    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608158    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608161    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608163    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608166    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608169    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608172    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608175    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608178    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608180    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608183    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608185    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608188    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608191    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:05:21.609673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608194    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608197    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608200    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608202    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608205    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608208    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608210    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608213    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608215    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608217    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608221    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608223    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608226    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608228    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608231    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608234    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608236    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608239    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608241    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608244    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:05:21.610156 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608246    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608249    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608251    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608254    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608256    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608259    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608262    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608264    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608267    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608269    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608272    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608274    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608277    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608280    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608282    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608286    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608288    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608290    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608293    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608296    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:05:21.610661 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608298    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608301    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608303    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608306    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608308    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608311    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608313    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608317    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608320    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608323    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608326    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608328    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608330    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608333    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608335    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608338    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608341    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608343    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608346    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:05:21.611242 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608349    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608351    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608354    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608356    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608359    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608362    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608364    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608367    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608369    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608372    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608374    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608376    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608380    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608384    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608386    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608389    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608391    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608394    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608397    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:05:21.611726 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608399    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608402    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608784    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608790    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608794    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608797    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608800    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608804    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608807    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608814    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608817    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608820    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608823    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608826    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608828    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608831    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608833    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608836    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608838    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:05:21.612215 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608841    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608843    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608846    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608848    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608851    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608853    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608856    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608859    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608861    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608864    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608866    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608869    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608871    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608874    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608876    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608878    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608881    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608884    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608886    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:05:21.612687 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608888    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608891    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608893    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608897    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608901    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608903    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608906    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608909    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608911    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608915    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608917    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608921    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608924    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608927    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608930    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608933    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608935    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608938    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608941    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:05:21.613136 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608943    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608945    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608948    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608950    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608952    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608955    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608958    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608961    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608964    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608967    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608969    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608973    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608975    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608977    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608980    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608982    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608986    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608990    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608992    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608995    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:05:21.613634 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.608997    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609000    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609003    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609005    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609008    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609010    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609013    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609015    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609018    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609020    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609023    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609025    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609028    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609030    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609033    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609036    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609038    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609040    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609043    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609046    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:05:21.614109 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609050    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609054    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609058    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609061    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609131    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609163    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609169    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609173    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609179    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609183    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609187    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609190    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609193    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609196    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609200    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609203    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609207    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609210    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609213    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609216    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609219    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609222    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609226    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609228    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 19:05:21.614620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609231    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609234    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609238    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609241    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609244    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609247    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609250    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609253    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609256    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609259    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609262    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609264    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609268    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609271    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609274    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609277    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609280    2583 flags.go:64] FLAG: --enable-server="true"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609284    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609288    2583 flags.go:64] FLAG: --event-burst="100"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609291    2583 flags.go:64] FLAG: --event-qps="50"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609294    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609297    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609299    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609303    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 19:05:21.615223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609306    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609309    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609312    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609314    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609317    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609320    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609323    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609325    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609328    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609331    2583 flags.go:64] FLAG: --feature-gates=""
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609334    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609337    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609340    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609343    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609346    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609349    2583 flags.go:64] FLAG: --help="false"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609351    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-139-148.ec2.internal"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609354    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609357    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609360    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609364    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609367    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609370    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 19:05:21.615798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609372    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609375    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609378    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609381    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609384    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609387    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609390    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609393    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609396    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609404    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609407    2583 flags.go:64] FLAG: --lock-file=""
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609410    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609413    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609415    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609424    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609427    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609430    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609433    2583 flags.go:64] FLAG: --logging-format="text"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609436    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609439    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609442    2583 flags.go:64] FLAG: --manifest-url=""
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609444    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609448    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609451    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609455    2583 flags.go:64] FLAG: --max-pods="110"
Sep 16 19:05:21.616379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609457    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609460    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609463    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609466    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609468    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609471    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609474    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609480    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609484    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609487    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609490    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609493    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609498    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609501    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609504    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609507    2583 flags.go:64] FLAG: --port="10250"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609509    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609513    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c49f2fa89b059985"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609516    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609518    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609521    2583 flags.go:64] FLAG: --register-node="true"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609524    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609527    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609530    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 16 19:05:21.617034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609533    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609536    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609539    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609542    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609545    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609548    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609551    2583 flags.go:64] FLAG: --runonce="false"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609553    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609569    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609572    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609575    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609578    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609581    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609584    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609587    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609589    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609592    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609595    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609598    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609601    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609604    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609607    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609612    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609614    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609617    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 19:05:21.617961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609621    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609624    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609626    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609629    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609632    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609635    2583 flags.go:64] FLAG: --v="2"
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609638    2583 flags.go:64] FLAG: --version="false"
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609642    2583 flags.go:64] FLAG: --vmodule=""
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609645    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.609648    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609750    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609754    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609756    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609759    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609762    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609764    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609767    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609769    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609772    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609774    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609777    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609779    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:05:21.618619 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609781    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609784    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609789    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609792    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609795    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609797    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609800    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609803    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609805    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609808    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609811    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609813    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609816    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609819    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609822    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609824    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609827    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609829    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609832    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:05:21.619191 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609834    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609837    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609839    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609842    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609844    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609848    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609852    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609855    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609858    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609860    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609863    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609866    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609868    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609870    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609873    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609880    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609882    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609885    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609888    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:05:21.619673 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609891    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609893    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609896    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609899    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609903    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609907    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609909    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609913    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609917    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609921    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609925    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609928    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609931    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609933    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609936    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609938    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609941    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609943    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609946    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609948    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:05:21.620133 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609951    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609953    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609956    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609958    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609961    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609964    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609966    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609969    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609973    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609976    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609979    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609981    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609984    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609986    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609989    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609992    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609994    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609997    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.609999    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610002    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610005    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:05:21.620646 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610008    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610010    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610013    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610015    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610018    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610020    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610023    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.610027    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.610706    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.618477    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.618491    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.618510    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.618515    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618593    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:05:21.621173 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618599    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618603    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618606    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618609    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618611    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618614    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618617    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618619    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618622    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618625    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618628    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618631    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618634    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618637    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618640    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618642    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618645    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618647    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618651    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618655    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:05:21.621553 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618658    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618661    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618665    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618668    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618670    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618673    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618676    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618678    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618681    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618684    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618687    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618690    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618692    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618695    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618698    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618700    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618703    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618706    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618708    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618711    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:05:21.622053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618713    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618716    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618718    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618721    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618723    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618726    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618729    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618731    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618734    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618737    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618740    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618742    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618745    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618747    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618751    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618754    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618757    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618760    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618763    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:05:21.622690 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618765    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618769    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618772    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618775    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618777    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618780    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618783    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618785    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618788    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618790    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618793    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618796    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618798    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618801    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618803    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618805    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618808    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618811    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618814    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618817    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:05:21.623165 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618819    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618822    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618824    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618827    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618830    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618834    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618838    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618840    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618843    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618845    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618848    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618851    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618853    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618856    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618858    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618861    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618863    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618866    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:05:21.623672 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618868    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.618873    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618994    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.618998    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619001    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619004    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619007    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619010    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619013    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619016    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619018    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619021    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619024    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619026    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619030    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:05:21.624111 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619032    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619035    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619037    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619039    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619042    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619045    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619047    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619050    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619052    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619055    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619059    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619063    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619066    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619069    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619071    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619074    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619077    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619080    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619082    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619084    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:05:21.624477 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619087    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619089    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619092    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619094    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619097    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619099    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619102    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619105    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619107    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619109    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619112    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619114    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619117    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619120    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619122    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619124    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619127    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619130    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619132    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619135    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:05:21.624984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619137    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619140    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619142    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619145    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619148    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619150    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619152    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619155    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619157    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619160    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619163    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619165    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619167    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619170    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619172    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619176    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619179    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619182    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619184    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:05:21.625456 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619187    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619189    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619192    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619194    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619197    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619199    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619202    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619205    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619208    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619211    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619213    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619216    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619220    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619223    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619226    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619229    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619231    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619234    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619236    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:05:21.625930 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619239    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619241    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619244    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619246    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619249    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619251    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619253    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:21.619256    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.619260    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.619359    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.621397    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.622549    2583 server.go:1019] "Starting client certificate rotation"
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.622655    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 19:05:21.626379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.622692    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 19:05:21.626773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.625727    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 19:05:21.626821 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.626805    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 19:05:21.636513 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.636496    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 16 19:05:21.639687 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.639673    2583 log.go:25] "Validated CRI v1 image API"
Sep 16 19:05:21.640534 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.640520    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 19:05:21.643801 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.643783    2583 fs.go:135] Filesystem UUIDs: map[1aa7d114-448e-4818-9c0a-aa2d6ebd1ae7:/dev/nvme0n1p3 772a1736-d45d-4990-b126-b9a381505064:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 19:05:21.643845 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.643802    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 19:05:21.647729 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.647625    2583 manager.go:216] Machine: {Timestamp:2026-09-16 19:05:21.647159512 +0000 UTC m=+0.360814848 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095657 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249598a317a924c29cbbcd43bec62d SystemUUID:ec249598-a317-a924-c29c-bbcd43bec62d BootID:49554f6b-7481-4a62-9eab-7dc8bec3f032 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:27:8f:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:27:8f:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:e6:3e:58:ce:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 19:05:21.647729 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.647725    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 19:05:21.647839 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.647789    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 19:05:21.648740 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.648719    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 19:05:21.648857 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.648743    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 19:05:21.649536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.649526    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 19:05:21.649578 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.649538    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 19:05:21.649578 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.649546    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 19:05:21.651005 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.650971    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 19:05:21.652029 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.652001    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 19:05:21.655474 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.655459    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 19:05:21.655602 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.655592    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 19:05:21.655647 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.655626    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 19:05:21.655647 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.655640    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 19:05:21.655699 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.655663    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 19:05:21.655699 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.655674    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 16 19:05:21.655699 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.655683    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 19:05:21.656051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.656030    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 19:05:21.656095 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.656055    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 19:05:21.658395 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.658379    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 19:05:21.658773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.658756    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 19:05:21.661090 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.661076    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 19:05:21.661143 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.661106    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 19:05:21.662746 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662734    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 19:05:21.662789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662750    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 19:05:21.662789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662756    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 19:05:21.662789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662762    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 19:05:21.662789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662770    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 19:05:21.662789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662775    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 19:05:21.662789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662788    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 19:05:21.662942 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662793    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 19:05:21.662942 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662799    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 19:05:21.662942 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662804    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 19:05:21.662942 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662812    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 19:05:21.662942 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.662817    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 19:05:21.663604 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.663594    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 19:05:21.663604 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.663604    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 19:05:21.666277 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.666255    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 19:05:21.666333 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.666316    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 19:05:21.667267 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.667256    2583 server.go:1268] "Started kubelet"
Sep 16 19:05:21.667375 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.667333    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 19:05:21.667457 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.667387    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 19:05:21.668044 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.667998    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 19:05:21.668135 ip-10-0-139-148 systemd[1]: Started Kubernetes Kubelet.
Sep 16 19:05:21.668734 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.668607    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 19:05:21.670155 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.670139    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 19:05:21.672103 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.672083    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 19:05:21.674300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.674283    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 19:05:21.675001 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.674983    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 19:05:21.675106 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.675078    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:21.675593 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.675577    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 19:05:21.675593 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.675578    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 19:05:21.675708 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.675602    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 19:05:21.675708 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.675666    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 19:05:21.675708 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.675675    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 19:05:21.677292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.677276    2583 factory.go:153] Registering CRI-O factory
Sep 16 19:05:21.677292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.677294    2583 factory.go:223] Registration of the crio container factory successfully
Sep 16 19:05:21.677428 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.677341    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 19:05:21.677428 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.677349    2583 factory.go:55] Registering systemd factory
Sep 16 19:05:21.677428 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.677356    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 16 19:05:21.677428 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.677376    2583 factory.go:103] Registering Raw factory
Sep 16 19:05:21.677428 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.677389    2583 manager.go:1196] Started watching for new ooms in manager
Sep 16 19:05:21.677764 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.677742    2583 manager.go:349] Starting recovery of all containers
Sep 16 19:05:21.680861 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.680826    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 19:05:21.689011 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.687836    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-148.ec2.internal.18d5e28ce2284698  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-148.ec2.internal,UID:ip-10-0-139-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-148.ec2.internal,},FirstTimestamp:2026-09-16 19:05:21.667229336 +0000 UTC m=+0.380884672,LastTimestamp:2026-09-16 19:05:21.667229336 +0000 UTC m=+0.380884672,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-148.ec2.internal,}"
Sep 16 19:05:21.690013 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.690000    2583 manager.go:354] Recovery completed
Sep 16 19:05:21.691614 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.691554    2583 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 19:05:21.694478 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.694465    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:05:21.696696 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.696676    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:05:21.696762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.696709    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:05:21.696762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.696719    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:05:21.697235 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.697223    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 19:05:21.697235 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.697230    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:05:21.697318 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.697233    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 19:05:21.697318 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.697269    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 19:05:21.697922 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.697908    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:05:21.697976 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.697934    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:05:21.697976 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.697945    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:05:21.698535 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.698523    2583 policy_none.go:49] "None policy: Start"
Sep 16 19:05:21.698585 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.698538    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 19:05:21.698585 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.698548    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 19:05:21.699908 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.699894    2583 policy_none.go:47] "Start"
Sep 16 19:05:21.702678 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.702641    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-148.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 19:05:21.702740 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.702642    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-148.ec2.internal.18d5e28ce3e9e7a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-148.ec2.internal,UID:ip-10-0-139-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-148.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-148.ec2.internal,},FirstTimestamp:2026-09-16 19:05:21.696696226 +0000 UTC m=+0.410351562,LastTimestamp:2026-09-16 19:05:21.696696226 +0000 UTC m=+0.410351562,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-148.ec2.internal,}"
Sep 16 19:05:21.702970 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.702948    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 19:05:21.707168 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.707115    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-148.ec2.internal.18d5e28ce3ea2b07  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-148.ec2.internal,UID:ip-10-0-139-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-148.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-148.ec2.internal,},FirstTimestamp:2026-09-16 19:05:21.696713479 +0000 UTC m=+0.410368815,LastTimestamp:2026-09-16 19:05:21.696713479 +0000 UTC m=+0.410368815,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-148.ec2.internal,}"
Sep 16 19:05:21.711965 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.711892    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-148.ec2.internal.18d5e28ce3ea4d96  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-148.ec2.internal,UID:ip-10-0-139-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-148.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-148.ec2.internal,},FirstTimestamp:2026-09-16 19:05:21.696722326 +0000 UTC m=+0.410377662,LastTimestamp:2026-09-16 19:05:21.696722326 +0000 UTC m=+0.410377662,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-148.ec2.internal,}"
Sep 16 19:05:21.716753 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.716724    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q9kpz"
Sep 16 19:05:21.726420 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.726357    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-148.ec2.internal.18d5e28ce3e9e7a2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-148.ec2.internal.18d5e28ce3e9e7a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-148.ec2.internal,UID:ip-10-0-139-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-148.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-148.ec2.internal,},FirstTimestamp:2026-09-16 19:05:21.696696226 +0000 UTC m=+0.410351562,LastTimestamp:2026-09-16 19:05:21.697920887 +0000 UTC m=+0.411576223,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-148.ec2.internal,}"
Sep 16 19:05:21.730697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.730681    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q9kpz"
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.731157    2583 manager.go:337] "Starting Device Plugin manager"
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.731192    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.731200    2583 server.go:87] "Starting device plugin registration server"
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.731473    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.731484    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.731602    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.731714    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.731723    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.732160    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 19:05:21.749705 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.732216    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:21.816053 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.816031    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 19:05:21.817347 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.817324    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 19:05:21.817415 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.817379    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 19:05:21.817415 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.817406    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 19:05:21.817479 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.817449    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 19:05:21.819778 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.819762    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:05:21.832583 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.832523    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:05:21.833494 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.833478    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:05:21.833590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.833511    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:05:21.833590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.833527    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:05:21.833590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.833550    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-148.ec2.internal"
Sep 16 19:05:21.839938 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.839925    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-148.ec2.internal"
Sep 16 19:05:21.839996 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.839943    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-148.ec2.internal\": node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:21.854926 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.854909    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:21.918056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.918038    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal"]
Sep 16 19:05:21.918108 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.918092    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:05:21.919455 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.919440    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:05:21.919533 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.919467    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:05:21.919533 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.919481    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:05:21.920935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.920923    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:05:21.921043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.921028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:21.921082 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.921058    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:05:21.921661 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.921645    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:05:21.921721 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.921658    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:05:21.921721 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.921676    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:05:21.921721 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.921680    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:05:21.921721 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.921689    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:05:21.921831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.921690    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:05:21.922963 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.922947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:21.923017 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.922981    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:05:21.927405 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.927388    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:05:21.927489 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.927422    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:05:21.927489 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.927438    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:05:21.939078 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.939061    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-148.ec2.internal\" not found" node="ip-10-0-139-148.ec2.internal"
Sep 16 19:05:21.944626 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.944612    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-148.ec2.internal\" not found" node="ip-10-0-139-148.ec2.internal"
Sep 16 19:05:21.955689 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:21.955671    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:21.977600 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.977573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbd3a56e8f3c433944bfaa6b0cafecc6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal\" (UID: \"fbd3a56e8f3c433944bfaa6b0cafecc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:21.977600 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.977598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbd3a56e8f3c433944bfaa6b0cafecc6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal\" (UID: \"fbd3a56e8f3c433944bfaa6b0cafecc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:21.977714 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:21.977615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7033f5edf1da39728ed37f3d87a19a2b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-148.ec2.internal\" (UID: \"7033f5edf1da39728ed37f3d87a19a2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.055940 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.055922    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:22.078300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.078276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbd3a56e8f3c433944bfaa6b0cafecc6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal\" (UID: \"fbd3a56e8f3c433944bfaa6b0cafecc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.078380 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.078304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7033f5edf1da39728ed37f3d87a19a2b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-148.ec2.internal\" (UID: \"7033f5edf1da39728ed37f3d87a19a2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.078380 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.078320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbd3a56e8f3c433944bfaa6b0cafecc6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal\" (UID: \"fbd3a56e8f3c433944bfaa6b0cafecc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.078380 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.078359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbd3a56e8f3c433944bfaa6b0cafecc6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal\" (UID: \"fbd3a56e8f3c433944bfaa6b0cafecc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.078380 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.078368    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbd3a56e8f3c433944bfaa6b0cafecc6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal\" (UID: \"fbd3a56e8f3c433944bfaa6b0cafecc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.078543 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.078393    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7033f5edf1da39728ed37f3d87a19a2b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-148.ec2.internal\" (UID: \"7033f5edf1da39728ed37f3d87a19a2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.156585 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.156523    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:22.241980 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.241960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.247699 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.247682    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.257615 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.257593    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:22.358232 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.358202    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:22.458815 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.458752    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-148.ec2.internal\" not found"
Sep 16 19:05:22.534218 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.534197    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:05:22.575296 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.575272    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.590214 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.590194    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 19:05:22.595389 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.595257    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal"
Sep 16 19:05:22.606684 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.606661    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 19:05:22.622687 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.622660    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 19:05:22.622805 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.622782    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 19:05:22.622857 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.622819    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 19:05:22.655967 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.655949    2583 apiserver.go:52] "Watching apiserver"
Sep 16 19:05:22.671316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.671295    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 19:05:22.671795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.671769    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-2jggs","openshift-ovn-kubernetes/ovnkube-node-gsb78","kube-system/global-pull-secret-syncer-698qk","kube-system/konnectivity-agent-xtjc2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v","openshift-cluster-node-tuning-operator/tuned-rm5vh","openshift-image-registry/node-ca-9ht7r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal","openshift-multus/multus-pcbx2","kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal","openshift-dns/node-resolver-76twg","openshift-multus/multus-additional-cni-plugins-v9d2x","openshift-multus/network-metrics-daemon-wvcsr","openshift-network-diagnostics/network-check-target-8jmnn"]
Sep 16 19:05:22.673596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.673581    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.674466 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.674447    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 19:05:22.674969 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.674956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:22.675044 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.675026    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:22.676624 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.676607    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:22.676709 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.676660    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:22.677939 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.677925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:22.679107 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.679091    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 19:05:22.679217 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.679183    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 19:05:22.679279 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.679186    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 19:05:22.679347 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.679190    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n7kf9\""
Sep 16 19:05:22.679636 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.679619    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.680741 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.680714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktx2\" (UniqueName: \"kubernetes.io/projected/944c14c7-3c60-43cc-9e04-a0903a555407-kube-api-access-tktx2\") pod \"node-ca-9ht7r\" (UID: \"944c14c7-3c60-43cc-9e04-a0903a555407\") " pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.680867 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.680752    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:22.680867 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.680778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvlx\" (UniqueName: \"kubernetes.io/projected/14abbfab-5c2d-4416-a29a-939762f17972-kube-api-access-ksvlx\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:22.680867 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.680794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3af2c548-b0f3-4a74-87f7-eb574626b50d-kubelet-config\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:22.680867 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.680816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944c14c7-3c60-43cc-9e04-a0903a555407-serviceca\") pod \"node-ca-9ht7r\" (UID: \"944c14c7-3c60-43cc-9e04-a0903a555407\") " pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.680867 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.680836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3af2c548-b0f3-4a74-87f7-eb574626b50d-dbus\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:22.680867 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.680857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:22.681162 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.680901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944c14c7-3c60-43cc-9e04-a0903a555407-host\") pod \"node-ca-9ht7r\" (UID: \"944c14c7-3c60-43cc-9e04-a0903a555407\") " pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.681162 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.681027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.681265 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.681189    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 19:05:22.682364 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.682345    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 19:05:22.682585 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.682545    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.682686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.682653    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fxsz9\""
Sep 16 19:05:22.682787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.682772    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 19:05:22.683130 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.683109    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 19:05:22.683254 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.683136    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 19:05:22.683254 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.683175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r7twt\""
Sep 16 19:05:22.683693 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.683645    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:05:22.683937 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.683923    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 19:05:22.684191 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.684172    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9t45s\""
Sep 16 19:05:22.684333 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.684273    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.686057 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.686030    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.687817 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.687799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:22.687898 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.687878    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:22.689267 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.689251    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 19:05:22.689367 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.689276    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 19:05:22.689418 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.689392    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 19:05:22.689471 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.689417    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 19:05:22.689789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.689542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.689789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.689680    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:05:22.689789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.689709    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 19:05:22.689789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.689753    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 19:05:22.690787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.690311    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 19:05:22.690787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.690324    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 19:05:22.690787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.690397    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wmx5s\""
Sep 16 19:05:22.690787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.690659    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jjnnw\""
Sep 16 19:05:22.691016 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.690837    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 19:05:22.691016 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.690894    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 19:05:22.691286 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.691266    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vwlzg\""
Sep 16 19:05:22.691286 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.691281    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 19:05:22.691677 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.691657    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 19:05:22.693492 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.693469    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.696710 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.696693    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6h2vz\""
Sep 16 19:05:22.696901 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.696888    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 19:05:22.707146 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.707120    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 19:05:22.707404 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.707384    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-96mf5\""
Sep 16 19:05:22.707485 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.707429    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 19:05:22.707485 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.707446    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 19:05:22.707590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.707434    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 19:05:22.728914 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.728898    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lk766"
Sep 16 19:05:22.733757 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.733721    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 19:00:23 +0000 UTC" deadline="2028-04-21 15:48:38.461301274 +0000 UTC"
Sep 16 19:05:22.733757 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.733756    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13988h43m15.727547793s"
Sep 16 19:05:22.738726 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.738710    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lk766"
Sep 16 19:05:22.776957 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.776943    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 19:05:22.781552 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-kubelet\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.781646 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-node-log\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.781646 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/340d58a9-8b5d-416e-8bd3-88d32ba5747f-hosts-file\") pod \"node-resolver-76twg\" (UID: \"340d58a9-8b5d-416e-8bd3-88d32ba5747f\") " pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.781646 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktx2\" (UniqueName: \"kubernetes.io/projected/944c14c7-3c60-43cc-9e04-a0903a555407-kube-api-access-tktx2\") pod \"node-ca-9ht7r\" (UID: \"944c14c7-3c60-43cc-9e04-a0903a555407\") " pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.781811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-cnibin\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.781811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.781811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-host\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.781811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944c14c7-3c60-43cc-9e04-a0903a555407-serviceca\") pod \"node-ca-9ht7r\" (UID: \"944c14c7-3c60-43cc-9e04-a0903a555407\") " pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.781811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48311335-3e25-4b9c-a365-1ac69298d999-host-slash\") pod \"iptables-alerter-2jggs\" (UID: \"48311335-3e25-4b9c-a365-1ac69298d999\") " pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.781811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-var-lib-openvswitch\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05d5b31c-3de4-44d3-a850-547bf19e0c07-ovnkube-script-lib\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a025317-d7c0-4050-8f6b-d6cf69387c55-konnectivity-ca\") pod \"konnectivity-agent-xtjc2\" (UID: \"6a025317-d7c0-4050-8f6b-d6cf69387c55\") " pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-run-k8s-cni-cncf-io\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-var-lib-cni-bin\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05d5b31c-3de4-44d3-a850-547bf19e0c07-ovnkube-config\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.781998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-system-cni-dir\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782032    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthll\" (UniqueName: \"kubernetes.io/projected/781e546d-fa86-4707-930b-089f49571e79-kube-api-access-pthll\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-run\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.782092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-sys\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-var-lib-kubelet\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-var-lib-cni-multus\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-run-ovn\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-registration-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-sysctl-conf\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944c14c7-3c60-43cc-9e04-a0903a555407-serviceca\") pod \"node-ca-9ht7r\" (UID: \"944c14c7-3c60-43cc-9e04-a0903a555407\") " pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-etc-kubernetes\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-run-systemd\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/601823a6-99aa-4683-b1db-06c7e34c3d86-cni-binary-copy\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340d58a9-8b5d-416e-8bd3-88d32ba5747f-tmp-dir\") pod \"node-resolver-76twg\" (UID: \"340d58a9-8b5d-416e-8bd3-88d32ba5747f\") " pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944c14c7-3c60-43cc-9e04-a0903a555407-host\") pod \"node-ca-9ht7r\" (UID: \"944c14c7-3c60-43cc-9e04-a0903a555407\") " pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvlx\" (UniqueName: \"kubernetes.io/projected/14abbfab-5c2d-4416-a29a-939762f17972-kube-api-access-ksvlx\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3af2c548-b0f3-4a74-87f7-eb574626b50d-kubelet-config\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-system-cni-dir\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-cnibin\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944c14c7-3c60-43cc-9e04-a0903a555407-host\") pod \"node-ca-9ht7r\" (UID: \"944c14c7-3c60-43cc-9e04-a0903a555407\") " pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.782596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3af2c548-b0f3-4a74-87f7-eb574626b50d-kubelet-config\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-run-netns\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-daemon-config\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnxj\" (UniqueName: \"kubernetes.io/projected/601823a6-99aa-4683-b1db-06c7e34c3d86-kube-api-access-cpnxj\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782525    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-conf-dir\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-slash\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-cni-bin\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05d5b31c-3de4-44d3-a850-547bf19e0c07-ovn-node-metrics-cert\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgwzc\" (UniqueName: \"kubernetes.io/projected/340d58a9-8b5d-416e-8bd3-88d32ba5747f-kube-api-access-dgwzc\") pod \"node-resolver-76twg\" (UID: \"340d58a9-8b5d-416e-8bd3-88d32ba5747f\") " pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48311335-3e25-4b9c-a365-1ac69298d999-iptables-alerter-script\") pod \"iptables-alerter-2jggs\" (UID: \"48311335-3e25-4b9c-a365-1ac69298d999\") " pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782748    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-sysconfig\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84jk\" (UniqueName: \"kubernetes.io/projected/6a5f5413-1eac-4ab3-8f93-99c030c6913c-kube-api-access-m84jk\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-run-openvswitch\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3af2c548-b0f3-4a74-87f7-eb574626b50d-dbus\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:22.783310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-systemd\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3af2c548-b0f3-4a74-87f7-eb574626b50d-dbus\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-tuned\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-etc-openvswitch\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.782996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8rz\" (UniqueName: \"kubernetes.io/projected/05d5b31c-3de4-44d3-a850-547bf19e0c07-kube-api-access-sf8rz\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a025317-d7c0-4050-8f6b-d6cf69387c55-agent-certs\") pod \"konnectivity-agent-xtjc2\" (UID: \"6a025317-d7c0-4050-8f6b-d6cf69387c55\") " pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-hostroot\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-sysctl-d\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-cni-netd\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05d5b31c-3de4-44d3-a850-547bf19e0c07-env-overrides\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-etc-selinux\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-sys-fs\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-kubernetes\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.784019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-var-lib-kubelet\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783403    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-device-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-modprobe-d\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5f5413-1eac-4ab3-8f93-99c030c6913c-tmp\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.783520    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-systemd-units\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-run-netns\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-log-socket\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.783625    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret podName:3af2c548-b0f3-4a74-87f7-eb574626b50d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:23.283604457 +0000 UTC m=+1.997259800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret") pod "global-pull-secret-syncer-698qk" (UID: "3af2c548-b0f3-4a74-87f7-eb574626b50d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j29f9\" (UniqueName: \"kubernetes.io/projected/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-kube-api-access-j29f9\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-socket-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-lib-modules\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-run-multus-certs\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5pn5\" (UniqueName: \"kubernetes.io/projected/48311335-3e25-4b9c-a365-1ac69298d999-kube-api-access-l5pn5\") pod \"iptables-alerter-2jggs\" (UID: \"48311335-3e25-4b9c-a365-1ac69298d999\") " pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.784676 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.783835    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:22.785216 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-cni-dir\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.785216 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.783887    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-socket-dir-parent\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.785216 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.783967    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs podName:14abbfab-5c2d-4416-a29a-939762f17972 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:23.283949501 +0000 UTC m=+1.997604830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs") pod "network-metrics-daemon-wvcsr" (UID: "14abbfab-5c2d-4416-a29a-939762f17972") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:22.798385 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.798365    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 19:05:22.798488 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:22.798415    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7033f5edf1da39728ed37f3d87a19a2b.slice/crio-11d23c071d012fb7ecbc114a4b2bef9eb60d445421d8e1ea9fdf07cbbadf309c WatchSource:0}: Error finding container 11d23c071d012fb7ecbc114a4b2bef9eb60d445421d8e1ea9fdf07cbbadf309c: Status 404 returned error can't find the container with id 11d23c071d012fb7ecbc114a4b2bef9eb60d445421d8e1ea9fdf07cbbadf309c
Sep 16 19:05:22.798677 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:22.798663    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd3a56e8f3c433944bfaa6b0cafecc6.slice/crio-c69c4bd8fb771a37fb28768e3c516a4fa329290cee1e674e1cf78d6a135c1da1 WatchSource:0}: Error finding container c69c4bd8fb771a37fb28768e3c516a4fa329290cee1e674e1cf78d6a135c1da1: Status 404 returned error can't find the container with id c69c4bd8fb771a37fb28768e3c516a4fa329290cee1e674e1cf78d6a135c1da1
Sep 16 19:05:22.801865 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.801846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvlx\" (UniqueName: \"kubernetes.io/projected/14abbfab-5c2d-4416-a29a-939762f17972-kube-api-access-ksvlx\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:22.801955 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.801883    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktx2\" (UniqueName: \"kubernetes.io/projected/944c14c7-3c60-43cc-9e04-a0903a555407-kube-api-access-tktx2\") pod \"node-ca-9ht7r\" (UID: \"944c14c7-3c60-43cc-9e04-a0903a555407\") " pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:22.803167 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.803150    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:05:22.820481 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.820436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal" event={"ID":"7033f5edf1da39728ed37f3d87a19a2b","Type":"ContainerStarted","Data":"11d23c071d012fb7ecbc114a4b2bef9eb60d445421d8e1ea9fdf07cbbadf309c"}
Sep 16 19:05:22.821466 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.821446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal" event={"ID":"fbd3a56e8f3c433944bfaa6b0cafecc6","Type":"ContainerStarted","Data":"c69c4bd8fb771a37fb28768e3c516a4fa329290cee1e674e1cf78d6a135c1da1"}
Sep 16 19:05:22.884251 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-systemd-units\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884251 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-run-netns\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-log-socket\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j29f9\" (UniqueName: \"kubernetes.io/projected/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-kube-api-access-j29f9\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.884383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-socket-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.884383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-lib-modules\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.884383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-run-multus-certs\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.884383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-run-netns\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-systemd-units\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-log-socket\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884394    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-run-multus-certs\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5pn5\" (UniqueName: \"kubernetes.io/projected/48311335-3e25-4b9c-a365-1ac69298d999-kube-api-access-l5pn5\") pod \"iptables-alerter-2jggs\" (UID: \"48311335-3e25-4b9c-a365-1ac69298d999\") " pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-socket-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-cni-dir\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884474    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-socket-dir-parent\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-kubelet\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-node-log\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/340d58a9-8b5d-416e-8bd3-88d32ba5747f-hosts-file\") pod \"node-resolver-76twg\" (UID: \"340d58a9-8b5d-416e-8bd3-88d32ba5747f\") " pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-cnibin\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-kubelet\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-lib-modules\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884585    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-socket-dir-parent\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884606    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-node-log\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-cni-dir\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-host\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48311335-3e25-4b9c-a365-1ac69298d999-host-slash\") pod \"iptables-alerter-2jggs\" (UID: \"48311335-3e25-4b9c-a365-1ac69298d999\") " pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.884736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884664    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-cnibin\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-var-lib-openvswitch\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05d5b31c-3de4-44d3-a850-547bf19e0c07-ovnkube-script-lib\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/340d58a9-8b5d-416e-8bd3-88d32ba5747f-hosts-file\") pod \"node-resolver-76twg\" (UID: \"340d58a9-8b5d-416e-8bd3-88d32ba5747f\") " pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48311335-3e25-4b9c-a365-1ac69298d999-host-slash\") pod \"iptables-alerter-2jggs\" (UID: \"48311335-3e25-4b9c-a365-1ac69298d999\") " pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884733    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-host\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a025317-d7c0-4050-8f6b-d6cf69387c55-konnectivity-ca\") pod \"konnectivity-agent-xtjc2\" (UID: \"6a025317-d7c0-4050-8f6b-d6cf69387c55\") " pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-var-lib-openvswitch\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-run-k8s-cni-cncf-io\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-var-lib-cni-bin\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05d5b31c-3de4-44d3-a850-547bf19e0c07-ovnkube-config\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-system-cni-dir\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-run-k8s-cni-cncf-io\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884928    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-var-lib-cni-bin\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthll\" (UniqueName: \"kubernetes.io/projected/781e546d-fa86-4707-930b-089f49571e79-kube-api-access-pthll\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-run\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.885504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-sys\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-var-lib-kubelet\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.884970    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-system-cni-dir\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-run\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-var-lib-cni-multus\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-var-lib-cni-multus\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-sys\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-run-ovn\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-var-lib-kubelet\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-registration-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-sysctl-conf\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885165    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-run-ovn\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-etc-kubernetes\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-run-systemd\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/601823a6-99aa-4683-b1db-06c7e34c3d86-cni-binary-copy\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885242    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05d5b31c-3de4-44d3-a850-547bf19e0c07-ovnkube-script-lib\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340d58a9-8b5d-416e-8bd3-88d32ba5747f-tmp-dir\") pod \"node-resolver-76twg\" (UID: \"340d58a9-8b5d-416e-8bd3-88d32ba5747f\") " pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.886314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-system-cni-dir\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-registration-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-cnibin\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885324    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a025317-d7c0-4050-8f6b-d6cf69387c55-konnectivity-ca\") pod \"konnectivity-agent-xtjc2\" (UID: \"6a025317-d7c0-4050-8f6b-d6cf69387c55\") " pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-run-netns\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-sysctl-conf\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-daemon-config\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-run-netns\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnxj\" (UniqueName: \"kubernetes.io/projected/601823a6-99aa-4683-b1db-06c7e34c3d86-kube-api-access-cpnxj\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-etc-kubernetes\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-conf-dir\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-slash\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-cni-bin\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-system-cni-dir\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885523    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-slash\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05d5b31c-3de4-44d3-a850-547bf19e0c07-ovn-node-metrics-cert\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgwzc\" (UniqueName: \"kubernetes.io/projected/340d58a9-8b5d-416e-8bd3-88d32ba5747f-kube-api-access-dgwzc\") pod \"node-resolver-76twg\" (UID: \"340d58a9-8b5d-416e-8bd3-88d32ba5747f\") " pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885585    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-conf-dir\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48311335-3e25-4b9c-a365-1ac69298d999-iptables-alerter-script\") pod \"iptables-alerter-2jggs\" (UID: \"48311335-3e25-4b9c-a365-1ac69298d999\") " pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05d5b31c-3de4-44d3-a850-547bf19e0c07-ovnkube-config\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-sysconfig\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-cni-bin\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885666    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-cnibin\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84jk\" (UniqueName: \"kubernetes.io/projected/6a5f5413-1eac-4ab3-8f93-99c030c6913c-kube-api-access-m84jk\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885712    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-sysconfig\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-run-openvswitch\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-systemd\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-tuned\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-etc-openvswitch\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8rz\" (UniqueName: \"kubernetes.io/projected/05d5b31c-3de4-44d3-a850-547bf19e0c07-kube-api-access-sf8rz\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.887724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a025317-d7c0-4050-8f6b-d6cf69387c55-agent-certs\") pod \"konnectivity-agent-xtjc2\" (UID: \"6a025317-d7c0-4050-8f6b-d6cf69387c55\") " pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-systemd\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-hostroot\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885961    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-etc-openvswitch\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340d58a9-8b5d-416e-8bd3-88d32ba5747f-tmp-dir\") pod \"node-resolver-76twg\" (UID: \"340d58a9-8b5d-416e-8bd3-88d32ba5747f\") " pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.885971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-run-openvswitch\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886006    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-hostroot\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/601823a6-99aa-4683-b1db-06c7e34c3d86-cni-binary-copy\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-sysctl-d\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/601823a6-99aa-4683-b1db-06c7e34c3d86-multus-daemon-config\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-cni-netd\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-run-systemd\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05d5b31c-3de4-44d3-a850-547bf19e0c07-env-overrides\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-etc-selinux\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-sys-fs\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.888201 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-sysctl-d\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-kubernetes\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-var-lib-kubelet\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-device-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-modprobe-d\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886435    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5f5413-1eac-4ab3-8f93-99c030c6913c-tmp\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-etc-selinux\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-modprobe-d\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-kubernetes\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48311335-3e25-4b9c-a365-1ac69298d999-iptables-alerter-script\") pod \"iptables-alerter-2jggs\" (UID: \"48311335-3e25-4b9c-a365-1ac69298d999\") " pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886678    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/601823a6-99aa-4683-b1db-06c7e34c3d86-host-var-lib-kubelet\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-device-dir\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05d5b31c-3de4-44d3-a850-547bf19e0c07-host-cni-netd\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.886852    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/781e546d-fa86-4707-930b-089f49571e79-sys-fs\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.888750 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.887174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.889268 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.887178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05d5b31c-3de4-44d3-a850-547bf19e0c07-env-overrides\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.889268 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.888780    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a5f5413-1eac-4ab3-8f93-99c030c6913c-etc-tuned\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.889268 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.888817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a025317-d7c0-4050-8f6b-d6cf69387c55-agent-certs\") pod \"konnectivity-agent-xtjc2\" (UID: \"6a025317-d7c0-4050-8f6b-d6cf69387c55\") " pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:22.889268 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.888831    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05d5b31c-3de4-44d3-a850-547bf19e0c07-ovn-node-metrics-cert\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.889268 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.888905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5f5413-1eac-4ab3-8f93-99c030c6913c-tmp\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.903757 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.903734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8rz\" (UniqueName: \"kubernetes.io/projected/05d5b31c-3de4-44d3-a850-547bf19e0c07-kube-api-access-sf8rz\") pod \"ovnkube-node-gsb78\" (UID: \"05d5b31c-3de4-44d3-a850-547bf19e0c07\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:22.905737 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.905718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j29f9\" (UniqueName: \"kubernetes.io/projected/b2fd7290-33ee-4c35-a7f7-62f0704a5de1-kube-api-access-j29f9\") pod \"multus-additional-cni-plugins-v9d2x\" (UID: \"b2fd7290-33ee-4c35-a7f7-62f0704a5de1\") " pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:22.906426 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.906399    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:05:22.906465 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.906437    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:05:22.906465 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.906449    2583 projected.go:196] Error preparing data for projected volume kube-api-access-n7rpv for pod openshift-network-diagnostics/network-check-target-8jmnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:22.906540 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:22.906529    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv podName:48c3e63a-7b54-4817-97de-38704d661bea nodeName:}" failed. No retries permitted until 2026-09-16 19:05:23.406509781 +0000 UTC m=+2.120165127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n7rpv" (UniqueName: "kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv") pod "network-check-target-8jmnn" (UID: "48c3e63a-7b54-4817-97de-38704d661bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:22.907061 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.907043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgwzc\" (UniqueName: \"kubernetes.io/projected/340d58a9-8b5d-416e-8bd3-88d32ba5747f-kube-api-access-dgwzc\") pod \"node-resolver-76twg\" (UID: \"340d58a9-8b5d-416e-8bd3-88d32ba5747f\") " pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:22.908484 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.908429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84jk\" (UniqueName: \"kubernetes.io/projected/6a5f5413-1eac-4ab3-8f93-99c030c6913c-kube-api-access-m84jk\") pod \"tuned-rm5vh\" (UID: \"6a5f5413-1eac-4ab3-8f93-99c030c6913c\") " pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:22.909681 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.909660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5pn5\" (UniqueName: \"kubernetes.io/projected/48311335-3e25-4b9c-a365-1ac69298d999-kube-api-access-l5pn5\") pod \"iptables-alerter-2jggs\" (UID: \"48311335-3e25-4b9c-a365-1ac69298d999\") " pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:22.910129 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.910113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthll\" (UniqueName: \"kubernetes.io/projected/781e546d-fa86-4707-930b-089f49571e79-kube-api-access-pthll\") pod \"aws-ebs-csi-driver-node-hb66v\" (UID: \"781e546d-fa86-4707-930b-089f49571e79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:22.911321 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:22.911306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnxj\" (UniqueName: \"kubernetes.io/projected/601823a6-99aa-4683-b1db-06c7e34c3d86-kube-api-access-cpnxj\") pod \"multus-pcbx2\" (UID: \"601823a6-99aa-4683-b1db-06c7e34c3d86\") " pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:24.588816 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.588767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ht7r"
Sep 16 19:05:24.594467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.594450    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:24.597425 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:24.597386    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944c14c7_3c60_43cc_9e04_a0903a555407.slice/crio-f27207ed4c0f6f9c51e2837fa3ca54e8332c2f93fa120ae6934f95c064272acc WatchSource:0}: Error finding container f27207ed4c0f6f9c51e2837fa3ca54e8332c2f93fa120ae6934f95c064272acc: Status 404 returned error can't find the container with id f27207ed4c0f6f9c51e2837fa3ca54e8332c2f93fa120ae6934f95c064272acc
Sep 16 19:05:24.602309 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:24.602291    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a025317_d7c0_4050_8f6b_d6cf69387c55.slice/crio-7341c180ff091e8f37b42772d64619c93cbe68e4015b032683279f3ca9445ebb WatchSource:0}: Error finding container 7341c180ff091e8f37b42772d64619c93cbe68e4015b032683279f3ca9445ebb: Status 404 returned error can't find the container with id 7341c180ff091e8f37b42772d64619c93cbe68e4015b032683279f3ca9445ebb
Sep 16 19:05:24.609526 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.609508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v"
Sep 16 19:05:24.618499 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:24.618475    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781e546d_fa86_4707_930b_089f49571e79.slice/crio-ac3a34397175cf422758327f98483f00ab79ea0045356aa8d851f7d4a4b7397a WatchSource:0}: Error finding container ac3a34397175cf422758327f98483f00ab79ea0045356aa8d851f7d4a4b7397a: Status 404 returned error can't find the container with id ac3a34397175cf422758327f98483f00ab79ea0045356aa8d851f7d4a4b7397a
Sep 16 19:05:24.621106 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.621089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rm5vh"
Sep 16 19:05:24.625532 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.625517    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:05:24.627325 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:24.627308    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5f5413_1eac_4ab3_8f93_99c030c6913c.slice/crio-9cdc5f6cadee2e49bcda89759f656bc3fa197291806bea4a67e446f2d32cdff3 WatchSource:0}: Error finding container 9cdc5f6cadee2e49bcda89759f656bc3fa197291806bea4a67e446f2d32cdff3: Status 404 returned error can't find the container with id 9cdc5f6cadee2e49bcda89759f656bc3fa197291806bea4a67e446f2d32cdff3
Sep 16 19:05:24.635318 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.635302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pcbx2"
Sep 16 19:05:24.641506 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:24.641488    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601823a6_99aa_4683_b1db_06c7e34c3d86.slice/crio-da1d371e8feedc6e2ad3ec774d804d3047e2ac4795ad039e5a5ba7513180dead WatchSource:0}: Error finding container da1d371e8feedc6e2ad3ec774d804d3047e2ac4795ad039e5a5ba7513180dead: Status 404 returned error can't find the container with id da1d371e8feedc6e2ad3ec774d804d3047e2ac4795ad039e5a5ba7513180dead
Sep 16 19:05:24.653082 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.653068    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2jggs"
Sep 16 19:05:24.658320 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.658304    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:24.659602 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:24.659581    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48311335_3e25_4b9c_a365_1ac69298d999.slice/crio-f68c7dbe625f8028fab3e9ff9165138853764f345293ed173d721d35c2addd96 WatchSource:0}: Error finding container f68c7dbe625f8028fab3e9ff9165138853764f345293ed173d721d35c2addd96: Status 404 returned error can't find the container with id f68c7dbe625f8028fab3e9ff9165138853764f345293ed173d721d35c2addd96
Sep 16 19:05:24.665507 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:24.665487    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d5b31c_3de4_44d3_a850_547bf19e0c07.slice/crio-91b7757a2f6537d04d5c306ebbf07b19a6cf7dab21d05e0f0e0c552a530a621c WatchSource:0}: Error finding container 91b7757a2f6537d04d5c306ebbf07b19a6cf7dab21d05e0f0e0c552a530a621c: Status 404 returned error can't find the container with id 91b7757a2f6537d04d5c306ebbf07b19a6cf7dab21d05e0f0e0c552a530a621c
Sep 16 19:05:24.690026 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.690009    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:05:24.693266 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.693252    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-76twg"
Sep 16 19:05:24.698362 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.698349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9d2x"
Sep 16 19:05:24.699878 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:24.699858    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340d58a9_8b5d_416e_8bd3_88d32ba5747f.slice/crio-7437e4e9e92e4f020fb4148d112a2db7760692a2090278ac0fc2a502287daba5 WatchSource:0}: Error finding container 7437e4e9e92e4f020fb4148d112a2db7760692a2090278ac0fc2a502287daba5: Status 404 returned error can't find the container with id 7437e4e9e92e4f020fb4148d112a2db7760692a2090278ac0fc2a502287daba5
Sep 16 19:05:24.705645 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:24.705625    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fd7290_33ee_4c35_a7f7_62f0704a5de1.slice/crio-aa4e6c688a1a4e3d5a2deea51a75fbd20d36ba74474524939963b3243fc3c9f0 WatchSource:0}: Error finding container aa4e6c688a1a4e3d5a2deea51a75fbd20d36ba74474524939963b3243fc3c9f0: Status 404 returned error can't find the container with id aa4e6c688a1a4e3d5a2deea51a75fbd20d36ba74474524939963b3243fc3c9f0
Sep 16 19:05:24.876290 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.876220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:24.876290 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:24.876260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:24.876522 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:24.876371    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:24.876522 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:24.876372    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:24.876522 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:24.876441    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs podName:14abbfab-5c2d-4416-a29a-939762f17972 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:25.876405616 +0000 UTC m=+3.002209984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs") pod "network-metrics-daemon-wvcsr" (UID: "14abbfab-5c2d-4416-a29a-939762f17972") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:24.876522 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:24.876458    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret podName:3af2c548-b0f3-4a74-87f7-eb574626b50d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:25.87645001 +0000 UTC m=+3.002254376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret") pod "global-pull-secret-syncer-698qk" (UID: "3af2c548-b0f3-4a74-87f7-eb574626b50d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:25.078579 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.078545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:25.078744 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:25.078718    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:05:25.078744 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:25.078737    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:05:25.078854 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:25.078749    2583 projected.go:196] Error preparing data for projected volume kube-api-access-n7rpv for pod openshift-network-diagnostics/network-check-target-8jmnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:25.078854 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:25.078802    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv podName:48c3e63a-7b54-4817-97de-38704d661bea nodeName:}" failed. No retries permitted until 2026-09-16 19:05:26.078784522 +0000 UTC m=+3.204588901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7rpv" (UniqueName: "kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv") pod "network-check-target-8jmnn" (UID: "48c3e63a-7b54-4817-97de-38704d661bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:25.283214 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.283150    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:05:25.328112 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.328056    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:00:24 +0000 UTC" deadline="2028-05-27 17:16:51.671586336 +0000 UTC"
Sep 16 19:05:25.328112 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.328106    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14854h11m26.343483941s"
Sep 16 19:05:25.427721 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.427685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-76twg" event={"ID":"340d58a9-8b5d-416e-8bd3-88d32ba5747f","Type":"ContainerStarted","Data":"7437e4e9e92e4f020fb4148d112a2db7760692a2090278ac0fc2a502287daba5"}
Sep 16 19:05:25.432495 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.432468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2jggs" event={"ID":"48311335-3e25-4b9c-a365-1ac69298d999","Type":"ContainerStarted","Data":"f68c7dbe625f8028fab3e9ff9165138853764f345293ed173d721d35c2addd96"}
Sep 16 19:05:25.450298 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.450274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pcbx2" event={"ID":"601823a6-99aa-4683-b1db-06c7e34c3d86","Type":"ContainerStarted","Data":"da1d371e8feedc6e2ad3ec774d804d3047e2ac4795ad039e5a5ba7513180dead"}
Sep 16 19:05:25.456614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.456588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rm5vh" event={"ID":"6a5f5413-1eac-4ab3-8f93-99c030c6913c","Type":"ContainerStarted","Data":"9cdc5f6cadee2e49bcda89759f656bc3fa197291806bea4a67e446f2d32cdff3"}
Sep 16 19:05:25.465537 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.465467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xtjc2" event={"ID":"6a025317-d7c0-4050-8f6b-d6cf69387c55","Type":"ContainerStarted","Data":"7341c180ff091e8f37b42772d64619c93cbe68e4015b032683279f3ca9445ebb"}
Sep 16 19:05:25.474603 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.474577    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" event={"ID":"b2fd7290-33ee-4c35-a7f7-62f0704a5de1","Type":"ContainerStarted","Data":"aa4e6c688a1a4e3d5a2deea51a75fbd20d36ba74474524939963b3243fc3c9f0"}
Sep 16 19:05:25.477394 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.477372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerStarted","Data":"91b7757a2f6537d04d5c306ebbf07b19a6cf7dab21d05e0f0e0c552a530a621c"}
Sep 16 19:05:25.480071 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.480044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v" event={"ID":"781e546d-fa86-4707-930b-089f49571e79","Type":"ContainerStarted","Data":"ac3a34397175cf422758327f98483f00ab79ea0045356aa8d851f7d4a4b7397a"}
Sep 16 19:05:25.489244 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.488975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ht7r" event={"ID":"944c14c7-3c60-43cc-9e04-a0903a555407","Type":"ContainerStarted","Data":"f27207ed4c0f6f9c51e2837fa3ca54e8332c2f93fa120ae6934f95c064272acc"}
Sep 16 19:05:25.886899 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.886867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:25.887093 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:25.886916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:25.887093 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:25.887059    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:25.887256 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:25.887113    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs podName:14abbfab-5c2d-4416-a29a-939762f17972 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:27.887096178 +0000 UTC m=+5.012900547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs") pod "network-metrics-daemon-wvcsr" (UID: "14abbfab-5c2d-4416-a29a-939762f17972") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:25.887546 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:25.887530    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:25.887636 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:25.887583    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret podName:3af2c548-b0f3-4a74-87f7-eb574626b50d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:27.887567091 +0000 UTC m=+5.013371459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret") pod "global-pull-secret-syncer-698qk" (UID: "3af2c548-b0f3-4a74-87f7-eb574626b50d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:26.089026 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:26.088914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:26.089256 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:26.089093    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:05:26.089256 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:26.089118    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:05:26.089256 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:26.089133    2583 projected.go:196] Error preparing data for projected volume kube-api-access-n7rpv for pod openshift-network-diagnostics/network-check-target-8jmnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:26.089256 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:26.089190    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv podName:48c3e63a-7b54-4817-97de-38704d661bea nodeName:}" failed. No retries permitted until 2026-09-16 19:05:28.089169245 +0000 UTC m=+5.214973621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7rpv" (UniqueName: "kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv") pod "network-check-target-8jmnn" (UID: "48c3e63a-7b54-4817-97de-38704d661bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:26.329269 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:26.329177    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:00:24 +0000 UTC" deadline="2028-02-25 13:25:29.864228093 +0000 UTC"
Sep 16 19:05:26.329269 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:26.329218    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12642h20m3.535014744s"
Sep 16 19:05:26.407126 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:26.406316    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:26.407126 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:26.406465    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:26.407126 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:26.406867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:26.407126 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:26.406950    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:26.407126 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:26.407018    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:26.407126 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:26.407089    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:27.011880 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:27.011844    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:05:27.908900 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:27.908087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:27.908900 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:27.908170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:27.908900 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:27.908328    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:27.908900 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:27.908386    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs podName:14abbfab-5c2d-4416-a29a-939762f17972 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:31.908366839 +0000 UTC m=+9.034171215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs") pod "network-metrics-daemon-wvcsr" (UID: "14abbfab-5c2d-4416-a29a-939762f17972") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:27.908900 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:27.908807    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:27.908900 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:27.908856    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret podName:3af2c548-b0f3-4a74-87f7-eb574626b50d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:31.908838846 +0000 UTC m=+9.034643219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret") pod "global-pull-secret-syncer-698qk" (UID: "3af2c548-b0f3-4a74-87f7-eb574626b50d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:28.110315 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:28.110284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:28.110558 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:28.110483    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:05:28.110558 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:28.110503    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:05:28.110558 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:28.110515    2583 projected.go:196] Error preparing data for projected volume kube-api-access-n7rpv for pod openshift-network-diagnostics/network-check-target-8jmnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:28.110792 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:28.110579    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv podName:48c3e63a-7b54-4817-97de-38704d661bea nodeName:}" failed. No retries permitted until 2026-09-16 19:05:32.110562768 +0000 UTC m=+9.236367148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7rpv" (UniqueName: "kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv") pod "network-check-target-8jmnn" (UID: "48c3e63a-7b54-4817-97de-38704d661bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:28.406329 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:28.406252    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:28.406543 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:28.406378    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:28.406747 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:28.406729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:28.406880 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:28.406834    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:28.406972 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:28.406881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:28.406972 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:28.406946    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:30.410656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:30.410609    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:30.411116 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:30.410741    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:30.411116 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:30.410819    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:30.411116 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:30.410896    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:30.411116 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:30.410942    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:30.411116 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:30.411002    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:31.944049 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:31.944010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:31.944455 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:31.944062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:31.944455 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:31.944156    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:31.944455 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:31.944202    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs podName:14abbfab-5c2d-4416-a29a-939762f17972 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:39.944188085 +0000 UTC m=+17.069992451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs") pod "network-metrics-daemon-wvcsr" (UID: "14abbfab-5c2d-4416-a29a-939762f17972") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:31.944683 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:31.944561    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:31.944683 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:31.944622    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret podName:3af2c548-b0f3-4a74-87f7-eb574626b50d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:39.944605634 +0000 UTC m=+17.070410000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret") pod "global-pull-secret-syncer-698qk" (UID: "3af2c548-b0f3-4a74-87f7-eb574626b50d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:32.145198 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:32.145111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:32.145362 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:32.145306    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:05:32.145362 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:32.145328    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:05:32.145362 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:32.145341    2583 projected.go:196] Error preparing data for projected volume kube-api-access-n7rpv for pod openshift-network-diagnostics/network-check-target-8jmnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:32.145511 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:32.145394    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv podName:48c3e63a-7b54-4817-97de-38704d661bea nodeName:}" failed. No retries permitted until 2026-09-16 19:05:40.145377218 +0000 UTC m=+17.271181588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7rpv" (UniqueName: "kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv") pod "network-check-target-8jmnn" (UID: "48c3e63a-7b54-4817-97de-38704d661bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:32.406162 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:32.406086    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:32.406343 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:32.406234    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:32.406343 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:32.406322    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:32.406487 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:32.406465    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:32.406542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:32.406514    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:32.406616 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:32.406587    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:34.406257 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:34.406175    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:34.406631 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:34.406312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:34.406631 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:34.406323    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:34.406631 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:34.406337    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:34.406631 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:34.406395    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:34.406631 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:34.406513    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:36.405570 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:36.405548    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:36.406021 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:36.405547    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:36.406021 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:36.405683    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:36.406021 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:36.405548    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:36.406021 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:36.405761    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:36.406021 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:36.405812    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:38.406017 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:38.405984    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:38.406468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:38.406077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:38.406468 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:38.406087    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:38.406468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:38.406111    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:38.406468 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:38.406224    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:38.406468 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:38.406305    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:40.007550 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:40.007523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:40.007954 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:40.007568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:40.007954 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.007672    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:40.007954 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.007683    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:40.007954 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.007735    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret podName:3af2c548-b0f3-4a74-87f7-eb574626b50d nodeName:}" failed. No retries permitted until 2026-09-16 19:05:56.007718622 +0000 UTC m=+33.133522991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret") pod "global-pull-secret-syncer-698qk" (UID: "3af2c548-b0f3-4a74-87f7-eb574626b50d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:40.007954 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.007751    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs podName:14abbfab-5c2d-4416-a29a-939762f17972 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:56.007742618 +0000 UTC m=+33.133546984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs") pod "network-metrics-daemon-wvcsr" (UID: "14abbfab-5c2d-4416-a29a-939762f17972") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:40.208392 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:40.208362    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:40.208573 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.208502    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:05:40.208573 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.208519    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:05:40.208573 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.208528    2583 projected.go:196] Error preparing data for projected volume kube-api-access-n7rpv for pod openshift-network-diagnostics/network-check-target-8jmnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:40.208729 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.208580    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv podName:48c3e63a-7b54-4817-97de-38704d661bea nodeName:}" failed. No retries permitted until 2026-09-16 19:05:56.208561329 +0000 UTC m=+33.334365712 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7rpv" (UniqueName: "kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv") pod "network-check-target-8jmnn" (UID: "48c3e63a-7b54-4817-97de-38704d661bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:40.406121 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:40.406094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:40.406329 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:40.406096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:40.406329 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.406219    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:40.406329 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:40.406098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:40.406329 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.406300    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:40.406518 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:40.406362    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:42.406200 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:42.406163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:42.406744 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:42.406205    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:42.406744 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:42.406288    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:42.406744 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:42.406342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:42.406744 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:42.406343    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:42.406744 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:42.406399    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:44.406034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.405886    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:44.406682 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.405929    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:44.406682 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:44.406112    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:44.406682 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:44.406177    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:44.406682 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.405929    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:44.406682 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:44.406251    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:44.554727 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.554520    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal" event={"ID":"7033f5edf1da39728ed37f3d87a19a2b","Type":"ContainerStarted","Data":"e1d1758f9cb42111c96850004e754ee3d591d9470d6e6705d9c020bc5a6cea8d"}
Sep 16 19:05:44.558019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.557974    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rm5vh" event={"ID":"6a5f5413-1eac-4ab3-8f93-99c030c6913c","Type":"ContainerStarted","Data":"778b87ca4eeedc5ba5a39c293caf114c56a6cecfbd9a76377874095ca945b2b5"}
Sep 16 19:05:44.564192 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.564113    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:05:44.564684 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.564647    2583 generic.go:356] "Generic (PLEG): container finished" podID="05d5b31c-3de4-44d3-a850-547bf19e0c07" containerID="b19cd7c439a363d35b9886f98746df4894dcfb50ffd750937c84070f0dc992ba" exitCode=1
Sep 16 19:05:44.564782 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.564690    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerStarted","Data":"987f8750880a1ecba937502f84ce7abb0bcdba3c9572d1a2a5ef76e8902e77dc"}
Sep 16 19:05:44.564782 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.564758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerStarted","Data":"a4504e72654bed6a2c0be0faa0cbfe153ba8934c9e698f0900d1ffb050102a13"}
Sep 16 19:05:44.564890 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.564809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerStarted","Data":"82b89630cf2090f52b53310ba96f436a88300fec961308dda4461f1a5cd2fe66"}
Sep 16 19:05:44.564890 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.564858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerDied","Data":"b19cd7c439a363d35b9886f98746df4894dcfb50ffd750937c84070f0dc992ba"}
Sep 16 19:05:44.564993 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.564907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerStarted","Data":"fd7d3855adc8cff6db040a6aeb334833951b636e295157edaf7457fa709b4fbb"}
Sep 16 19:05:44.573916 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:44.573821    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-148.ec2.internal" podStartSLOduration=20.573809194 podStartE2EDuration="20.573809194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:44.573744971 +0000 UTC m=+21.699549360" watchObservedRunningTime="2026-09-16 19:05:44.573809194 +0000 UTC m=+21.699613581"
Sep 16 19:05:45.568406 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.568118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ht7r" event={"ID":"944c14c7-3c60-43cc-9e04-a0903a555407","Type":"ContainerStarted","Data":"1a662e88a062868a7dde1c4c045988e7be371f442dae3e2fbef2d618e8e88152"}
Sep 16 19:05:45.569902 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.569872    2583 generic.go:356] "Generic (PLEG): container finished" podID="fbd3a56e8f3c433944bfaa6b0cafecc6" containerID="b3f9639742766524703b49db8cd1b82c4b4a8d58223918936fd104330043ce23" exitCode=0
Sep 16 19:05:45.570028 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.569954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal" event={"ID":"fbd3a56e8f3c433944bfaa6b0cafecc6","Type":"ContainerDied","Data":"b3f9639742766524703b49db8cd1b82c4b4a8d58223918936fd104330043ce23"}
Sep 16 19:05:45.571671 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.571642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-76twg" event={"ID":"340d58a9-8b5d-416e-8bd3-88d32ba5747f","Type":"ContainerStarted","Data":"4fcbc66ab335fc4f8edd26f66f7b735326a29286ed70727d62e0830810a850ff"}
Sep 16 19:05:45.573378 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.573344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2jggs" event={"ID":"48311335-3e25-4b9c-a365-1ac69298d999","Type":"ContainerStarted","Data":"942bbd87f7dbc63658c2c1500f61abd29956d8f3ad9b3393349cac5c7d758d86"}
Sep 16 19:05:45.575085 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.575059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pcbx2" event={"ID":"601823a6-99aa-4683-b1db-06c7e34c3d86","Type":"ContainerStarted","Data":"c316fe47ba79a5322938cccb0bfff52d1758d2130704b4007f4e196a01fc1799"}
Sep 16 19:05:45.576792 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.576768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xtjc2" event={"ID":"6a025317-d7c0-4050-8f6b-d6cf69387c55","Type":"ContainerStarted","Data":"2b92fc32018744c72d36a3401d3a540b9e857794f4e7545d221ee57a8b2f7e63"}
Sep 16 19:05:45.578593 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.578562    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2fd7290-33ee-4c35-a7f7-62f0704a5de1" containerID="6b2b713736f3fa2de0abf17a831da8821996e39844a3ee5a52269f6f336f0153" exitCode=0
Sep 16 19:05:45.578686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.578611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" event={"ID":"b2fd7290-33ee-4c35-a7f7-62f0704a5de1","Type":"ContainerDied","Data":"6b2b713736f3fa2de0abf17a831da8821996e39844a3ee5a52269f6f336f0153"}
Sep 16 19:05:45.582105 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.582086    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:05:45.582562 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.582493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerStarted","Data":"98f4a25c290de032e83354c35036685e6ef7b9f544d161894f027559423ff709"}
Sep 16 19:05:45.584363 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.584339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v" event={"ID":"781e546d-fa86-4707-930b-089f49571e79","Type":"ContainerStarted","Data":"84729ee22182f05b82b0d9f2754f69ff41564ba93a8cbcce5298aa2818d381d4"}
Sep 16 19:05:45.587862 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.587818    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rm5vh" podStartSLOduration=3.472959564 podStartE2EDuration="22.587803578s" totalImagesPullingTime="19.114844014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:44.597338183 +0000 UTC m=+21.723142573" watchObservedRunningTime="2026-09-16 19:05:45.587803578 +0000 UTC m=+22.713607965"
Sep 16 19:05:45.587990 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.587957    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9ht7r" podStartSLOduration=3.553713278 podStartE2EDuration="22.58794829s" totalImagesPullingTime="19.034235012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:45.587945499 +0000 UTC m=+22.713749887" watchObservedRunningTime="2026-09-16 19:05:45.58794829 +0000 UTC m=+22.713752680"
Sep 16 19:05:45.660309 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.660267    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xtjc2" podStartSLOduration=3.51937508 podStartE2EDuration="22.66025043s" totalImagesPullingTime="19.14087535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:45.627944146 +0000 UTC m=+22.753748546" watchObservedRunningTime="2026-09-16 19:05:45.66025043 +0000 UTC m=+22.786054819"
Sep 16 19:05:45.660668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.660633    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pcbx2" podStartSLOduration=2.785704388 podStartE2EDuration="22.660618032s" totalImagesPullingTime="19.874913644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:45.660184579 +0000 UTC m=+22.785988967" watchObservedRunningTime="2026-09-16 19:05:45.660618032 +0000 UTC m=+22.786422420"
Sep 16 19:05:45.687197 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.685075    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2jggs" podStartSLOduration=3.6023927000000002 podStartE2EDuration="22.68505788s" totalImagesPullingTime="19.08266518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:45.68461791 +0000 UTC m=+22.810422281" watchObservedRunningTime="2026-09-16 19:05:45.68505788 +0000 UTC m=+22.810862268"
Sep 16 19:05:45.722038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.721993    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-76twg" podStartSLOduration=3.698054082 podStartE2EDuration="22.721976189s" totalImagesPullingTime="19.023922107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:45.721879495 +0000 UTC m=+22.847683882" watchObservedRunningTime="2026-09-16 19:05:45.721976189 +0000 UTC m=+22.847780579"
Sep 16 19:05:45.733857 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:45.733833    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 19:05:46.331747 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:46.331636    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T19:05:45.733853514Z","UUID":"0ffcbe1a-4b9a-41e2-831e-841f860cc5af","Handler":null,"Name":"","Endpoint":""}
Sep 16 19:05:46.333399 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:46.333375    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 19:05:46.333560 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:46.333431    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 19:05:46.406656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:46.406192    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:46.406656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:46.406226    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:46.406656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:46.406192    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:46.406656 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:46.406322    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:46.406656 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:46.406382    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:46.406656 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:46.406493    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:46.588719 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:46.588649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v" event={"ID":"781e546d-fa86-4707-930b-089f49571e79","Type":"ContainerStarted","Data":"1df82de7bcd2ee0b6051e69a0ce82617cb7a2a43f8ac818ba5c0113a0c7c53d3"}
Sep 16 19:05:46.590697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:46.590628    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal" event={"ID":"fbd3a56e8f3c433944bfaa6b0cafecc6","Type":"ContainerStarted","Data":"b2853fe5ee1a81d0b27d8dd4e615f0d2ce024d6d049611cec922517a6660f466"}
Sep 16 19:05:46.610989 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:46.610945    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-148.ec2.internal" podStartSLOduration=22.610929289 podStartE2EDuration="22.610929289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:46.610645846 +0000 UTC m=+23.736450233" watchObservedRunningTime="2026-09-16 19:05:46.610929289 +0000 UTC m=+23.736733684"
Sep 16 19:05:47.441232 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:47.440961    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:47.441688 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:47.441666    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:47.596304 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:47.596263    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:05:47.596763 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:47.596730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerStarted","Data":"6a0d591d3247650da1234fc0e4ef4c5f197018a2db169a9ed89bedc7d500c48e"}
Sep 16 19:05:47.600265 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:47.600215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v" event={"ID":"781e546d-fa86-4707-930b-089f49571e79","Type":"ContainerStarted","Data":"269f0a93a8cd91b56646052fc2802090f0fee772cc48224b92d30d5b6a07ad27"}
Sep 16 19:05:47.600774 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:47.600740    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:47.601528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:47.601507    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xtjc2"
Sep 16 19:05:47.619895 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:47.619845    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hb66v" podStartSLOduration=3.576051368 podStartE2EDuration="24.619832736s" totalImagesPullingTime="21.043781368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:47.619459836 +0000 UTC m=+24.745264234" watchObservedRunningTime="2026-09-16 19:05:47.619832736 +0000 UTC m=+24.745637129"
Sep 16 19:05:48.405599 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:48.405574    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:48.405799 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:48.405576    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:48.405799 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:48.405670    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:48.405799 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:48.405763    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:48.405799 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:48.405576    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:48.405987 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:48.405874    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:49.606344 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:49.606295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" event={"ID":"b2fd7290-33ee-4c35-a7f7-62f0704a5de1","Type":"ContainerStarted","Data":"0486d6a80c1906237f81fc269351389cb8af22cd22443db9d11dfa2490b40774"}
Sep 16 19:05:49.610494 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:49.610477    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:05:49.610874 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:49.610845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerStarted","Data":"5fcfe2a302991d0191372831744563a1cbb15d883d987b5997b6c15cd94a5929"}
Sep 16 19:05:49.611174 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:49.611119    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:49.611459 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:49.611194    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:49.611459 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:49.611245    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:49.611459 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:49.611359    2583 scope.go:117] "RemoveContainer" containerID="b19cd7c439a363d35b9886f98746df4894dcfb50ffd750937c84070f0dc992ba"
Sep 16 19:05:49.628754 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:49.628594    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:49.630762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:49.630743    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:05:50.406292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:50.406264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:50.406475 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:50.406377    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:50.406475 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:50.406403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:50.406475 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:50.406399    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:50.406579 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:50.406484    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:50.406579 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:50.406539    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:50.615046 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:50.615012    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2fd7290-33ee-4c35-a7f7-62f0704a5de1" containerID="0486d6a80c1906237f81fc269351389cb8af22cd22443db9d11dfa2490b40774" exitCode=0
Sep 16 19:05:50.615395 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:50.615070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" event={"ID":"b2fd7290-33ee-4c35-a7f7-62f0704a5de1","Type":"ContainerDied","Data":"0486d6a80c1906237f81fc269351389cb8af22cd22443db9d11dfa2490b40774"}
Sep 16 19:05:50.619520 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:50.619503    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:05:50.619911 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:50.619893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" event={"ID":"05d5b31c-3de4-44d3-a850-547bf19e0c07","Type":"ContainerStarted","Data":"0361df726892e870d22b83a2eac46797e0ad10b8ecc0cc632ba47b1c819b681f"}
Sep 16 19:05:51.625013 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:51.624983    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2fd7290-33ee-4c35-a7f7-62f0704a5de1" containerID="a1a6feebdcb34e4dcc6c9c76470453217f27eecdefb1eac1aea2277804f0c89d" exitCode=0
Sep 16 19:05:51.625636 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:51.625064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" event={"ID":"b2fd7290-33ee-4c35-a7f7-62f0704a5de1","Type":"ContainerDied","Data":"a1a6feebdcb34e4dcc6c9c76470453217f27eecdefb1eac1aea2277804f0c89d"}
Sep 16 19:05:51.652465 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:51.652403    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78" podStartSLOduration=9.351144076 podStartE2EDuration="28.652389187s" totalImagesPullingTime="19.301245111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:50.733975023 +0000 UTC m=+27.859779411" watchObservedRunningTime="2026-09-16 19:05:51.652389187 +0000 UTC m=+28.778193575"
Sep 16 19:05:51.864817 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:51.864792    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-698qk"]
Sep 16 19:05:51.864951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:51.864895    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:51.865033 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:51.865009    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:51.865728 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:51.865707    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jmnn"]
Sep 16 19:05:51.865804 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:51.865779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:51.865867 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:51.865852    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:51.870954 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:51.870931    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wvcsr"]
Sep 16 19:05:51.871038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:51.871022    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:51.871163 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:51.871139    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:52.630102 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:52.629883    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2fd7290-33ee-4c35-a7f7-62f0704a5de1" containerID="b35c7dfae77e4bc9169918ed443c957e9670ee7258495d02c54019c18709598a" exitCode=0
Sep 16 19:05:52.630679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:52.629969    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" event={"ID":"b2fd7290-33ee-4c35-a7f7-62f0704a5de1","Type":"ContainerDied","Data":"b35c7dfae77e4bc9169918ed443c957e9670ee7258495d02c54019c18709598a"}
Sep 16 19:05:53.407155 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:53.407123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:53.407322 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:53.407223    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:53.407322 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:53.407265    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:53.407322 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:53.407274    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:53.407513 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:53.407341    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:53.407513 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:53.407448    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:55.407338 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:55.407305    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:55.407774 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:55.407511    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvcsr" podUID="14abbfab-5c2d-4416-a29a-939762f17972"
Sep 16 19:05:55.408023 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:55.407993    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:55.408164 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:55.408137    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-698qk" podUID="3af2c548-b0f3-4a74-87f7-eb574626b50d"
Sep 16 19:05:55.408239 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:55.408199    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:55.408306 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:55.408288    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jmnn" podUID="48c3e63a-7b54-4817-97de-38704d661bea"
Sep 16 19:05:56.024144 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:56.024109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:56.024144 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:56.024158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:56.024373 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:56.024263    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:56.024373 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:56.024263    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:56.024373 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:56.024326    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs podName:14abbfab-5c2d-4416-a29a-939762f17972 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:28.024307401 +0000 UTC m=+65.150111766 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs") pod "network-metrics-daemon-wvcsr" (UID: "14abbfab-5c2d-4416-a29a-939762f17972") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:05:56.024373 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:56.024343    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret podName:3af2c548-b0f3-4a74-87f7-eb574626b50d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:28.024334857 +0000 UTC m=+65.150139225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret") pod "global-pull-secret-syncer-698qk" (UID: "3af2c548-b0f3-4a74-87f7-eb574626b50d") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:05:56.225564 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:56.225528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:56.225808 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:56.225694    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:05:56.225808 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:56.225710    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:05:56.225808 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:56.225720    2583 projected.go:196] Error preparing data for projected volume kube-api-access-n7rpv for pod openshift-network-diagnostics/network-check-target-8jmnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:56.225808 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:56.225768    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv podName:48c3e63a-7b54-4817-97de-38704d661bea nodeName:}" failed. No retries permitted until 2026-09-16 19:06:28.225755557 +0000 UTC m=+65.351559923 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7rpv" (UniqueName: "kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv") pod "network-check-target-8jmnn" (UID: "48c3e63a-7b54-4817-97de-38704d661bea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:05:57.143977 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.143540    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-148.ec2.internal" event="NodeReady"
Sep 16 19:05:57.143977 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.143698    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 19:05:57.222329 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.222303    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jnthf"]
Sep 16 19:05:57.237104 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.237084    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tzklw"]
Sep 16 19:05:57.237211 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.237194    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.241318 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.241301    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 19:05:57.241318 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.241316    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pvzq\""
Sep 16 19:05:57.242910 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.242880    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 19:05:57.259101 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.259069    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jnthf"]
Sep 16 19:05:57.259226 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.259128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tzklw"]
Sep 16 19:05:57.259226 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.259174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tzklw"
Sep 16 19:05:57.263018 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.263001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d25ds\""
Sep 16 19:05:57.263128 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.263050    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 19:05:57.263195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.263175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 19:05:57.263355 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.263341    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 19:05:57.333085 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.333056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-tmp-dir\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.333203 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.333092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.333203 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.333120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gz7r\" (UniqueName: \"kubernetes.io/projected/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-kube-api-access-2gz7r\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.333315 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.333197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-config-volume\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.406526 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.406500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:05:57.406627 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.406535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:05:57.406733 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.406715    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:05:57.411664 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.411641    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 19:05:57.412586 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.412361    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45r44\""
Sep 16 19:05:57.412586 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.412456    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 19:05:57.412586 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.412462    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjb6w\""
Sep 16 19:05:57.416475 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.416459    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 19:05:57.417071 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.417058    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 19:05:57.434199 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.434181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.434279 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.434218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gz7r\" (UniqueName: \"kubernetes.io/projected/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-kube-api-access-2gz7r\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.434279 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.434242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-config-volume\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.434346 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.434277    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpdl\" (UniqueName: \"kubernetes.io/projected/ba6bc7d8-815f-4234-8744-3f5f14d850bf-kube-api-access-ffpdl\") pod \"ingress-canary-tzklw\" (UID: \"ba6bc7d8-815f-4234-8744-3f5f14d850bf\") " pod="openshift-ingress-canary/ingress-canary-tzklw"
Sep 16 19:05:57.434346 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:57.434306    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:05:57.434435 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:57.434354    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls podName:cd6974b2-a024-4ebe-930b-e3df0c1ba35b nodeName:}" failed. No retries permitted until 2026-09-16 19:05:57.934336877 +0000 UTC m=+35.060141255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls") pod "dns-default-jnthf" (UID: "cd6974b2-a024-4ebe-930b-e3df0c1ba35b") : secret "dns-default-metrics-tls" not found
Sep 16 19:05:57.434435 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.434306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-tmp-dir\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.434503 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.434434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba6bc7d8-815f-4234-8744-3f5f14d850bf-cert\") pod \"ingress-canary-tzklw\" (UID: \"ba6bc7d8-815f-4234-8744-3f5f14d850bf\") " pod="openshift-ingress-canary/ingress-canary-tzklw"
Sep 16 19:05:57.434635 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.434619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-tmp-dir\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.434828 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.434815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-config-volume\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.455148 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.455125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gz7r\" (UniqueName: \"kubernetes.io/projected/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-kube-api-access-2gz7r\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.535633 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.535590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba6bc7d8-815f-4234-8744-3f5f14d850bf-cert\") pod \"ingress-canary-tzklw\" (UID: \"ba6bc7d8-815f-4234-8744-3f5f14d850bf\") " pod="openshift-ingress-canary/ingress-canary-tzklw"
Sep 16 19:05:57.535729 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.535649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpdl\" (UniqueName: \"kubernetes.io/projected/ba6bc7d8-815f-4234-8744-3f5f14d850bf-kube-api-access-ffpdl\") pod \"ingress-canary-tzklw\" (UID: \"ba6bc7d8-815f-4234-8744-3f5f14d850bf\") " pod="openshift-ingress-canary/ingress-canary-tzklw"
Sep 16 19:05:57.537911 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.537887    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba6bc7d8-815f-4234-8744-3f5f14d850bf-cert\") pod \"ingress-canary-tzklw\" (UID: \"ba6bc7d8-815f-4234-8744-3f5f14d850bf\") " pod="openshift-ingress-canary/ingress-canary-tzklw"
Sep 16 19:05:57.560370 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.560350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpdl\" (UniqueName: \"kubernetes.io/projected/ba6bc7d8-815f-4234-8744-3f5f14d850bf-kube-api-access-ffpdl\") pod \"ingress-canary-tzklw\" (UID: \"ba6bc7d8-815f-4234-8744-3f5f14d850bf\") " pod="openshift-ingress-canary/ingress-canary-tzklw"
Sep 16 19:05:57.568355 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.568342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tzklw"
Sep 16 19:05:57.645630 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.645391    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2fd7290-33ee-4c35-a7f7-62f0704a5de1" containerID="a3cc4d362b9894eb313e4f33f4569952419da6ccfbaf999a8b43fc872dbc65ba" exitCode=0
Sep 16 19:05:57.645794 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.645444    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" event={"ID":"b2fd7290-33ee-4c35-a7f7-62f0704a5de1","Type":"ContainerDied","Data":"a3cc4d362b9894eb313e4f33f4569952419da6ccfbaf999a8b43fc872dbc65ba"}
Sep 16 19:05:57.903944 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.903913    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tzklw"]
Sep 16 19:05:57.907932 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:05:57.907908    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6bc7d8_815f_4234_8744_3f5f14d850bf.slice/crio-748bdd064540a7a4dbac8a1701b22f1af439407c6d108314db56219012b8d7c2 WatchSource:0}: Error finding container 748bdd064540a7a4dbac8a1701b22f1af439407c6d108314db56219012b8d7c2: Status 404 returned error can't find the container with id 748bdd064540a7a4dbac8a1701b22f1af439407c6d108314db56219012b8d7c2
Sep 16 19:05:57.938498 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:57.938474    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:57.938640 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:57.938624    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:05:57.938705 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:57.938691    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls podName:cd6974b2-a024-4ebe-930b-e3df0c1ba35b nodeName:}" failed. No retries permitted until 2026-09-16 19:05:58.938669996 +0000 UTC m=+36.064474379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls") pod "dns-default-jnthf" (UID: "cd6974b2-a024-4ebe-930b-e3df0c1ba35b") : secret "dns-default-metrics-tls" not found
Sep 16 19:05:58.649069 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:58.649007    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tzklw" event={"ID":"ba6bc7d8-815f-4234-8744-3f5f14d850bf","Type":"ContainerStarted","Data":"748bdd064540a7a4dbac8a1701b22f1af439407c6d108314db56219012b8d7c2"}
Sep 16 19:05:58.652697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:58.652667    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2fd7290-33ee-4c35-a7f7-62f0704a5de1" containerID="87490c855dcc331fabd7f525da6009bbef58f4b6030fb41efe1a8bf8aaee3c0e" exitCode=0
Sep 16 19:05:58.652843 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:58.652725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" event={"ID":"b2fd7290-33ee-4c35-a7f7-62f0704a5de1","Type":"ContainerDied","Data":"87490c855dcc331fabd7f525da6009bbef58f4b6030fb41efe1a8bf8aaee3c0e"}
Sep 16 19:05:58.944346 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:58.944276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:05:58.944489 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:58.944438    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:05:58.944552 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:05:58.944503    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls podName:cd6974b2-a024-4ebe-930b-e3df0c1ba35b nodeName:}" failed. No retries permitted until 2026-09-16 19:06:00.94448319 +0000 UTC m=+38.070287572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls") pod "dns-default-jnthf" (UID: "cd6974b2-a024-4ebe-930b-e3df0c1ba35b") : secret "dns-default-metrics-tls" not found
Sep 16 19:05:59.659947 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:59.659912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" event={"ID":"b2fd7290-33ee-4c35-a7f7-62f0704a5de1","Type":"ContainerStarted","Data":"37e79238701e8a8acd6215a66f00ce16a222793aa910f970d7ed54637d402f37"}
Sep 16 19:05:59.686551 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:05:59.686505    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v9d2x" podStartSLOduration=7.903040303 podStartE2EDuration="36.686490811s" totalImagesPullingTime="28.783450508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:59.685026066 +0000 UTC m=+36.810830453" watchObservedRunningTime="2026-09-16 19:05:59.686490811 +0000 UTC m=+36.812295198"
Sep 16 19:06:00.957603 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:00.957567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:06:00.958045 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:00.957710    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:06:00.958045 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:00.957771    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls podName:cd6974b2-a024-4ebe-930b-e3df0c1ba35b nodeName:}" failed. No retries permitted until 2026-09-16 19:06:04.957757117 +0000 UTC m=+42.083561487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls") pod "dns-default-jnthf" (UID: "cd6974b2-a024-4ebe-930b-e3df0c1ba35b") : secret "dns-default-metrics-tls" not found
Sep 16 19:06:01.665345 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:01.665302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tzklw" event={"ID":"ba6bc7d8-815f-4234-8744-3f5f14d850bf","Type":"ContainerStarted","Data":"dbd219ef6e3b5a3c30fdf95ee0442ecc43a9bd7d7a88a55a144b21c38518f7ee"}
Sep 16 19:06:04.983636 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:04.983435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:06:04.983980 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:04.983563    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:06:04.983980 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:04.983693    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls podName:cd6974b2-a024-4ebe-930b-e3df0c1ba35b nodeName:}" failed. No retries permitted until 2026-09-16 19:06:12.983677806 +0000 UTC m=+50.109482211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls") pod "dns-default-jnthf" (UID: "cd6974b2-a024-4ebe-930b-e3df0c1ba35b") : secret "dns-default-metrics-tls" not found
Sep 16 19:06:13.039304 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:13.039260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:06:13.039818 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:13.039433    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:06:13.039818 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:13.039516    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls podName:cd6974b2-a024-4ebe-930b-e3df0c1ba35b nodeName:}" failed. No retries permitted until 2026-09-16 19:06:29.039499741 +0000 UTC m=+66.165304145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls") pod "dns-default-jnthf" (UID: "cd6974b2-a024-4ebe-930b-e3df0c1ba35b") : secret "dns-default-metrics-tls" not found
Sep 16 19:06:21.637601 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:21.637568    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsb78"
Sep 16 19:06:21.673041 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:21.672992    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tzklw" podStartSLOduration=21.566094071 podStartE2EDuration="24.672978693s" totalImagesPullingTime="3.106884622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:01.690850876 +0000 UTC m=+38.816655264" watchObservedRunningTime="2026-09-16 19:06:21.672978693 +0000 UTC m=+58.798783084"
Sep 16 19:06:28.042817 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.042768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:06:28.042817 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.042816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:06:28.047081 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.047056    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 19:06:28.047492 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.047475    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 19:06:28.053373 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:28.053354    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 19:06:28.053485 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:28.053469    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs podName:14abbfab-5c2d-4416-a29a-939762f17972 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:32.053444675 +0000 UTC m=+129.179249042 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs") pod "network-metrics-daemon-wvcsr" (UID: "14abbfab-5c2d-4416-a29a-939762f17972") : secret "metrics-daemon-secret" not found
Sep 16 19:06:28.057032 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.057002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3af2c548-b0f3-4a74-87f7-eb574626b50d-original-pull-secret\") pod \"global-pull-secret-syncer-698qk\" (UID: \"3af2c548-b0f3-4a74-87f7-eb574626b50d\") " pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:06:28.244226 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.244198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:06:28.247394 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.247376    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 19:06:28.257216 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.257196    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 19:06:28.267552 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.267531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rpv\" (UniqueName: \"kubernetes.io/projected/48c3e63a-7b54-4817-97de-38704d661bea-kube-api-access-n7rpv\") pod \"network-check-target-8jmnn\" (UID: \"48c3e63a-7b54-4817-97de-38704d661bea\") " pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:06:28.317661 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.317608    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-698qk"
Sep 16 19:06:28.331759 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.331743    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45r44\""
Sep 16 19:06:28.338107 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.338087    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:06:28.609384 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.609305    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-698qk"]
Sep 16 19:06:28.613228 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:06:28.613202    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af2c548_b0f3_4a74_87f7_eb574626b50d.slice/crio-7b49fe67e8dfd4f202c375136887393e4313e6bd11c50ebd64b56e538030deb3 WatchSource:0}: Error finding container 7b49fe67e8dfd4f202c375136887393e4313e6bd11c50ebd64b56e538030deb3: Status 404 returned error can't find the container with id 7b49fe67e8dfd4f202c375136887393e4313e6bd11c50ebd64b56e538030deb3
Sep 16 19:06:28.628452 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.628430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jmnn"]
Sep 16 19:06:28.631038 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:06:28.631013    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c3e63a_7b54_4817_97de_38704d661bea.slice/crio-de52078c0e1e5311690b8568876e9b8f139c6afd2375426e9c580cce8051ca79 WatchSource:0}: Error finding container de52078c0e1e5311690b8568876e9b8f139c6afd2375426e9c580cce8051ca79: Status 404 returned error can't find the container with id de52078c0e1e5311690b8568876e9b8f139c6afd2375426e9c580cce8051ca79
Sep 16 19:06:28.747435 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.747384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jmnn" event={"ID":"48c3e63a-7b54-4817-97de-38704d661bea","Type":"ContainerStarted","Data":"de52078c0e1e5311690b8568876e9b8f139c6afd2375426e9c580cce8051ca79"}
Sep 16 19:06:28.748491 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:28.748472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-698qk" event={"ID":"3af2c548-b0f3-4a74-87f7-eb574626b50d","Type":"ContainerStarted","Data":"7b49fe67e8dfd4f202c375136887393e4313e6bd11c50ebd64b56e538030deb3"}
Sep 16 19:06:29.048977 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:29.048946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:06:29.049366 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:29.049122    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:06:29.049366 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:29.049196    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls podName:cd6974b2-a024-4ebe-930b-e3df0c1ba35b nodeName:}" failed. No retries permitted until 2026-09-16 19:07:01.049176076 +0000 UTC m=+98.174980465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls") pod "dns-default-jnthf" (UID: "cd6974b2-a024-4ebe-930b-e3df0c1ba35b") : secret "dns-default-metrics-tls" not found
Sep 16 19:06:33.763927 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:33.763886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jmnn" event={"ID":"48c3e63a-7b54-4817-97de-38704d661bea","Type":"ContainerStarted","Data":"d4f48726fe35194878a8533055ece40b1998399e20471d001739ebb3a1d4cf31"}
Sep 16 19:06:33.764400 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:33.764018    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:06:33.765766 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:33.765742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-698qk" event={"ID":"3af2c548-b0f3-4a74-87f7-eb574626b50d","Type":"ContainerStarted","Data":"58d27b3adb9400ec30a928042f322b41555d7c0abde65e6107a18d2acf58c090"}
Sep 16 19:06:33.784044 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:33.783987    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8jmnn" podStartSLOduration=66.532009696 podStartE2EDuration="1m10.78397305s" totalImagesPullingTime="4.251963354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:33.782341029 +0000 UTC m=+70.908145418" watchObservedRunningTime="2026-09-16 19:06:33.78397305 +0000 UTC m=+70.909777438"
Sep 16 19:06:33.801234 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:33.801185    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-698qk" podStartSLOduration=66.527526578 podStartE2EDuration="1m10.801172459s" totalImagesPullingTime="4.273645881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:33.800253754 +0000 UTC m=+70.926058141" watchObservedRunningTime="2026-09-16 19:06:33.801172459 +0000 UTC m=+70.926976846"
Sep 16 19:06:42.005066 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.005025    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"]
Sep 16 19:06:42.007516 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.007496    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.011629 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.011606    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 19:06:42.012394 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.012373    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8d684\""
Sep 16 19:06:42.013097 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.013076    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 19:06:42.014363 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.014338    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 19:06:42.014512 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.014459    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 19:06:42.029398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.029373    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"]
Sep 16 19:06:42.107232 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.107206    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"]
Sep 16 19:06:42.109203 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.109187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:42.116762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.116740    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 19:06:42.117920 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.117902    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 19:06:42.118022 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.118005    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jqvl5\""
Sep 16 19:06:42.119930 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.119913    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:06:42.134508 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.134483    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-btngq"]
Sep 16 19:06:42.136186 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.136171    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.140434 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.140389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8l2l\" (UniqueName: \"kubernetes.io/projected/2e31c156-f5ae-4b96-8e08-c9ec51203b64-kube-api-access-q8l2l\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.140557 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.140485    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2e31c156-f5ae-4b96-8e08-c9ec51203b64-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.140557 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.140523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.144935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.144916    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 19:06:42.145983 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.145965    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 19:06:42.146067 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.146051    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:06:42.148393 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.148378    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 19:06:42.149312 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.149289    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"]
Sep 16 19:06:42.156019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.156001    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 19:06:42.156296 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.156282    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ftc4t\""
Sep 16 19:06:42.166116 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.166096    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-btngq"]
Sep 16 19:06:42.241797 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.241773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.241882 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.241813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqj4\" (UniqueName: \"kubernetes.io/projected/1a7447f7-35b1-45f4-af73-2ebf91399392-kube-api-access-gxqj4\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.241882 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.241833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a7447f7-35b1-45f4-af73-2ebf91399392-config\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.241882 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.241857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:42.242048 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.241904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a7447f7-35b1-45f4-af73-2ebf91399392-trusted-ca\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.242048 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.241918    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nxmn\" (UniqueName: \"kubernetes.io/projected/3700d6cd-271c-4def-a213-45d38b153ee2-kube-api-access-8nxmn\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:42.242048 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:42.241925    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:42.242048 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.241942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a7447f7-35b1-45f4-af73-2ebf91399392-serving-cert\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.242048 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:42.241983    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls podName:2e31c156-f5ae-4b96-8e08-c9ec51203b64 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:42.741966001 +0000 UTC m=+79.867770370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tffxw" (UID: "2e31c156-f5ae-4b96-8e08-c9ec51203b64") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:42.242048 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.242015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8l2l\" (UniqueName: \"kubernetes.io/projected/2e31c156-f5ae-4b96-8e08-c9ec51203b64-kube-api-access-q8l2l\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.242348 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.242053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2e31c156-f5ae-4b96-8e08-c9ec51203b64-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.242710 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.242690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2e31c156-f5ae-4b96-8e08-c9ec51203b64-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.257785 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.257740    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8l2l\" (UniqueName: \"kubernetes.io/projected/2e31c156-f5ae-4b96-8e08-c9ec51203b64-kube-api-access-q8l2l\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.343300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.343280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqj4\" (UniqueName: \"kubernetes.io/projected/1a7447f7-35b1-45f4-af73-2ebf91399392-kube-api-access-gxqj4\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.343395 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.343305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a7447f7-35b1-45f4-af73-2ebf91399392-config\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.343395 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.343325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:42.343395 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.343351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a7447f7-35b1-45f4-af73-2ebf91399392-trusted-ca\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.343591 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:42.343475    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:06:42.343591 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.343475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nxmn\" (UniqueName: \"kubernetes.io/projected/3700d6cd-271c-4def-a213-45d38b153ee2-kube-api-access-8nxmn\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:42.343591 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:42.343544    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls podName:3700d6cd-271c-4def-a213-45d38b153ee2 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:42.843525815 +0000 UTC m=+79.969330200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9txmj" (UID: "3700d6cd-271c-4def-a213-45d38b153ee2") : secret "samples-operator-tls" not found
Sep 16 19:06:42.343591 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.343586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a7447f7-35b1-45f4-af73-2ebf91399392-serving-cert\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.343985 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.343964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a7447f7-35b1-45f4-af73-2ebf91399392-config\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.344101 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.344081    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a7447f7-35b1-45f4-af73-2ebf91399392-trusted-ca\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.345828 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.345810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a7447f7-35b1-45f4-af73-2ebf91399392-serving-cert\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.358101 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.358076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqj4\" (UniqueName: \"kubernetes.io/projected/1a7447f7-35b1-45f4-af73-2ebf91399392-kube-api-access-gxqj4\") pod \"console-operator-696884fb9c-btngq\" (UID: \"1a7447f7-35b1-45f4-af73-2ebf91399392\") " pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.358504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.358486    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nxmn\" (UniqueName: \"kubernetes.io/projected/3700d6cd-271c-4def-a213-45d38b153ee2-kube-api-access-8nxmn\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:42.444067 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.444044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:42.737705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.737671    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-btngq"]
Sep 16 19:06:42.741570 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:06:42.741539    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7447f7_35b1_45f4_af73_2ebf91399392.slice/crio-713fc341538a3bc3aacf191997f202714b1fcd60a34665b600566c96daeaee7d WatchSource:0}: Error finding container 713fc341538a3bc3aacf191997f202714b1fcd60a34665b600566c96daeaee7d: Status 404 returned error can't find the container with id 713fc341538a3bc3aacf191997f202714b1fcd60a34665b600566c96daeaee7d
Sep 16 19:06:42.746189 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.746148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:42.746382 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:42.746304    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:42.746382 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:42.746367    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls podName:2e31c156-f5ae-4b96-8e08-c9ec51203b64 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:43.746351492 +0000 UTC m=+80.872155858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tffxw" (UID: "2e31c156-f5ae-4b96-8e08-c9ec51203b64") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:42.789191 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.789163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-btngq" event={"ID":"1a7447f7-35b1-45f4-af73-2ebf91399392","Type":"ContainerStarted","Data":"713fc341538a3bc3aacf191997f202714b1fcd60a34665b600566c96daeaee7d"}
Sep 16 19:06:42.847173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:42.847151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:42.847308 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:42.847290    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:06:42.847355 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:42.847345    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls podName:3700d6cd-271c-4def-a213-45d38b153ee2 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:43.847331233 +0000 UTC m=+80.973135599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9txmj" (UID: "3700d6cd-271c-4def-a213-45d38b153ee2") : secret "samples-operator-tls" not found
Sep 16 19:06:43.753022 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:43.752981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:43.753481 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:43.753139    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:43.753481 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:43.753221    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls podName:2e31c156-f5ae-4b96-8e08-c9ec51203b64 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:45.753200151 +0000 UTC m=+82.879004541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tffxw" (UID: "2e31c156-f5ae-4b96-8e08-c9ec51203b64") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:43.853688 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:43.853657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:43.853847 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:43.853827    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:06:43.853926 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:43.853894    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls podName:3700d6cd-271c-4def-a213-45d38b153ee2 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:45.853874513 +0000 UTC m=+82.979678893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9txmj" (UID: "3700d6cd-271c-4def-a213-45d38b153ee2") : secret "samples-operator-tls" not found
Sep 16 19:06:43.994766 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:43.994733    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"]
Sep 16 19:06:43.997231 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:43.997212    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.002056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.002035    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 19:06:44.002165 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.002126    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 19:06:44.002245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.002170    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 19:06:44.003250 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.003201    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w4bzl\""
Sep 16 19:06:44.003338 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.003296    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:06:44.018270 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.018244    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"]
Sep 16 19:06:44.055592 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.055567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hsw\" (UniqueName: \"kubernetes.io/projected/0ef80565-c6f2-487c-9086-3f98f46e17a4-kube-api-access-z4hsw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rq8bn\" (UID: \"0ef80565-c6f2-487c-9086-3f98f46e17a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.055687 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.055599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef80565-c6f2-487c-9086-3f98f46e17a4-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rq8bn\" (UID: \"0ef80565-c6f2-487c-9086-3f98f46e17a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.055687 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.055618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ef80565-c6f2-487c-9086-3f98f46e17a4-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rq8bn\" (UID: \"0ef80565-c6f2-487c-9086-3f98f46e17a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.156039 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.155945    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hsw\" (UniqueName: \"kubernetes.io/projected/0ef80565-c6f2-487c-9086-3f98f46e17a4-kube-api-access-z4hsw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rq8bn\" (UID: \"0ef80565-c6f2-487c-9086-3f98f46e17a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.156039 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.155978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef80565-c6f2-487c-9086-3f98f46e17a4-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rq8bn\" (UID: \"0ef80565-c6f2-487c-9086-3f98f46e17a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.156039 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.155997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ef80565-c6f2-487c-9086-3f98f46e17a4-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rq8bn\" (UID: \"0ef80565-c6f2-487c-9086-3f98f46e17a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.156599 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.156578    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ef80565-c6f2-487c-9086-3f98f46e17a4-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rq8bn\" (UID: \"0ef80565-c6f2-487c-9086-3f98f46e17a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.158266 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.158248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef80565-c6f2-487c-9086-3f98f46e17a4-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rq8bn\" (UID: \"0ef80565-c6f2-487c-9086-3f98f46e17a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.165194 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.165171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hsw\" (UniqueName: \"kubernetes.io/projected/0ef80565-c6f2-487c-9086-3f98f46e17a4-kube-api-access-z4hsw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rq8bn\" (UID: \"0ef80565-c6f2-487c-9086-3f98f46e17a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.308025 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.307936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"
Sep 16 19:06:44.795986 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.795960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-btngq" event={"ID":"1a7447f7-35b1-45f4-af73-2ebf91399392","Type":"ContainerStarted","Data":"cd82110d237057b23f8629e4572d14593c16f43fdc63fd4360e0d0130d1e5315"}
Sep 16 19:06:44.796368 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.796080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:44.797454 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.797428    2583 patch_prober.go:28] interesting pod/console-operator-696884fb9c-btngq container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.10:8443/readyz\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body=
Sep 16 19:06:44.797571 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.797479    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podUID="1a7447f7-35b1-45f4-af73-2ebf91399392" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.10:8443/readyz\": dial tcp 10.134.0.10:8443: connect: connection refused"
Sep 16 19:06:44.821306 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.821265    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podStartSLOduration=0.859700485 podStartE2EDuration="2.82125133s" totalImagesPullingTime="1.961550845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:44.819988441 +0000 UTC m=+81.945792828" watchObservedRunningTime="2026-09-16 19:06:44.82125133 +0000 UTC m=+81.947055768"
Sep 16 19:06:44.914925 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:44.914878    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn"]
Sep 16 19:06:44.917795 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:06:44.917762    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef80565_c6f2_487c_9086_3f98f46e17a4.slice/crio-ccd7417b05c83b0c08754e6671937bef624141533bb9099a9a3cdaa75087630b WatchSource:0}: Error finding container ccd7417b05c83b0c08754e6671937bef624141533bb9099a9a3cdaa75087630b: Status 404 returned error can't find the container with id ccd7417b05c83b0c08754e6671937bef624141533bb9099a9a3cdaa75087630b
Sep 16 19:06:45.768785 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.768751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:45.768946 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:45.768918    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:45.769003 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:45.768996    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls podName:2e31c156-f5ae-4b96-8e08-c9ec51203b64 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:49.768976065 +0000 UTC m=+86.894780444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tffxw" (UID: "2e31c156-f5ae-4b96-8e08-c9ec51203b64") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:45.803197 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.803166    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/0.log"
Sep 16 19:06:45.803627 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.803213    2583 generic.go:356] "Generic (PLEG): container finished" podID="1a7447f7-35b1-45f4-af73-2ebf91399392" containerID="cd82110d237057b23f8629e4572d14593c16f43fdc63fd4360e0d0130d1e5315" exitCode=255
Sep 16 19:06:45.803627 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.803310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-btngq" event={"ID":"1a7447f7-35b1-45f4-af73-2ebf91399392","Type":"ContainerDied","Data":"cd82110d237057b23f8629e4572d14593c16f43fdc63fd4360e0d0130d1e5315"}
Sep 16 19:06:45.803627 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.803545    2583 scope.go:117] "RemoveContainer" containerID="cd82110d237057b23f8629e4572d14593c16f43fdc63fd4360e0d0130d1e5315"
Sep 16 19:06:45.804716 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.804694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn" event={"ID":"0ef80565-c6f2-487c-9086-3f98f46e17a4","Type":"ContainerStarted","Data":"ccd7417b05c83b0c08754e6671937bef624141533bb9099a9a3cdaa75087630b"}
Sep 16 19:06:45.821327 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.821294    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tbqz7"]
Sep 16 19:06:45.825601 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.825583    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.829076 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.829059    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 19:06:45.829483 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.829465    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 19:06:45.829587 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.829469    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 19:06:45.829587 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.829501    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-27w29\""
Sep 16 19:06:45.829831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.829805    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 19:06:45.850404 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.850381    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbqz7"]
Sep 16 19:06:45.869574 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.869553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5fg\" (UniqueName: \"kubernetes.io/projected/86222f33-a533-4e08-ba27-ff62049bcba7-kube-api-access-js5fg\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.869667 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.869587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86222f33-a533-4e08-ba27-ff62049bcba7-data-volume\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.869745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.869722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86222f33-a533-4e08-ba27-ff62049bcba7-crio-socket\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.869843 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.869780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:45.869843 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.869831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.869953 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.869876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.870484 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:45.870468    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:06:45.870553 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:45.870528    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls podName:3700d6cd-271c-4def-a213-45d38b153ee2 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:49.870510633 +0000 UTC m=+86.996315000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9txmj" (UID: "3700d6cd-271c-4def-a213-45d38b153ee2") : secret "samples-operator-tls" not found
Sep 16 19:06:45.970653 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.970623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86222f33-a533-4e08-ba27-ff62049bcba7-crio-socket\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.970781 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.970668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.970781 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.970691    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.970781 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.970713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5fg\" (UniqueName: \"kubernetes.io/projected/86222f33-a533-4e08-ba27-ff62049bcba7-kube-api-access-js5fg\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.970781 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.970731    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86222f33-a533-4e08-ba27-ff62049bcba7-data-volume\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.970781 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.970757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86222f33-a533-4e08-ba27-ff62049bcba7-crio-socket\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.970781 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:45.970762    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:06:45.971032 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:45.970819    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:45.971032 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:45.970861    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:46.470843231 +0000 UTC m=+83.596647610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : configmap "kube-rbac-proxy" not found
Sep 16 19:06:45.971032 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:45.970879    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:46.470870119 +0000 UTC m=+83.596674485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:45.971165 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.971065    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86222f33-a533-4e08-ba27-ff62049bcba7-data-volume\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:45.987299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:45.987271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5fg\" (UniqueName: \"kubernetes.io/projected/86222f33-a533-4e08-ba27-ff62049bcba7-kube-api-access-js5fg\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:46.475051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:46.475013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:46.475232 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:46.475129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:46.475232 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:46.475174    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:46.475232 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:46.475209    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:06:46.475399 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:46.475247    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:47.475227117 +0000 UTC m=+84.601031482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:46.475399 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:46.475279    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:47.475263286 +0000 UTC m=+84.601067672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : configmap "kube-rbac-proxy" not found
Sep 16 19:06:46.809840 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:46.809783    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/1.log"
Sep 16 19:06:46.810193 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:46.810169    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/0.log"
Sep 16 19:06:46.810247 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:46.810215    2583 generic.go:356] "Generic (PLEG): container finished" podID="1a7447f7-35b1-45f4-af73-2ebf91399392" containerID="42eea3f3a90599310c2f70aa0946bf9d2d4845548279d22e7001e7eabf431305" exitCode=255
Sep 16 19:06:46.810283 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:46.810249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-btngq" event={"ID":"1a7447f7-35b1-45f4-af73-2ebf91399392","Type":"ContainerDied","Data":"42eea3f3a90599310c2f70aa0946bf9d2d4845548279d22e7001e7eabf431305"}
Sep 16 19:06:46.810316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:46.810303    2583 scope.go:117] "RemoveContainer" containerID="cd82110d237057b23f8629e4572d14593c16f43fdc63fd4360e0d0130d1e5315"
Sep 16 19:06:46.810547 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:46.810529    2583 scope.go:117] "RemoveContainer" containerID="42eea3f3a90599310c2f70aa0946bf9d2d4845548279d22e7001e7eabf431305"
Sep 16 19:06:46.810749 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:46.810728    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-btngq_openshift-console-operator(1a7447f7-35b1-45f4-af73-2ebf91399392)\"" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podUID="1a7447f7-35b1-45f4-af73-2ebf91399392"
Sep 16 19:06:47.106212 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.106145    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"]
Sep 16 19:06:47.108962 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.108947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.111375 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.111348    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 19:06:47.111651 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.111636    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 19:06:47.111998 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.111983    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 19:06:47.112437 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.112401    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kr4kj\""
Sep 16 19:06:47.112538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.112521    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:06:47.122120 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.122098    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"]
Sep 16 19:06:47.179539 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.179515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh52k\" (UniqueName: \"kubernetes.io/projected/a24fdd13-7ba7-45e6-9d0a-5d52ff23188c-kube-api-access-sh52k\") pod \"service-ca-operator-cf7c776b7-l82nv\" (UID: \"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.179628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.179570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a24fdd13-7ba7-45e6-9d0a-5d52ff23188c-serving-cert\") pod \"service-ca-operator-cf7c776b7-l82nv\" (UID: \"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.179628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.179585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a24fdd13-7ba7-45e6-9d0a-5d52ff23188c-config\") pod \"service-ca-operator-cf7c776b7-l82nv\" (UID: \"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.280161 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.280139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a24fdd13-7ba7-45e6-9d0a-5d52ff23188c-serving-cert\") pod \"service-ca-operator-cf7c776b7-l82nv\" (UID: \"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.280161 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.280161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a24fdd13-7ba7-45e6-9d0a-5d52ff23188c-config\") pod \"service-ca-operator-cf7c776b7-l82nv\" (UID: \"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.280315 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.280213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh52k\" (UniqueName: \"kubernetes.io/projected/a24fdd13-7ba7-45e6-9d0a-5d52ff23188c-kube-api-access-sh52k\") pod \"service-ca-operator-cf7c776b7-l82nv\" (UID: \"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.280677 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.280657    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a24fdd13-7ba7-45e6-9d0a-5d52ff23188c-config\") pod \"service-ca-operator-cf7c776b7-l82nv\" (UID: \"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.282426 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.282392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a24fdd13-7ba7-45e6-9d0a-5d52ff23188c-serving-cert\") pod \"service-ca-operator-cf7c776b7-l82nv\" (UID: \"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.292181 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.292163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh52k\" (UniqueName: \"kubernetes.io/projected/a24fdd13-7ba7-45e6-9d0a-5d52ff23188c-kube-api-access-sh52k\") pod \"service-ca-operator-cf7c776b7-l82nv\" (UID: \"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.417110 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.417090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"
Sep 16 19:06:47.488435 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.482775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:47.488435 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.482832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:47.488435 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:47.483038    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:47.488435 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:47.483100    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:49.483081892 +0000 UTC m=+86.608886265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:47.488435 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:47.487515    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:06:47.488435 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:47.487588    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:49.487568321 +0000 UTC m=+86.613372692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : configmap "kube-rbac-proxy" not found
Sep 16 19:06:47.510486 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.510466    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-76twg_340d58a9-8b5d-416e-8bd3-88d32ba5747f/dns-node-resolver/0.log"
Sep 16 19:06:47.703457 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.703390    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv"]
Sep 16 19:06:47.706134 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:06:47.706106    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24fdd13_7ba7_45e6_9d0a_5d52ff23188c.slice/crio-621618e66635e6fb7b91dc4fe43fed0d452b27b981212a5eddcc154323268442 WatchSource:0}: Error finding container 621618e66635e6fb7b91dc4fe43fed0d452b27b981212a5eddcc154323268442: Status 404 returned error can't find the container with id 621618e66635e6fb7b91dc4fe43fed0d452b27b981212a5eddcc154323268442
Sep 16 19:06:47.814096 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.814071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv" event={"ID":"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c","Type":"ContainerStarted","Data":"621618e66635e6fb7b91dc4fe43fed0d452b27b981212a5eddcc154323268442"}
Sep 16 19:06:47.815674 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.815652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn" event={"ID":"0ef80565-c6f2-487c-9086-3f98f46e17a4","Type":"ContainerStarted","Data":"d5e1b775a5ba61da99907c011a784d4828cee147da21e5bc393ae431169e99fb"}
Sep 16 19:06:47.817214 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.817195    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/1.log"
Sep 16 19:06:47.817469 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.817454    2583 scope.go:117] "RemoveContainer" containerID="42eea3f3a90599310c2f70aa0946bf9d2d4845548279d22e7001e7eabf431305"
Sep 16 19:06:47.817586 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:47.817569    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-btngq_openshift-console-operator(1a7447f7-35b1-45f4-af73-2ebf91399392)\"" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podUID="1a7447f7-35b1-45f4-af73-2ebf91399392"
Sep 16 19:06:47.834699 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:47.834657    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn" podStartSLOduration=2.937105608 podStartE2EDuration="4.834644042s" totalImagesPullingTime="1.897538434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:47.833897889 +0000 UTC m=+84.959702276" watchObservedRunningTime="2026-09-16 19:06:47.834644042 +0000 UTC m=+84.960448430"
Sep 16 19:06:48.505244 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.505216    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6"]
Sep 16 19:06:48.508300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.508229    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6"
Sep 16 19:06:48.512679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.512467    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 19:06:48.512679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.512481    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n9rq4\""
Sep 16 19:06:48.512679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.512557    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 19:06:48.513205 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.513182    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ht7r_944c14c7-3c60-43cc-9e04-a0903a555407/node-ca/0.log"
Sep 16 19:06:48.522078 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.522002    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6"]
Sep 16 19:06:48.592706 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.592679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5c7\" (UniqueName: \"kubernetes.io/projected/b96d0288-1dff-46a1-a3fe-68b68c696520-kube-api-access-xq5c7\") pod \"migrator-8b4d697c5-t6nf6\" (UID: \"b96d0288-1dff-46a1-a3fe-68b68c696520\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6"
Sep 16 19:06:48.692984 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.692958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5c7\" (UniqueName: \"kubernetes.io/projected/b96d0288-1dff-46a1-a3fe-68b68c696520-kube-api-access-xq5c7\") pod \"migrator-8b4d697c5-t6nf6\" (UID: \"b96d0288-1dff-46a1-a3fe-68b68c696520\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6"
Sep 16 19:06:48.704215 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.704190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5c7\" (UniqueName: \"kubernetes.io/projected/b96d0288-1dff-46a1-a3fe-68b68c696520-kube-api-access-xq5c7\") pod \"migrator-8b4d697c5-t6nf6\" (UID: \"b96d0288-1dff-46a1-a3fe-68b68c696520\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6"
Sep 16 19:06:48.820663 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:48.820610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6"
Sep 16 19:06:49.102178 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:49.102105    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6"]
Sep 16 19:06:49.105759 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:06:49.105730    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96d0288_1dff_46a1_a3fe_68b68c696520.slice/crio-52dcc399ac772919629fa934f27b834b94983986ac2827c9dc6a4152efdcf592 WatchSource:0}: Error finding container 52dcc399ac772919629fa934f27b834b94983986ac2827c9dc6a4152efdcf592: Status 404 returned error can't find the container with id 52dcc399ac772919629fa934f27b834b94983986ac2827c9dc6a4152efdcf592
Sep 16 19:06:49.309644 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:49.309609    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tzklw_ba6bc7d8-815f-4234-8744-3f5f14d850bf/serve-healthcheck-canary/0.log"
Sep 16 19:06:49.500392 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:49.500356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:49.500560 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:49.500402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:49.500560 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:49.500493    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:06:49.500560 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:49.500560    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:53.500540384 +0000 UTC m=+90.626344772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : configmap "kube-rbac-proxy" not found
Sep 16 19:06:49.500703 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:49.500563    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:49.500703 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:49.500610    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:53.500595294 +0000 UTC m=+90.626399661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:49.804227 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:49.804193    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:49.804348 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:49.804330    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:49.804400 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:49.804388    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls podName:2e31c156-f5ae-4b96-8e08-c9ec51203b64 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:57.804372784 +0000 UTC m=+94.930177149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tffxw" (UID: "2e31c156-f5ae-4b96-8e08-c9ec51203b64") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:49.824470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:49.824438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv" event={"ID":"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c","Type":"ContainerStarted","Data":"028ee16fda204650513f6ca1e04f8adb5666f156c33c832c83ee22c27ab70c21"}
Sep 16 19:06:49.825707 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:49.825684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6" event={"ID":"b96d0288-1dff-46a1-a3fe-68b68c696520","Type":"ContainerStarted","Data":"52dcc399ac772919629fa934f27b834b94983986ac2827c9dc6a4152efdcf592"}
Sep 16 19:06:49.846717 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:49.846677    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv" podStartSLOduration=0.853812141 podStartE2EDuration="2.846662794s" totalImagesPullingTime="1.992850653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:49.845055827 +0000 UTC m=+86.970860194" watchObservedRunningTime="2026-09-16 19:06:49.846662794 +0000 UTC m=+86.972467182"
Sep 16 19:06:49.905745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:49.905715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:49.905857 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:49.905834    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:06:49.905914 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:49.905895    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls podName:3700d6cd-271c-4def-a213-45d38b153ee2 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:57.905878234 +0000 UTC m=+95.031682614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9txmj" (UID: "3700d6cd-271c-4def-a213-45d38b153ee2") : secret "samples-operator-tls" not found
Sep 16 19:06:50.833478 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:50.833441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6" event={"ID":"b96d0288-1dff-46a1-a3fe-68b68c696520","Type":"ContainerStarted","Data":"d42569eff9b0ca04fbb9f543354932674b1e1133257a29c17b04205f81dfa284"}
Sep 16 19:06:51.838862 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:51.838814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6" event={"ID":"b96d0288-1dff-46a1-a3fe-68b68c696520","Type":"ContainerStarted","Data":"fa6f74dc1164a9379a3fb40b931cc53b1c551a0c5ff53b48e2dd771b04399017"}
Sep 16 19:06:51.871005 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:51.870957    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t6nf6" podStartSLOduration=2.211579811 podStartE2EDuration="3.870941774s" totalImagesPullingTime="1.659361963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:51.870695664 +0000 UTC m=+88.996500054" watchObservedRunningTime="2026-09-16 19:06:51.870941774 +0000 UTC m=+88.996746162"
Sep 16 19:06:52.444154 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:52.444124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:52.444514 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:52.444498    2583 scope.go:117] "RemoveContainer" containerID="42eea3f3a90599310c2f70aa0946bf9d2d4845548279d22e7001e7eabf431305"
Sep 16 19:06:52.444660 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:52.444640    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-btngq_openshift-console-operator(1a7447f7-35b1-45f4-af73-2ebf91399392)\"" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podUID="1a7447f7-35b1-45f4-af73-2ebf91399392"
Sep 16 19:06:53.535328 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:53.535296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:53.535689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:53.535335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:06:53.535689 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:53.535422    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:06:53.535689 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:53.535436    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:53.535689 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:53.535480    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:01.535467862 +0000 UTC m=+98.661272227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:06:53.535689 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:53.535525    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:01.535518238 +0000 UTC m=+98.661322604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : configmap "kube-rbac-proxy" not found
Sep 16 19:06:54.796893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:54.796852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:06:54.797300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:54.797220    2583 scope.go:117] "RemoveContainer" containerID="42eea3f3a90599310c2f70aa0946bf9d2d4845548279d22e7001e7eabf431305"
Sep 16 19:06:54.797384 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:54.797361    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-btngq_openshift-console-operator(1a7447f7-35b1-45f4-af73-2ebf91399392)\"" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podUID="1a7447f7-35b1-45f4-af73-2ebf91399392"
Sep 16 19:06:57.866867 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:57.866830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:06:57.867260 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:57.866953    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:57.867260 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:57.867004    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls podName:2e31c156-f5ae-4b96-8e08-c9ec51203b64 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:13.866990195 +0000 UTC m=+110.992794561 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tffxw" (UID: "2e31c156-f5ae-4b96-8e08-c9ec51203b64") : secret "cluster-monitoring-operator-tls" not found
Sep 16 19:06:57.967614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:06:57.967575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:06:57.967778 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:57.967723    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:06:57.967825 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:06:57.967805    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls podName:3700d6cd-271c-4def-a213-45d38b153ee2 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:13.967789506 +0000 UTC m=+111.093593872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9txmj" (UID: "3700d6cd-271c-4def-a213-45d38b153ee2") : secret "samples-operator-tls" not found
Sep 16 19:07:01.092117 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:01.092066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:07:01.092600 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:07:01.092200    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:07:01.092600 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:07:01.092257    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls podName:cd6974b2-a024-4ebe-930b-e3df0c1ba35b nodeName:}" failed. No retries permitted until 2026-09-16 19:08:05.092243783 +0000 UTC m=+162.218048149 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls") pod "dns-default-jnthf" (UID: "cd6974b2-a024-4ebe-930b-e3df0c1ba35b") : secret "dns-default-metrics-tls" not found
Sep 16 19:07:01.596007 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:01.595963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:07:01.596168 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:01.596022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:07:01.596168 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:07:01.596138    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:07:01.596250 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:07:01.596205    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls podName:86222f33-a533-4e08-ba27-ff62049bcba7 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:17.596186008 +0000 UTC m=+114.721990374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqz7" (UID: "86222f33-a533-4e08-ba27-ff62049bcba7") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:07:01.596658 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:01.596636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86222f33-a533-4e08-ba27-ff62049bcba7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:07:04.770720 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:04.770690    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8jmnn"
Sep 16 19:07:09.405903 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:09.405869    2583 scope.go:117] "RemoveContainer" containerID="42eea3f3a90599310c2f70aa0946bf9d2d4845548279d22e7001e7eabf431305"
Sep 16 19:07:09.898727 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:09.898696    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:07:09.899234 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:09.899215    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/1.log"
Sep 16 19:07:09.899304 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:09.899257    2583 generic.go:356] "Generic (PLEG): container finished" podID="1a7447f7-35b1-45f4-af73-2ebf91399392" containerID="ceb6c81543d4355b53f1fb5da1fe7a658effef516eeb00398fd864d8210cbfb1" exitCode=255
Sep 16 19:07:09.899350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:09.899338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-btngq" event={"ID":"1a7447f7-35b1-45f4-af73-2ebf91399392","Type":"ContainerDied","Data":"ceb6c81543d4355b53f1fb5da1fe7a658effef516eeb00398fd864d8210cbfb1"}
Sep 16 19:07:09.899398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:09.899391    2583 scope.go:117] "RemoveContainer" containerID="42eea3f3a90599310c2f70aa0946bf9d2d4845548279d22e7001e7eabf431305"
Sep 16 19:07:09.899749 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:09.899720    2583 scope.go:117] "RemoveContainer" containerID="ceb6c81543d4355b53f1fb5da1fe7a658effef516eeb00398fd864d8210cbfb1"
Sep 16 19:07:09.899962 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:07:09.899938    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-btngq_openshift-console-operator(1a7447f7-35b1-45f4-af73-2ebf91399392)\"" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podUID="1a7447f7-35b1-45f4-af73-2ebf91399392"
Sep 16 19:07:10.905104 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:10.905078    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:07:12.444342 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:12.444298    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:07:12.444820 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:12.444679    2583 scope.go:117] "RemoveContainer" containerID="ceb6c81543d4355b53f1fb5da1fe7a658effef516eeb00398fd864d8210cbfb1"
Sep 16 19:07:12.444893 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:07:12.444869    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-btngq_openshift-console-operator(1a7447f7-35b1-45f4-af73-2ebf91399392)\"" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podUID="1a7447f7-35b1-45f4-af73-2ebf91399392"
Sep 16 19:07:13.890090 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:13.890043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:07:13.896502 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:13.896471    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2e31c156-f5ae-4b96-8e08-c9ec51203b64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tffxw\" (UID: \"2e31c156-f5ae-4b96-8e08-c9ec51203b64\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:07:13.990652 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:13.990612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:07:13.993268 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:13.993245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3700d6cd-271c-4def-a213-45d38b153ee2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9txmj\" (UID: \"3700d6cd-271c-4def-a213-45d38b153ee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:07:14.116832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:14.116797    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"
Sep 16 19:07:14.217066 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:14.217037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"
Sep 16 19:07:14.401157 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:14.401129    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e31c156_f5ae_4b96_8e08_c9ec51203b64.slice/crio-34d3a4a94f9c6082526e53d4ef3a6e3009b2d925b6a6d64b3f7ae29c24baa458 WatchSource:0}: Error finding container 34d3a4a94f9c6082526e53d4ef3a6e3009b2d925b6a6d64b3f7ae29c24baa458: Status 404 returned error can't find the container with id 34d3a4a94f9c6082526e53d4ef3a6e3009b2d925b6a6d64b3f7ae29c24baa458
Sep 16 19:07:14.404194 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:14.404167    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw"]
Sep 16 19:07:14.515058 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:14.515037    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj"]
Sep 16 19:07:14.797141 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:14.797073    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:07:14.797493 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:14.797475    2583 scope.go:117] "RemoveContainer" containerID="ceb6c81543d4355b53f1fb5da1fe7a658effef516eeb00398fd864d8210cbfb1"
Sep 16 19:07:14.797658 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:07:14.797636    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-btngq_openshift-console-operator(1a7447f7-35b1-45f4-af73-2ebf91399392)\"" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podUID="1a7447f7-35b1-45f4-af73-2ebf91399392"
Sep 16 19:07:14.919017 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:14.918988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj" event={"ID":"3700d6cd-271c-4def-a213-45d38b153ee2","Type":"ContainerStarted","Data":"6eb578d5649f5b8a634cb3bf75a2622d86159fd10f61e6f8405df4e771996a3d"}
Sep 16 19:07:14.920271 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:14.920245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw" event={"ID":"2e31c156-f5ae-4b96-8e08-c9ec51203b64","Type":"ContainerStarted","Data":"34d3a4a94f9c6082526e53d4ef3a6e3009b2d925b6a6d64b3f7ae29c24baa458"}
Sep 16 19:07:16.928679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:16.928629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj" event={"ID":"3700d6cd-271c-4def-a213-45d38b153ee2","Type":"ContainerStarted","Data":"560fab13a6f6f9f094ca07b88e91528049f97870340e01168805b2ed9db07961"}
Sep 16 19:07:16.929101 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:16.928694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj" event={"ID":"3700d6cd-271c-4def-a213-45d38b153ee2","Type":"ContainerStarted","Data":"f2a49f981f34c84621e7ecfff15a17baa2202a5f6edccee106a38fdc7e7dc943"}
Sep 16 19:07:16.930198 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:16.930173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw" event={"ID":"2e31c156-f5ae-4b96-8e08-c9ec51203b64","Type":"ContainerStarted","Data":"ba571fa88e1144bfc7b7d5de987e29590a7b5a4ecb3727c76b1d4b08da0415cb"}
Sep 16 19:07:16.964394 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:16.964357    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9txmj" podStartSLOduration=33.111156762 podStartE2EDuration="34.964344446s" totalImagesPullingTime="1.853187684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:16.963647668 +0000 UTC m=+114.089452055" watchObservedRunningTime="2026-09-16 19:07:16.964344446 +0000 UTC m=+114.090148827"
Sep 16 19:07:16.996957 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:16.996923    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tffxw" podStartSLOduration=34.005896571 podStartE2EDuration="35.996910437s" totalImagesPullingTime="1.991013866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:16.994489953 +0000 UTC m=+114.120294341" watchObservedRunningTime="2026-09-16 19:07:16.996910437 +0000 UTC m=+114.122714825"
Sep 16 19:07:17.619321 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:17.619279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:07:17.622020 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:17.621993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86222f33-a533-4e08-ba27-ff62049bcba7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqz7\" (UID: \"86222f33-a533-4e08-ba27-ff62049bcba7\") " pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:07:17.651184 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:17.651160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbqz7"
Sep 16 19:07:17.934985 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:17.934954    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86222f33_a533_4e08_ba27_ff62049bcba7.slice/crio-26092868ba381ccb715b498db897cf1e3449b6a4973dc8d7db5d3bef70d51cf9 WatchSource:0}: Error finding container 26092868ba381ccb715b498db897cf1e3449b6a4973dc8d7db5d3bef70d51cf9: Status 404 returned error can't find the container with id 26092868ba381ccb715b498db897cf1e3449b6a4973dc8d7db5d3bef70d51cf9
Sep 16 19:07:17.936831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:17.936806    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbqz7"]
Sep 16 19:07:18.632135 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.632104    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"]
Sep 16 19:07:18.635035 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.635019    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"
Sep 16 19:07:18.659909 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.659882    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gqsc8\""
Sep 16 19:07:18.660012 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.659965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 19:07:18.660910 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.660893    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"]
Sep 16 19:07:18.727308 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.727283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c033d37c-40f8-46fb-aa0f-cfd1a54f25f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mnh5b\" (UID: \"c033d37c-40f8-46fb-aa0f-cfd1a54f25f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"
Sep 16 19:07:18.790969 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.790945    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bfc5889f8-4mdgf"]
Sep 16 19:07:18.794117 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.794098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:18.812605 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.812581    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 19:07:18.812697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.812606    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 19:07:18.827767 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.827748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c033d37c-40f8-46fb-aa0f-cfd1a54f25f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mnh5b\" (UID: \"c033d37c-40f8-46fb-aa0f-cfd1a54f25f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"
Sep 16 19:07:18.830171 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.830154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c033d37c-40f8-46fb-aa0f-cfd1a54f25f5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mnh5b\" (UID: \"c033d37c-40f8-46fb-aa0f-cfd1a54f25f5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"
Sep 16 19:07:18.840174 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.840154    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 19:07:18.840174 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.840168    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pps2s\""
Sep 16 19:07:18.891835 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.891768    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 19:07:18.899269 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.899248    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bfc5889f8-4mdgf"]
Sep 16 19:07:18.928974 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.928948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5p5m\" (UniqueName: \"kubernetes.io/projected/9d684e18-c88c-4221-9311-6ae86f09dbbf-kube-api-access-j5p5m\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:18.929076 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.928992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d684e18-c88c-4221-9311-6ae86f09dbbf-image-registry-private-configuration\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:18.929076 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.929014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d684e18-c88c-4221-9311-6ae86f09dbbf-installation-pull-secrets\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:18.929076 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.929031    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d684e18-c88c-4221-9311-6ae86f09dbbf-trusted-ca\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:18.929236 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.929086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d684e18-c88c-4221-9311-6ae86f09dbbf-registry-certificates\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:18.929236 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.929142    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d684e18-c88c-4221-9311-6ae86f09dbbf-ca-trust-extracted\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:18.929236 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.929192    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d684e18-c88c-4221-9311-6ae86f09dbbf-bound-sa-token\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:18.929356 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.929270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d684e18-c88c-4221-9311-6ae86f09dbbf-registry-tls\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:18.938219 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.938198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbqz7" event={"ID":"86222f33-a533-4e08-ba27-ff62049bcba7","Type":"ContainerStarted","Data":"730d61e20a95609e1319096a9679b8d242b82c29844b531f3319b6b0fab879f8"}
Sep 16 19:07:18.938526 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.938242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbqz7" event={"ID":"86222f33-a533-4e08-ba27-ff62049bcba7","Type":"ContainerStarted","Data":"26092868ba381ccb715b498db897cf1e3449b6a4973dc8d7db5d3bef70d51cf9"}
Sep 16 19:07:18.943316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:18.943301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"
Sep 16 19:07:19.030245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.029960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d684e18-c88c-4221-9311-6ae86f09dbbf-registry-tls\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.030245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.029996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5p5m\" (UniqueName: \"kubernetes.io/projected/9d684e18-c88c-4221-9311-6ae86f09dbbf-kube-api-access-j5p5m\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.030245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.030028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d684e18-c88c-4221-9311-6ae86f09dbbf-image-registry-private-configuration\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.030245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.030045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d684e18-c88c-4221-9311-6ae86f09dbbf-installation-pull-secrets\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.030245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.030060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d684e18-c88c-4221-9311-6ae86f09dbbf-trusted-ca\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.030245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.030089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d684e18-c88c-4221-9311-6ae86f09dbbf-registry-certificates\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.030245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.030120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d684e18-c88c-4221-9311-6ae86f09dbbf-ca-trust-extracted\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.030245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.030148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d684e18-c88c-4221-9311-6ae86f09dbbf-bound-sa-token\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.031306 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.031169    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d684e18-c88c-4221-9311-6ae86f09dbbf-trusted-ca\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.031562 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.031526    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d684e18-c88c-4221-9311-6ae86f09dbbf-ca-trust-extracted\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.032058 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.032034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d684e18-c88c-4221-9311-6ae86f09dbbf-registry-certificates\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.033964 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.033944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d684e18-c88c-4221-9311-6ae86f09dbbf-image-registry-private-configuration\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.034170 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.034155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d684e18-c88c-4221-9311-6ae86f09dbbf-installation-pull-secrets\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.034400 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.034380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d684e18-c88c-4221-9311-6ae86f09dbbf-registry-tls\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.100053 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.100028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5p5m\" (UniqueName: \"kubernetes.io/projected/9d684e18-c88c-4221-9311-6ae86f09dbbf-kube-api-access-j5p5m\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.110565 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.110547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d684e18-c88c-4221-9311-6ae86f09dbbf-bound-sa-token\") pod \"image-registry-6bfc5889f8-4mdgf\" (UID: \"9d684e18-c88c-4221-9311-6ae86f09dbbf\") " pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.270015 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:19.269997    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc033d37c_40f8_46fb_aa0f_cfd1a54f25f5.slice/crio-717413b27d12b72f12fc53e3329849e5eadfdc2d6687f82d2d1ee331c5c52579 WatchSource:0}: Error finding container 717413b27d12b72f12fc53e3329849e5eadfdc2d6687f82d2d1ee331c5c52579: Status 404 returned error can't find the container with id 717413b27d12b72f12fc53e3329849e5eadfdc2d6687f82d2d1ee331c5c52579
Sep 16 19:07:19.273862 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.273844    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"]
Sep 16 19:07:19.403957 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.403938    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.713556 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:19.713527    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d684e18_c88c_4221_9311_6ae86f09dbbf.slice/crio-e2430ce0576d0d8463180044e107794a264812fcacc834d42239f5f0a8dacea5 WatchSource:0}: Error finding container e2430ce0576d0d8463180044e107794a264812fcacc834d42239f5f0a8dacea5: Status 404 returned error can't find the container with id e2430ce0576d0d8463180044e107794a264812fcacc834d42239f5f0a8dacea5
Sep 16 19:07:19.714762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.714734    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bfc5889f8-4mdgf"]
Sep 16 19:07:19.944084 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.944055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf" event={"ID":"9d684e18-c88c-4221-9311-6ae86f09dbbf","Type":"ContainerStarted","Data":"600001866eebfe3bf802fc3f002b70bd357a3adfd86e6891462d1c25c3cc24f7"}
Sep 16 19:07:19.944514 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.944104    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf" event={"ID":"9d684e18-c88c-4221-9311-6ae86f09dbbf","Type":"ContainerStarted","Data":"e2430ce0576d0d8463180044e107794a264812fcacc834d42239f5f0a8dacea5"}
Sep 16 19:07:19.944514 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.944138    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:19.945603 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.945578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b" event={"ID":"c033d37c-40f8-46fb-aa0f-cfd1a54f25f5","Type":"ContainerStarted","Data":"717413b27d12b72f12fc53e3329849e5eadfdc2d6687f82d2d1ee331c5c52579"}
Sep 16 19:07:19.947747 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:19.947720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbqz7" event={"ID":"86222f33-a533-4e08-ba27-ff62049bcba7","Type":"ContainerStarted","Data":"ac94ec14418fd186e0afb8549b118e947bb9dfd45c35bfe9b6a3df5f061b3e48"}
Sep 16 19:07:21.956912 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:21.956871    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbqz7" event={"ID":"86222f33-a533-4e08-ba27-ff62049bcba7","Type":"ContainerStarted","Data":"7fdce35f7109aa2cd9cffac08dd2f764cfbea8d81527dd779825ecd556c62c87"}
Sep 16 19:07:21.963519 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:21.963484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b" event={"ID":"c033d37c-40f8-46fb-aa0f-cfd1a54f25f5","Type":"ContainerStarted","Data":"15c83d2b1044726755b05397b0bd0a3682be3f839ffd970bee55df0a7a566bb0"}
Sep 16 19:07:21.963748 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:21.963697    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"
Sep 16 19:07:21.968830 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:21.968807    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b"
Sep 16 19:07:22.012117 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:22.012077    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tbqz7" podStartSLOduration=34.023764671 podStartE2EDuration="37.012064369s" totalImagesPullingTime="2.988299698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:06:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:22.012005197 +0000 UTC m=+119.137809585" watchObservedRunningTime="2026-09-16 19:07:22.012064369 +0000 UTC m=+119.137868757"
Sep 16 19:07:22.012492 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:22.012466    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf" podStartSLOduration=4.01245972 podStartE2EDuration="4.01245972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:20.001050657 +0000 UTC m=+117.126855070" watchObservedRunningTime="2026-09-16 19:07:22.01245972 +0000 UTC m=+119.138264107"
Sep 16 19:07:22.042027 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:22.041990    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mnh5b" podStartSLOduration=2.24235613 podStartE2EDuration="4.041978354s" totalImagesPullingTime="1.799622224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:22.041847009 +0000 UTC m=+119.167651397" watchObservedRunningTime="2026-09-16 19:07:22.041978354 +0000 UTC m=+119.167782741"
Sep 16 19:07:27.693847 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.693814    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8rddf"]
Sep 16 19:07:27.696846 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.696823    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.706246 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.706229    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 19:07:27.706423 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.706388    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6rp57\""
Sep 16 19:07:27.713748 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.713733    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 19:07:27.713813 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.713744    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 19:07:27.713857 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.713735    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 19:07:27.891832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.891804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-textfile\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.891908 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.891846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-sys\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.891908 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.891869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-wtmp\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.891985 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.891922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.892019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.892004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-metrics-client-ca\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.892052 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.892034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-root\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.892084 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.892057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.892117 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.892096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-tls\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.892151 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.892125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4g5p\" (UniqueName: \"kubernetes.io/projected/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-kube-api-access-z4g5p\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992541 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-wtmp\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992541 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992700 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-metrics-client-ca\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992700 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-root\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992700 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992700 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-wtmp\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992700 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-root\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992915 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-tls\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992915 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4g5p\" (UniqueName: \"kubernetes.io/projected/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-kube-api-access-z4g5p\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992915 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992792    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-textfile\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992915 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-sys\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.992915 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.992894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-sys\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.993158 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.993144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-textfile\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.993245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.993223    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.993300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.993247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-metrics-client-ca\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.994857 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.994840    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:27.995015 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:27.994997    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-node-exporter-tls\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:28.013748 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.013723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4g5p\" (UniqueName: \"kubernetes.io/projected/a2e0b062-6abf-4157-9a36-dcc7c14c1cef-kube-api-access-z4g5p\") pod \"node-exporter-8rddf\" (UID: \"a2e0b062-6abf-4157-9a36-dcc7c14c1cef\") " pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:28.305270 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.305213    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8rddf"
Sep 16 19:07:28.314984 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:28.314960    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e0b062_6abf_4157_9a36_dcc7c14c1cef.slice/crio-0ca24f6dc270b2bb671b2034e77ef26915591861729e995fa4fd761ebc6adcb8 WatchSource:0}: Error finding container 0ca24f6dc270b2bb671b2034e77ef26915591861729e995fa4fd761ebc6adcb8: Status 404 returned error can't find the container with id 0ca24f6dc270b2bb671b2034e77ef26915591861729e995fa4fd761ebc6adcb8
Sep 16 19:07:28.778006 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.777980    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:07:28.781181 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.781160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.796484 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.796458    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:07:28.800060 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.800040    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 19:07:28.803741 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.803719    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 19:07:28.803852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.803760    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 19:07:28.803852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.803766    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 19:07:28.803852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.803770    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c664w\""
Sep 16 19:07:28.803852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.803770    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 19:07:28.803852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.803770    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 19:07:28.804728 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.804712    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 19:07:28.804728 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.804720    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 19:07:28.804863 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.804740    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 19:07:28.900678 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.900678 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.900841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.900841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900712    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.900841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.900841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-web-config\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.900841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-out\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.900841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.901117 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.901117 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chf5r\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-kube-api-access-chf5r\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.901117 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900898    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.901117 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.901117 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.900956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:28.995837 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:28.995800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rddf" event={"ID":"a2e0b062-6abf-4157-9a36-dcc7c14c1cef","Type":"ContainerStarted","Data":"0ca24f6dc270b2bb671b2034e77ef26915591861729e995fa4fd761ebc6adcb8"}
Sep 16 19:07:29.002161 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002286 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002286 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002286 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002286 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002509 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-web-config\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002509 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-out\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002509 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002509 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002509 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chf5r\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-kube-api-access-chf5r\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002749 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002749 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002749 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.002898 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.002763    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.003885 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.003765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.004120 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.004074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.005495 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.005450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.007112 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.007089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.007903 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.007855    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-out\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.008001 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.007960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.008427 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.008072    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.008427 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.008368    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.009533 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.009101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.009533 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.009496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.010276 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.010249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-web-config\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.019745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.019722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chf5r\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-kube-api-access-chf5r\") pod \"alertmanager-main-0\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.090975 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.090915    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:07:29.395455 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:29.393158    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac109ee_7d12_4d9f_b571_c24ffeed49c1.slice/crio-10004f61121f9c71ae4f03a1f34d4dfeb7a528f00b225f2c86be322a0b331767 WatchSource:0}: Error finding container 10004f61121f9c71ae4f03a1f34d4dfeb7a528f00b225f2c86be322a0b331767: Status 404 returned error can't find the container with id 10004f61121f9c71ae4f03a1f34d4dfeb7a528f00b225f2c86be322a0b331767
Sep 16 19:07:29.405013 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.404987    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:07:29.406184 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:29.406165    2583 scope.go:117] "RemoveContainer" containerID="ceb6c81543d4355b53f1fb5da1fe7a658effef516eeb00398fd864d8210cbfb1"
Sep 16 19:07:29.406514 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:07:29.406362    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-btngq_openshift-console-operator(1a7447f7-35b1-45f4-af73-2ebf91399392)\"" pod="openshift-console-operator/console-operator-696884fb9c-btngq" podUID="1a7447f7-35b1-45f4-af73-2ebf91399392"
Sep 16 19:07:30.000226 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:30.000189    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerStarted","Data":"10004f61121f9c71ae4f03a1f34d4dfeb7a528f00b225f2c86be322a0b331767"}
Sep 16 19:07:30.001986 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:30.001958    2583 generic.go:356] "Generic (PLEG): container finished" podID="a2e0b062-6abf-4157-9a36-dcc7c14c1cef" containerID="eb264accc865d927079bd1e1fe1f0c9e5a0de42f325142b970209ab21c4aff34" exitCode=0
Sep 16 19:07:30.002197 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:30.002000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rddf" event={"ID":"a2e0b062-6abf-4157-9a36-dcc7c14c1cef","Type":"ContainerDied","Data":"eb264accc865d927079bd1e1fe1f0c9e5a0de42f325142b970209ab21c4aff34"}
Sep 16 19:07:31.007671 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:31.007637    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerID="5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d" exitCode=0
Sep 16 19:07:31.008028 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:31.007726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerDied","Data":"5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d"}
Sep 16 19:07:31.010341 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:31.010315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rddf" event={"ID":"a2e0b062-6abf-4157-9a36-dcc7c14c1cef","Type":"ContainerStarted","Data":"ea1ab85af794caa88a01fb19642e54bb147ca93907784b17efb7b8ebc6cb444a"}
Sep 16 19:07:31.010456 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:31.010371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rddf" event={"ID":"a2e0b062-6abf-4157-9a36-dcc7c14c1cef","Type":"ContainerStarted","Data":"b2505528419e13b79086d3fa1cc0e78e8bf63b20ecd1af3c40cc465f5ff4dc87"}
Sep 16 19:07:32.069602 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.069549    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8rddf" podStartSLOduration=3.897867095 podStartE2EDuration="5.069533907s" totalImagesPullingTime="1.102022809s" totalInitContainerRuntime="69.644003ms" isStatefulPod=true podCreationTimestamp="2026-09-16 19:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:31.127815902 +0000 UTC m=+128.253620300" watchObservedRunningTime="2026-09-16 19:07:32.069533907 +0000 UTC m=+129.195338295"
Sep 16 19:07:32.070694 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.070672    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9487fbf4f-2rv8m"]
Sep 16 19:07:32.073125 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.073105    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.093188 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.093162    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-feaafs8td2spd\""
Sep 16 19:07:32.093280 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.093222    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 19:07:32.093587 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.093569    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 19:07:32.093671 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.093615    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 19:07:32.093671 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.093641    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 19:07:32.093743 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.093731    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jzqq4\""
Sep 16 19:07:32.101586 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.101566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9487fbf4f-2rv8m"]
Sep 16 19:07:32.128873 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.128853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a1274f6-49a0-4959-b46d-8c6bf82980c7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.128964 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.128887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:07:32.128964 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.128905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a1274f6-49a0-4959-b46d-8c6bf82980c7-client-ca-bundle\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.128964 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.128929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a1274f6-49a0-4959-b46d-8c6bf82980c7-metrics-server-audit-profiles\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.129071 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.128971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a1274f6-49a0-4959-b46d-8c6bf82980c7-secret-metrics-server-client-certs\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.129071 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.129014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a1274f6-49a0-4959-b46d-8c6bf82980c7-secret-metrics-server-tls\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.129071 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.129058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncp8\" (UniqueName: \"kubernetes.io/projected/0a1274f6-49a0-4959-b46d-8c6bf82980c7-kube-api-access-7ncp8\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.129180 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.129082    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a1274f6-49a0-4959-b46d-8c6bf82980c7-audit-log\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.131233 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.131210    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14abbfab-5c2d-4416-a29a-939762f17972-metrics-certs\") pod \"network-metrics-daemon-wvcsr\" (UID: \"14abbfab-5c2d-4416-a29a-939762f17972\") " pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:07:32.230010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.229983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a1274f6-49a0-4959-b46d-8c6bf82980c7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.230118 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.230022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a1274f6-49a0-4959-b46d-8c6bf82980c7-client-ca-bundle\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.230118 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.230050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a1274f6-49a0-4959-b46d-8c6bf82980c7-metrics-server-audit-profiles\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.230118 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.230079    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a1274f6-49a0-4959-b46d-8c6bf82980c7-secret-metrics-server-client-certs\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.230284 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.230122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a1274f6-49a0-4959-b46d-8c6bf82980c7-secret-metrics-server-tls\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.230284 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.230183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncp8\" (UniqueName: \"kubernetes.io/projected/0a1274f6-49a0-4959-b46d-8c6bf82980c7-kube-api-access-7ncp8\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.230284 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.230221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a1274f6-49a0-4959-b46d-8c6bf82980c7-audit-log\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.230764 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.230743    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a1274f6-49a0-4959-b46d-8c6bf82980c7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.230919 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.230748    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a1274f6-49a0-4959-b46d-8c6bf82980c7-audit-log\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.231250 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.231185    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a1274f6-49a0-4959-b46d-8c6bf82980c7-metrics-server-audit-profiles\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.231605 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.231512    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjb6w\""
Sep 16 19:07:32.233112 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.232677    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a1274f6-49a0-4959-b46d-8c6bf82980c7-secret-metrics-server-client-certs\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.233112 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.232730    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvcsr"
Sep 16 19:07:32.233112 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.232766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a1274f6-49a0-4959-b46d-8c6bf82980c7-client-ca-bundle\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.233372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.233272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a1274f6-49a0-4959-b46d-8c6bf82980c7-secret-metrics-server-tls\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.251088 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.251063    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncp8\" (UniqueName: \"kubernetes.io/projected/0a1274f6-49a0-4959-b46d-8c6bf82980c7-kube-api-access-7ncp8\") pod \"metrics-server-9487fbf4f-2rv8m\" (UID: \"0a1274f6-49a0-4959-b46d-8c6bf82980c7\") " pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.325191 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.325166    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"]
Sep 16 19:07:32.327211 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.327193    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:32.336966 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.336941    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 19:07:32.337076 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.337052    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5dm8q\""
Sep 16 19:07:32.350241 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.350184    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"]
Sep 16 19:07:32.383532 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.383511    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:32.432161 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.432062    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmgq\" (UniqueName: \"kubernetes.io/projected/cca56bad-aaaf-4439-8c80-5bf60dd2dfa0-kube-api-access-dwmgq\") pod \"monitoring-plugin-5d85778b9d-r4xg7\" (UID: \"cca56bad-aaaf-4439-8c80-5bf60dd2dfa0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:32.432161 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.432115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cca56bad-aaaf-4439-8c80-5bf60dd2dfa0-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-r4xg7\" (UID: \"cca56bad-aaaf-4439-8c80-5bf60dd2dfa0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:32.523088 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.523066    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wvcsr"]
Sep 16 19:07:32.523889 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:32.523869    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14abbfab_5c2d_4416_a29a_939762f17972.slice/crio-c5388ba6a61716988a940499fd9af9b44c9ba2ba88f317e926f23578a2885efb WatchSource:0}: Error finding container c5388ba6a61716988a940499fd9af9b44c9ba2ba88f317e926f23578a2885efb: Status 404 returned error can't find the container with id c5388ba6a61716988a940499fd9af9b44c9ba2ba88f317e926f23578a2885efb
Sep 16 19:07:32.533395 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.533373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmgq\" (UniqueName: \"kubernetes.io/projected/cca56bad-aaaf-4439-8c80-5bf60dd2dfa0-kube-api-access-dwmgq\") pod \"monitoring-plugin-5d85778b9d-r4xg7\" (UID: \"cca56bad-aaaf-4439-8c80-5bf60dd2dfa0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:32.533512 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.533470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cca56bad-aaaf-4439-8c80-5bf60dd2dfa0-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-r4xg7\" (UID: \"cca56bad-aaaf-4439-8c80-5bf60dd2dfa0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:32.535817 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.535794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cca56bad-aaaf-4439-8c80-5bf60dd2dfa0-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-r4xg7\" (UID: \"cca56bad-aaaf-4439-8c80-5bf60dd2dfa0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:32.551428 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.551392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmgq\" (UniqueName: \"kubernetes.io/projected/cca56bad-aaaf-4439-8c80-5bf60dd2dfa0-kube-api-access-dwmgq\") pod \"monitoring-plugin-5d85778b9d-r4xg7\" (UID: \"cca56bad-aaaf-4439-8c80-5bf60dd2dfa0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:32.636598 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.636578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:32.668476 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:32.668447    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1274f6_49a0_4959_b46d_8c6bf82980c7.slice/crio-4054c3012f1702abac9720901219ab739efc1967f773bcf32561379dd8cf7505 WatchSource:0}: Error finding container 4054c3012f1702abac9720901219ab739efc1967f773bcf32561379dd8cf7505: Status 404 returned error can't find the container with id 4054c3012f1702abac9720901219ab739efc1967f773bcf32561379dd8cf7505
Sep 16 19:07:32.677788 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.677763    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9487fbf4f-2rv8m"]
Sep 16 19:07:32.919590 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:32.919545    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca56bad_aaaf_4439_8c80_5bf60dd2dfa0.slice/crio-dfc2faeac81d644244ba7ed73b02b1f6256ecd7e294ab2d31f35efcb49b4974d WatchSource:0}: Error finding container dfc2faeac81d644244ba7ed73b02b1f6256ecd7e294ab2d31f35efcb49b4974d: Status 404 returned error can't find the container with id dfc2faeac81d644244ba7ed73b02b1f6256ecd7e294ab2d31f35efcb49b4974d
Sep 16 19:07:32.922192 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:32.922161    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"]
Sep 16 19:07:33.026231 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:33.026182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerStarted","Data":"c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5"}
Sep 16 19:07:33.026403 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:33.026276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerStarted","Data":"ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3"}
Sep 16 19:07:33.026403 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:33.026322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerStarted","Data":"3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0"}
Sep 16 19:07:33.026403 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:33.026362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerStarted","Data":"b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f"}
Sep 16 19:07:33.026403 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:33.026397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerStarted","Data":"5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7"}
Sep 16 19:07:33.027756 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:33.027725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7" event={"ID":"cca56bad-aaaf-4439-8c80-5bf60dd2dfa0","Type":"ContainerStarted","Data":"dfc2faeac81d644244ba7ed73b02b1f6256ecd7e294ab2d31f35efcb49b4974d"}
Sep 16 19:07:33.029138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:33.029102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wvcsr" event={"ID":"14abbfab-5c2d-4416-a29a-939762f17972","Type":"ContainerStarted","Data":"c5388ba6a61716988a940499fd9af9b44c9ba2ba88f317e926f23578a2885efb"}
Sep 16 19:07:33.030607 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:33.030569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m" event={"ID":"0a1274f6-49a0-4959-b46d-8c6bf82980c7","Type":"ContainerStarted","Data":"4054c3012f1702abac9720901219ab739efc1967f773bcf32561379dd8cf7505"}
Sep 16 19:07:34.042367 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.042327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerStarted","Data":"f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47"}
Sep 16 19:07:34.156148 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.156092    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.991125382 podStartE2EDuration="6.156070803s" totalImagesPullingTime="3.146716685s" totalInitContainerRuntime="18.228736ms" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:34.137812439 +0000 UTC m=+131.263616828" watchObservedRunningTime="2026-09-16 19:07:34.156070803 +0000 UTC m=+131.281875192"
Sep 16 19:07:34.345741 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.345666    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:07:34.350137 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.350115    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.374618 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.374492    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 19:07:34.374618 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.374514    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 19:07:34.374618 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.374523    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 19:07:34.380654 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.380635    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gvjwh\""
Sep 16 19:07:34.380778 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.380747    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f0dted9i830jh\""
Sep 16 19:07:34.383110 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.383089    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 19:07:34.383213 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.383110    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 19:07:34.383213 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.383089    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 19:07:34.396004 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.395984    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 19:07:34.400113 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.400091    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 19:07:34.402703 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.402680    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 19:07:34.402816 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.402686    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 19:07:34.402816 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.402741    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 19:07:34.402816 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.402778    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 19:07:34.451054 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-config-out\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451149 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-web-config\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451149 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-config\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451246 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451142    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451246 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451246 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451366 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451366 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451366 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451366 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451539 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451539 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451539 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451539 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451810 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451810 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnzz\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-kube-api-access-zcnzz\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.451810 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.451768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.473660 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.473640    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 19:07:34.484489 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.484465    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:07:34.503123 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.503101    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 19:07:34.552966 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.552944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553065 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.552979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553065 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.552995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553065 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553209 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553209 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnzz\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-kube-api-access-zcnzz\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-config-out\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-web-config\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-config\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.553964 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.553923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.554579 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.554231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.554579 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.554349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.554579 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.554532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.554979 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.554908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.556818 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.556676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.558393 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.558090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-web-config\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.558393 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.558108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.558393 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.558130    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.558622 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.558493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-config\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.558622 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.558514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.559153 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.558865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.559153 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.559001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.559153 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.559111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.559341 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.559201    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.559433 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.559394    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.559502 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.559402    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-config-out\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.560593 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.560570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.560712 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.560696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.574338 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.574317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnzz\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-kube-api-access-zcnzz\") pod \"prometheus-k8s-0\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.661966 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:34.661937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:34.989916 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:07:34.989888    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71408e6a_93ab_432e_83af_985f5896f931.slice/crio-da9a91b2eedf74106bfdedc6e0f755cca977f4c9d4af7d400b73317d061cfa68 WatchSource:0}: Error finding container da9a91b2eedf74106bfdedc6e0f755cca977f4c9d4af7d400b73317d061cfa68: Status 404 returned error can't find the container with id da9a91b2eedf74106bfdedc6e0f755cca977f4c9d4af7d400b73317d061cfa68
Sep 16 19:07:35.007702 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.007682    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:07:35.048090 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.048061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7" event={"ID":"cca56bad-aaaf-4439-8c80-5bf60dd2dfa0","Type":"ContainerStarted","Data":"c10ad50759a303829a11cfe912302c9db3d09188f531fbeca6e2f02d50d09bcd"}
Sep 16 19:07:35.048468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.048123    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:35.050091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.050047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m" event={"ID":"0a1274f6-49a0-4959-b46d-8c6bf82980c7","Type":"ContainerStarted","Data":"ac239deb7afe06f27be0af2fde41815a68b93f16c8036abc8d9803f99b1f7b35"}
Sep 16 19:07:35.051848 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.051823    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerStarted","Data":"da9a91b2eedf74106bfdedc6e0f755cca977f4c9d4af7d400b73317d061cfa68"}
Sep 16 19:07:35.054361 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.053907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wvcsr" event={"ID":"14abbfab-5c2d-4416-a29a-939762f17972","Type":"ContainerStarted","Data":"46fe44eb161cf531fb5713ac844c958d3b47606210a118d8f40ab8230c6a1551"}
Sep 16 19:07:35.054361 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.053977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wvcsr" event={"ID":"14abbfab-5c2d-4416-a29a-939762f17972","Type":"ContainerStarted","Data":"20bff6e0e244cdedefc2e197963a65a72da37b4f046b9f90a00945a0e9f94bc1"}
Sep 16 19:07:35.054579 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.054564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7"
Sep 16 19:07:35.087941 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.087902    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-r4xg7" podStartSLOduration=1.363512455 podStartE2EDuration="3.087887571s" totalImagesPullingTime="1.724375116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:35.086910161 +0000 UTC m=+132.212714548" watchObservedRunningTime="2026-09-16 19:07:35.087887571 +0000 UTC m=+132.213691958"
Sep 16 19:07:35.161462 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:35.161390    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m" podStartSLOduration=1.188624642 podStartE2EDuration="3.161371687s" totalImagesPullingTime="1.972747045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:35.153301507 +0000 UTC m=+132.279105897" watchObservedRunningTime="2026-09-16 19:07:35.161371687 +0000 UTC m=+132.287176076"
Sep 16 19:07:36.059909 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:36.059871    2583 generic.go:356] "Generic (PLEG): container finished" podID="71408e6a-93ab-432e-83af-985f5896f931" containerID="deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06" exitCode=0
Sep 16 19:07:36.060329 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:36.059983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerDied","Data":"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"}
Sep 16 19:07:36.140065 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:36.140015    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wvcsr" podStartSLOduration=131.022732174 podStartE2EDuration="2m13.13999943s" totalImagesPullingTime="2.117267256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:35.185585409 +0000 UTC m=+132.311389798" watchObservedRunningTime="2026-09-16 19:07:36.13999943 +0000 UTC m=+133.265803817"
Sep 16 19:07:39.408957 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:39.408926    2583 patch_prober.go:28] interesting pod/image-registry-6bfc5889f8-4mdgf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 19:07:39.409285 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:39.408989    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf" podUID="9d684e18-c88c-4221-9311-6ae86f09dbbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:07:40.079786 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:40.079756    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerStarted","Data":"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"}
Sep 16 19:07:40.079956 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:40.079820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerStarted","Data":"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"}
Sep 16 19:07:40.956297 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:40.956265    2583 patch_prober.go:28] interesting pod/image-registry-6bfc5889f8-4mdgf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 19:07:40.956644 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:40.956321    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf" podUID="9d684e18-c88c-4221-9311-6ae86f09dbbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:07:41.086020 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:41.085996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerStarted","Data":"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"}
Sep 16 19:07:42.093716 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:42.093671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerStarted","Data":"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"}
Sep 16 19:07:42.094044 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:42.093748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerStarted","Data":"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"}
Sep 16 19:07:42.094044 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:42.093788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerStarted","Data":"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"}
Sep 16 19:07:42.171849 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:42.171804    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.359808968 podStartE2EDuration="8.171790924s" totalImagesPullingTime="4.792970205s" totalInitContainerRuntime="19.011751ms" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:42.170143929 +0000 UTC m=+139.295948317" watchObservedRunningTime="2026-09-16 19:07:42.171790924 +0000 UTC m=+139.297595312"
Sep 16 19:07:43.409535 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:43.409508    2583 scope.go:117] "RemoveContainer" containerID="ceb6c81543d4355b53f1fb5da1fe7a658effef516eeb00398fd864d8210cbfb1"
Sep 16 19:07:44.107354 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:44.107330    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:07:44.107519 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:44.107452    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-btngq" event={"ID":"1a7447f7-35b1-45f4-af73-2ebf91399392","Type":"ContainerStarted","Data":"2c246dc0adb1b6fbab58496f942b9d63746be7835d29d0bda6c8080ba83d3a90"}
Sep 16 19:07:44.107774 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:44.107740    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:07:44.112121 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:44.112102    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-btngq"
Sep 16 19:07:44.662725 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:44.662675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:07:49.408680 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:49.408637    2583 patch_prober.go:28] interesting pod/image-registry-6bfc5889f8-4mdgf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 19:07:49.409229 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:49.408708    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf" podUID="9d684e18-c88c-4221-9311-6ae86f09dbbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:07:50.955687 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:50.955652    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bfc5889f8-4mdgf"
Sep 16 19:07:52.383962 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:52.383926    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:07:52.384377 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:07:52.384011    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:08:00.246283 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:00.246246    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jnthf" podUID="cd6974b2-a024-4ebe-930b-e3df0c1ba35b"
Sep 16 19:08:01.175845 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:01.175812    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jnthf"
Sep 16 19:08:03.185984 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:03.185940    2583 generic.go:356] "Generic (PLEG): container finished" podID="0ef80565-c6f2-487c-9086-3f98f46e17a4" containerID="d5e1b775a5ba61da99907c011a784d4828cee147da21e5bc393ae431169e99fb" exitCode=0
Sep 16 19:08:03.186516 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:03.186023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn" event={"ID":"0ef80565-c6f2-487c-9086-3f98f46e17a4","Type":"ContainerDied","Data":"d5e1b775a5ba61da99907c011a784d4828cee147da21e5bc393ae431169e99fb"}
Sep 16 19:08:03.186602 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:03.186523    2583 scope.go:117] "RemoveContainer" containerID="d5e1b775a5ba61da99907c011a784d4828cee147da21e5bc393ae431169e99fb"
Sep 16 19:08:04.191738 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:04.191710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rq8bn" event={"ID":"0ef80565-c6f2-487c-9086-3f98f46e17a4","Type":"ContainerStarted","Data":"b9a679dd4d9a28ab02d3ac4a0d488988630b670bd5b00f184c56fcabf1bf496c"}
Sep 16 19:08:05.189110 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:05.189085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:08:05.191374 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:05.191344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd6974b2-a024-4ebe-930b-e3df0c1ba35b-metrics-tls\") pod \"dns-default-jnthf\" (UID: \"cd6974b2-a024-4ebe-930b-e3df0c1ba35b\") " pod="openshift-dns/dns-default-jnthf"
Sep 16 19:08:05.200494 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:05.200467    2583 generic.go:356] "Generic (PLEG): container finished" podID="a24fdd13-7ba7-45e6-9d0a-5d52ff23188c" containerID="028ee16fda204650513f6ca1e04f8adb5666f156c33c832c83ee22c27ab70c21" exitCode=0
Sep 16 19:08:05.200781 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:05.200524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv" event={"ID":"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c","Type":"ContainerDied","Data":"028ee16fda204650513f6ca1e04f8adb5666f156c33c832c83ee22c27ab70c21"}
Sep 16 19:08:05.200829 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:05.200785    2583 scope.go:117] "RemoveContainer" containerID="028ee16fda204650513f6ca1e04f8adb5666f156c33c832c83ee22c27ab70c21"
Sep 16 19:08:05.406871 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:05.402321    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pvzq\""
Sep 16 19:08:05.406871 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:05.406855    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jnthf"
Sep 16 19:08:05.694337 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:08:05.694310    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6974b2_a024_4ebe_930b_e3df0c1ba35b.slice/crio-0594f54587a6728812678feaac793f2fba906cc575209fc1c6cf63503ad78009 WatchSource:0}: Error finding container 0594f54587a6728812678feaac793f2fba906cc575209fc1c6cf63503ad78009: Status 404 returned error can't find the container with id 0594f54587a6728812678feaac793f2fba906cc575209fc1c6cf63503ad78009
Sep 16 19:08:05.696027 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:05.695987    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jnthf"]
Sep 16 19:08:06.206294 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:06.206259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l82nv" event={"ID":"a24fdd13-7ba7-45e6-9d0a-5d52ff23188c","Type":"ContainerStarted","Data":"ba803b28e2631f7d105437528eb1d1b19dc3efac7e05ba99f135a690b3d74dd6"}
Sep 16 19:08:06.207620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:06.207600    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnthf" event={"ID":"cd6974b2-a024-4ebe-930b-e3df0c1ba35b","Type":"ContainerStarted","Data":"0594f54587a6728812678feaac793f2fba906cc575209fc1c6cf63503ad78009"}
Sep 16 19:08:08.218402 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:08.218353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnthf" event={"ID":"cd6974b2-a024-4ebe-930b-e3df0c1ba35b","Type":"ContainerStarted","Data":"860d86e639dc396510bce018bc8857c8312f0c355bc4fe53f95b778cf9dd9f3d"}
Sep 16 19:08:08.218800 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:08.218456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnthf" event={"ID":"cd6974b2-a024-4ebe-930b-e3df0c1ba35b","Type":"ContainerStarted","Data":"2d2cbcf55f40b1aaaf44268834783222c1e7f8d3d61b3f5101342296d74d64cf"}
Sep 16 19:08:08.218800 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:08.218577    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jnthf"
Sep 16 19:08:08.260138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:08.260090    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jnthf" podStartSLOduration=129.441805078 podStartE2EDuration="2m11.260075854s" totalImagesPullingTime="1.818270776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:08.259878793 +0000 UTC m=+165.385683209" watchObservedRunningTime="2026-09-16 19:08:08.260075854 +0000 UTC m=+165.385880242"
Sep 16 19:08:12.388916 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:12.388891    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:08:12.392544 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:12.392523    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9487fbf4f-2rv8m"
Sep 16 19:08:18.224038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:18.223970    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jnthf"
Sep 16 19:08:34.663103 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:34.663069    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:34.682983 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:34.682962    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:35.337152 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:35.337131    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:48.444302 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:48.444256    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:08:48.444790 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:48.444598    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="alertmanager" containerID="cri-o://5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7" gracePeriod=120
Sep 16 19:08:48.444790 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:48.444628    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy-metric" containerID="cri-o://c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5" gracePeriod=120
Sep 16 19:08:48.444790 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:48.444681    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="config-reloader" containerID="cri-o://b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f" gracePeriod=120
Sep 16 19:08:48.444790 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:48.444705    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="prom-label-proxy" containerID="cri-o://f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47" gracePeriod=120
Sep 16 19:08:48.444790 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:48.444732    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy" containerID="cri-o://ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3" gracePeriod=120
Sep 16 19:08:48.445104 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:48.444651    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy-web" containerID="cri-o://3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0" gracePeriod=120
Sep 16 19:08:49.379835 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.379805    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerID="f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47" exitCode=0
Sep 16 19:08:49.379835 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.379829    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerID="ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3" exitCode=0
Sep 16 19:08:49.379835 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.379835    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerID="b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f" exitCode=0
Sep 16 19:08:49.379835 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.379840    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerID="5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7" exitCode=0
Sep 16 19:08:49.380095 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.379877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerDied","Data":"f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47"}
Sep 16 19:08:49.380095 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.379936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerDied","Data":"ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3"}
Sep 16 19:08:49.380095 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.379963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerDied","Data":"b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f"}
Sep 16 19:08:49.380095 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.379986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerDied","Data":"5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7"}
Sep 16 19:08:49.681657 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.681633    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:49.712342 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.712310    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-volume\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.712473 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.712358    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-tls-assets\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.712473 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.712404    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.712473 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.712449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-web-config\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.712774    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-metrics-client-ca\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.712841    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-out\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.712938    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-trusted-ca-bundle\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.712988    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-cluster-tls-config\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.713016    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.713056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chf5r\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-kube-api-access-chf5r\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.713199    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-main-db\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.713289    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.713330    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-main-tls\") pod \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\" (UID: \"1ac109ee-7d12-4d9f-b571-c24ffeed49c1\") "
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.713548    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.713597    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.713681    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-metrics-client-ca\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.713832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.713701    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.715316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.715233    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:49.715316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.715331    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:49.716240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.716204    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-volume" (OuterVolumeSpecName: "config-volume") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:49.716521 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.716495    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:49.716797 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.716715    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:49.716908 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.716844    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-out" (OuterVolumeSpecName: "config-out") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:49.717834 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.717799    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:49.718103 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.718075    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:49.719255 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.719228    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-kube-api-access-chf5r" (OuterVolumeSpecName: "kube-api-access-chf5r") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "kube-api-access-chf5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:49.721501 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.721475    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:49.727858 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.727840    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-web-config" (OuterVolumeSpecName: "web-config") pod "1ac109ee-7d12-4d9f-b571-c24ffeed49c1" (UID: "1ac109ee-7d12-4d9f-b571-c24ffeed49c1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:49.814675 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814642    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-cluster-tls-config\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814679    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814696    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chf5r\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-kube-api-access-chf5r\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814712    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-alertmanager-main-db\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814727    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814742    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-main-tls\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814755    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-volume\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814768    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-tls-assets\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814781    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814794    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-web-config\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:49.814823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:49.814806    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ac109ee-7d12-4d9f-b571-c24ffeed49c1-config-out\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:50.386504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.386474    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerID="c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5" exitCode=0
Sep 16 19:08:50.386504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.386500    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerID="3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0" exitCode=0
Sep 16 19:08:50.386712 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.386530    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerDied","Data":"c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5"}
Sep 16 19:08:50.386712 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.386561    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.386712 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.386609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerDied","Data":"3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0"}
Sep 16 19:08:50.386712 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.386639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ac109ee-7d12-4d9f-b571-c24ffeed49c1","Type":"ContainerDied","Data":"10004f61121f9c71ae4f03a1f34d4dfeb7a528f00b225f2c86be322a0b331767"}
Sep 16 19:08:50.387657 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.386682    2583 scope.go:117] "RemoveContainer" containerID="f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47"
Sep 16 19:08:50.395787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.395769    2583 scope.go:117] "RemoveContainer" containerID="c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5"
Sep 16 19:08:50.402691 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.402670    2583 scope.go:117] "RemoveContainer" containerID="ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3"
Sep 16 19:08:50.409585 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.409560    2583 scope.go:117] "RemoveContainer" containerID="3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0"
Sep 16 19:08:50.416128 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.416111    2583 scope.go:117] "RemoveContainer" containerID="b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f"
Sep 16 19:08:50.423528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.423512    2583 scope.go:117] "RemoveContainer" containerID="5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7"
Sep 16 19:08:50.430316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.430300    2583 scope.go:117] "RemoveContainer" containerID="5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d"
Sep 16 19:08:50.436972 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.436954    2583 scope.go:117] "RemoveContainer" containerID="f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47"
Sep 16 19:08:50.437279 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:50.437259    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47\": container with ID starting with f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47 not found: ID does not exist" containerID="f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47"
Sep 16 19:08:50.437366 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.437288    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47"} err="failed to get container status \"f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47\": rpc error: code = NotFound desc = could not find container \"f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47\": container with ID starting with f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47 not found: ID does not exist"
Sep 16 19:08:50.437366 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.437318    2583 scope.go:117] "RemoveContainer" containerID="c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5"
Sep 16 19:08:50.437692 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:50.437666    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5\": container with ID starting with c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5 not found: ID does not exist" containerID="c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5"
Sep 16 19:08:50.437767 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.437697    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5"} err="failed to get container status \"c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5\": rpc error: code = NotFound desc = could not find container \"c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5\": container with ID starting with c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5 not found: ID does not exist"
Sep 16 19:08:50.437767 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.437718    2583 scope.go:117] "RemoveContainer" containerID="ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3"
Sep 16 19:08:50.438013 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:50.437992    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3\": container with ID starting with ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3 not found: ID does not exist" containerID="ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3"
Sep 16 19:08:50.438061 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.438018    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3"} err="failed to get container status \"ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3\": rpc error: code = NotFound desc = could not find container \"ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3\": container with ID starting with ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3 not found: ID does not exist"
Sep 16 19:08:50.438061 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.438038    2583 scope.go:117] "RemoveContainer" containerID="3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0"
Sep 16 19:08:50.438329 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:50.438299    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0\": container with ID starting with 3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0 not found: ID does not exist" containerID="3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0"
Sep 16 19:08:50.438373 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.438333    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0"} err="failed to get container status \"3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0\": rpc error: code = NotFound desc = could not find container \"3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0\": container with ID starting with 3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0 not found: ID does not exist"
Sep 16 19:08:50.438373 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.438347    2583 scope.go:117] "RemoveContainer" containerID="b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f"
Sep 16 19:08:50.438629 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:50.438611    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f\": container with ID starting with b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f not found: ID does not exist" containerID="b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f"
Sep 16 19:08:50.438675 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.438634    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f"} err="failed to get container status \"b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f\": rpc error: code = NotFound desc = could not find container \"b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f\": container with ID starting with b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f not found: ID does not exist"
Sep 16 19:08:50.438675 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.438648    2583 scope.go:117] "RemoveContainer" containerID="5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7"
Sep 16 19:08:50.438878 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:50.438864    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7\": container with ID starting with 5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7 not found: ID does not exist" containerID="5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7"
Sep 16 19:08:50.438923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.438882    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7"} err="failed to get container status \"5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7\": rpc error: code = NotFound desc = could not find container \"5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7\": container with ID starting with 5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7 not found: ID does not exist"
Sep 16 19:08:50.438923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.438894    2583 scope.go:117] "RemoveContainer" containerID="5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d"
Sep 16 19:08:50.439148 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:50.439132    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d\": container with ID starting with 5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d not found: ID does not exist" containerID="5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d"
Sep 16 19:08:50.439196 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.439151    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d"} err="failed to get container status \"5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d\": rpc error: code = NotFound desc = could not find container \"5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d\": container with ID starting with 5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d not found: ID does not exist"
Sep 16 19:08:50.439196 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.439164    2583 scope.go:117] "RemoveContainer" containerID="f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47"
Sep 16 19:08:50.439375 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.439360    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47"} err="failed to get container status \"f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47\": rpc error: code = NotFound desc = could not find container \"f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47\": container with ID starting with f69b76a24bfc07361bc2ef85529c4d01b5bca03358d114d9aa0cfff8c10b2e47 not found: ID does not exist"
Sep 16 19:08:50.439443 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.439376    2583 scope.go:117] "RemoveContainer" containerID="c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5"
Sep 16 19:08:50.439603 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.439588    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5"} err="failed to get container status \"c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5\": rpc error: code = NotFound desc = could not find container \"c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5\": container with ID starting with c0807bae994e926a32b8dd7fa89806aa8d3d636feea3d8bc9cdfa9c8ae5112f5 not found: ID does not exist"
Sep 16 19:08:50.439651 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.439604    2583 scope.go:117] "RemoveContainer" containerID="ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3"
Sep 16 19:08:50.439786 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.439770    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3"} err="failed to get container status \"ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3\": rpc error: code = NotFound desc = could not find container \"ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3\": container with ID starting with ec300b233f4103ebc6a03b61429fc32da9a1d0499544addf2d1816fa693c0aa3 not found: ID does not exist"
Sep 16 19:08:50.439843 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.439787    2583 scope.go:117] "RemoveContainer" containerID="3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0"
Sep 16 19:08:50.440008 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.439991    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0"} err="failed to get container status \"3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0\": rpc error: code = NotFound desc = could not find container \"3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0\": container with ID starting with 3a92152a232c1ea562fd436ab76e6690f40288e5e5bfa3ebde10475a368a58f0 not found: ID does not exist"
Sep 16 19:08:50.440053 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.440009    2583 scope.go:117] "RemoveContainer" containerID="b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f"
Sep 16 19:08:50.440229 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.440207    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f"} err="failed to get container status \"b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f\": rpc error: code = NotFound desc = could not find container \"b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f\": container with ID starting with b81474d3f11ba3915a308c4370c81a6734bbcbf8740b2bae0ddd4a5f3c1fd94f not found: ID does not exist"
Sep 16 19:08:50.440229 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.440225    2583 scope.go:117] "RemoveContainer" containerID="5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7"
Sep 16 19:08:50.440583 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.440562    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7"} err="failed to get container status \"5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7\": rpc error: code = NotFound desc = could not find container \"5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7\": container with ID starting with 5245250cd06b7ff03c28d3786cbdcf1294dfae82c44192defc0500216d6a0ec7 not found: ID does not exist"
Sep 16 19:08:50.440583 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.440582    2583 scope.go:117] "RemoveContainer" containerID="5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d"
Sep 16 19:08:50.440847 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.440821    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d"} err="failed to get container status \"5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d\": rpc error: code = NotFound desc = could not find container \"5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d\": container with ID starting with 5c3472d96df409c4c521407722022723b977492b727b3067c3dc4a489cd3a22d not found: ID does not exist"
Sep 16 19:08:50.446547 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.446512    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:08:50.461464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.461438    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:08:50.499202 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499175    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:08:50.499492 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499480    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="config-reloader"
Sep 16 19:08:50.499542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499494    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="config-reloader"
Sep 16 19:08:50.499542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499503    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="alertmanager"
Sep 16 19:08:50.499542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499508    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="alertmanager"
Sep 16 19:08:50.499542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499517    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy-web"
Sep 16 19:08:50.499542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499522    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy-web"
Sep 16 19:08:50.499542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499531    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy-metric"
Sep 16 19:08:50.499542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499536    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy-metric"
Sep 16 19:08:50.499542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499545    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="init-config-reloader"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499550    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="init-config-reloader"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499558    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499562    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499571    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="prom-label-proxy"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499576    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="prom-label-proxy"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499633    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="alertmanager"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499640    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy-web"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499647    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499653    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="config-reloader"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499658    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="kube-rbac-proxy-metric"
Sep 16 19:08:50.499798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.499665    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" containerName="prom-label-proxy"
Sep 16 19:08:50.504593 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.504578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.516255 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.516233    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 19:08:50.516351 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.516276    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 19:08:50.516351 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.516242    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c664w\""
Sep 16 19:08:50.518391 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f7da67c-519b-4876-a21f-0357b71b7d2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518455 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f7da67c-519b-4876-a21f-0357b71b7d2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518455 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7da67c-519b-4876-a21f-0357b71b7d2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518516 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f7da67c-519b-4876-a21f-0357b71b7d2b-config-out\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518516 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-web-config\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518516 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrrz\" (UniqueName: \"kubernetes.io/projected/1f7da67c-519b-4876-a21f-0357b71b7d2b-kube-api-access-5vrrz\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518607 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518607 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518607 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f7da67c-519b-4876-a21f-0357b71b7d2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518785 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.518817 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.518806    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 19:08:50.519155 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.519140    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 19:08:50.519189 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.519143    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 19:08:50.519189 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.519173    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 19:08:50.519257 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.519191    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 19:08:50.519257 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.519201    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 19:08:50.546372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.546351    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 19:08:50.554335 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.554316    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:08:50.619129 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f7da67c-519b-4876-a21f-0357b71b7d2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619274 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619335 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619382 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f7da67c-519b-4876-a21f-0357b71b7d2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f7da67c-519b-4876-a21f-0357b71b7d2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7da67c-519b-4876-a21f-0357b71b7d2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f7da67c-519b-4876-a21f-0357b71b7d2b-config-out\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-web-config\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrrz\" (UniqueName: \"kubernetes.io/projected/1f7da67c-519b-4876-a21f-0357b71b7d2b-kube-api-access-5vrrz\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619610    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.619868 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.619634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.620039 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.620013    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f7da67c-519b-4876-a21f-0357b71b7d2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.621422 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.620584    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f7da67c-519b-4876-a21f-0357b71b7d2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.621422 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.621368    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7da67c-519b-4876-a21f-0357b71b7d2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.622611 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.622530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.622611 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.622549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.623046 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.622818    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f7da67c-519b-4876-a21f-0357b71b7d2b-config-out\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.623046 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.622919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.623046 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.622929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.623332 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.623311    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-web-config\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.623552 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.623535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.624195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.624177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f7da67c-519b-4876-a21f-0357b71b7d2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.624265 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.624248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f7da67c-519b-4876-a21f-0357b71b7d2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.642732 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.642686    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrrz\" (UniqueName: \"kubernetes.io/projected/1f7da67c-519b-4876-a21f-0357b71b7d2b-kube-api-access-5vrrz\") pod \"alertmanager-main-0\" (UID: \"1f7da67c-519b-4876-a21f-0357b71b7d2b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:50.813175 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:50.813145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 19:08:51.109197 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:08:51.109159    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7da67c_519b_4876_a21f_0357b71b7d2b.slice/crio-676ec12cd37ca6ca7a46a4f435a5d104e0c9fe2109e4aec3c08eb28eac31d36d WatchSource:0}: Error finding container 676ec12cd37ca6ca7a46a4f435a5d104e0c9fe2109e4aec3c08eb28eac31d36d: Status 404 returned error can't find the container with id 676ec12cd37ca6ca7a46a4f435a5d104e0c9fe2109e4aec3c08eb28eac31d36d
Sep 16 19:08:51.117903 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:51.117880    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 19:08:51.392864 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:51.392836    2583 generic.go:356] "Generic (PLEG): container finished" podID="1f7da67c-519b-4876-a21f-0357b71b7d2b" containerID="fb6fa21454525ab426122092fac1b5c9e119a5fad0f9e77f26542a0d74d37830" exitCode=0
Sep 16 19:08:51.393051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:51.392924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f7da67c-519b-4876-a21f-0357b71b7d2b","Type":"ContainerDied","Data":"fb6fa21454525ab426122092fac1b5c9e119a5fad0f9e77f26542a0d74d37830"}
Sep 16 19:08:51.393051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:51.393010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f7da67c-519b-4876-a21f-0357b71b7d2b","Type":"ContainerStarted","Data":"676ec12cd37ca6ca7a46a4f435a5d104e0c9fe2109e4aec3c08eb28eac31d36d"}
Sep 16 19:08:51.410755 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:51.410733    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ac109ee-7d12-4d9f-b571-c24ffeed49c1" path="/var/lib/kubelet/pods/1ac109ee-7d12-4d9f-b571-c24ffeed49c1/volumes"
Sep 16 19:08:52.399853 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:52.399820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f7da67c-519b-4876-a21f-0357b71b7d2b","Type":"ContainerStarted","Data":"616eaaebfa81cf20ebdb32f38a6cf299c95d4281e32dc88b95d4a2ee718d4041"}
Sep 16 19:08:52.400278 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:52.399930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f7da67c-519b-4876-a21f-0357b71b7d2b","Type":"ContainerStarted","Data":"7624b419cced00bce9661dd6c0d03c0ac912031eba7288b4ca997f883c086fc7"}
Sep 16 19:08:52.400278 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:52.399977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f7da67c-519b-4876-a21f-0357b71b7d2b","Type":"ContainerStarted","Data":"e5ec3e7e117c78611e68cae421e289c270dc0d2e043cbdccea12dae0b4c7503f"}
Sep 16 19:08:52.400278 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:52.400024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f7da67c-519b-4876-a21f-0357b71b7d2b","Type":"ContainerStarted","Data":"2cff0dd681c8090865356368cbf9608344339c7a396780e4236ed92f9b2fb645"}
Sep 16 19:08:52.400278 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:52.400061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f7da67c-519b-4876-a21f-0357b71b7d2b","Type":"ContainerStarted","Data":"4f0a455bf2965dd3623c76279540930db04cbe8feb7fa967eebc091bafd33272"}
Sep 16 19:08:52.400278 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:52.400099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f7da67c-519b-4876-a21f-0357b71b7d2b","Type":"ContainerStarted","Data":"33b3446a84dd8d4c31e8bb31291bd158dbf527564467389f51c09ae53451a760"}
Sep 16 19:08:52.467824 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:52.467772    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.4511505 podStartE2EDuration="2.46775815s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.60765ms" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:52.466445586 +0000 UTC m=+209.592249974" watchObservedRunningTime="2026-09-16 19:08:52.46775815 +0000 UTC m=+209.593562538"
Sep 16 19:08:53.009523 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.009491    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:08:53.010043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.009850    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="prometheus" containerID="cri-o://9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a" gracePeriod=600
Sep 16 19:08:53.010043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.009850    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy-web" containerID="cri-o://d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b" gracePeriod=600
Sep 16 19:08:53.010043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.009893    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="thanos-sidecar" containerID="cri-o://3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64" gracePeriod=600
Sep 16 19:08:53.010043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.009904    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58" gracePeriod=600
Sep 16 19:08:53.010043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.010016    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy" containerID="cri-o://37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f" gracePeriod=600
Sep 16 19:08:53.010382 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.010076    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="config-reloader" containerID="cri-o://b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763" gracePeriod=600
Sep 16 19:08:53.249884 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.249860    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.342906 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.342842    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.342906 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.342878    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-config\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343143 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.342910    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcnzz\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-kube-api-access-zcnzz\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343210 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343159    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-trusted-ca-bundle\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343275 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343208    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-1\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343275 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343240    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-serving-certs-ca-bundle\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343367 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343299    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-kubelet-serving-ca-bundle\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343367 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343326    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-metrics-client-ca\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343367 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343358    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-tls\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343382    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-grpc-tls\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343432    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-config-out\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-thanos-prometheus-http-client-file\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-tls-assets\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-2\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-db\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343586    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-kube-rbac-proxy\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343614    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343663    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-0\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.343704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343697    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-web-config\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.344735 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.343723    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-metrics-client-certs\") pod \"71408e6a-93ab-432e-83af-985f5896f931\" (UID: \"71408e6a-93ab-432e-83af-985f5896f931\") "
Sep 16 19:08:53.344735 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.344359    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:53.344735 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.344716    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:53.344925 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.344764    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:53.345195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.345165    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:53.345599 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.345567    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:53.345736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.345592    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:53.346232 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.346194    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:53.346457 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.346429    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-config" (OuterVolumeSpecName: "config") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:53.346778 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.346752    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:53.347827 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.347802    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-kube-api-access-zcnzz" (OuterVolumeSpecName: "kube-api-access-zcnzz") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "kube-api-access-zcnzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:53.348771 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.348742    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:53.349529 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.349491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:53.349529 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.349510    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:53.349938 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.349918    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:53.350378 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.350345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-config-out" (OuterVolumeSpecName: "config-out") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:53.350378 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.350345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:53.350668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.350456    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:53.351081 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.351051    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:53.352129 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.352105    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:53.360826 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.360798    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-web-config" (OuterVolumeSpecName: "web-config") pod "71408e6a-93ab-432e-83af-985f5896f931" (UID: "71408e6a-93ab-432e-83af-985f5896f931"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:53.406731 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.406696    2583 generic.go:356] "Generic (PLEG): container finished" podID="71408e6a-93ab-432e-83af-985f5896f931" containerID="3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58" exitCode=0
Sep 16 19:08:53.406731 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.406728    2583 generic.go:356] "Generic (PLEG): container finished" podID="71408e6a-93ab-432e-83af-985f5896f931" containerID="37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f" exitCode=0
Sep 16 19:08:53.406731 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.406738    2583 generic.go:356] "Generic (PLEG): container finished" podID="71408e6a-93ab-432e-83af-985f5896f931" containerID="d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b" exitCode=0
Sep 16 19:08:53.407238 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.406746    2583 generic.go:356] "Generic (PLEG): container finished" podID="71408e6a-93ab-432e-83af-985f5896f931" containerID="3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64" exitCode=0
Sep 16 19:08:53.407238 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.406754    2583 generic.go:356] "Generic (PLEG): container finished" podID="71408e6a-93ab-432e-83af-985f5896f931" containerID="b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763" exitCode=0
Sep 16 19:08:53.407238 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.406762    2583 generic.go:356] "Generic (PLEG): container finished" podID="71408e6a-93ab-432e-83af-985f5896f931" containerID="9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a" exitCode=0
Sep 16 19:08:53.407527 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.407508    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.410731 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.410703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerDied","Data":"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"}
Sep 16 19:08:53.410845 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.410771    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerDied","Data":"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"}
Sep 16 19:08:53.410845 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.410818    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerDied","Data":"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"}
Sep 16 19:08:53.410845 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.410834    2583 scope.go:117] "RemoveContainer" containerID="3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"
Sep 16 19:08:53.410965 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.410860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerDied","Data":"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"}
Sep 16 19:08:53.410965 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.410902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerDied","Data":"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"}
Sep 16 19:08:53.410965 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.410943    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerDied","Data":"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"}
Sep 16 19:08:53.411067 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.410983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71408e6a-93ab-432e-83af-985f5896f931","Type":"ContainerDied","Data":"da9a91b2eedf74106bfdedc6e0f755cca977f4c9d4af7d400b73317d061cfa68"}
Sep 16 19:08:53.418774 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.418758    2583 scope.go:117] "RemoveContainer" containerID="37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"
Sep 16 19:08:53.425694 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.425668    2583 scope.go:117] "RemoveContainer" containerID="d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"
Sep 16 19:08:53.432291 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.432275    2583 scope.go:117] "RemoveContainer" containerID="3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"
Sep 16 19:08:53.439018 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.439003    2583 scope.go:117] "RemoveContainer" containerID="b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"
Sep 16 19:08:53.445687 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445667    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445692    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445708    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-configmap-metrics-client-ca\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445722    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445736    2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-grpc-tls\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445748    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-config-out\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445760    2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445772    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-tls-assets\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445780    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445788    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-db\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.445795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445797    2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-kube-rbac-proxy\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445806    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445814    2583 scope.go:117] "RemoveContainer" containerID="9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445815    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445889    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-web-config\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445901    2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-metrics-client-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445911    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445927    2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/71408e6a-93ab-432e-83af-985f5896f931-config\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445940    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcnzz\" (UniqueName: \"kubernetes.io/projected/71408e6a-93ab-432e-83af-985f5896f931-kube-api-access-zcnzz\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445954    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.446169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.445968    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71408e6a-93ab-432e-83af-985f5896f931-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.452805 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.452789    2583 scope.go:117] "RemoveContainer" containerID="deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"
Sep 16 19:08:53.459346 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.459326    2583 scope.go:117] "RemoveContainer" containerID="3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"
Sep 16 19:08:53.459622 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:53.459605    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": container with ID starting with 3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58 not found: ID does not exist" containerID="3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"
Sep 16 19:08:53.459679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.459630    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"} err="failed to get container status \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": rpc error: code = NotFound desc = could not find container \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": container with ID starting with 3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58 not found: ID does not exist"
Sep 16 19:08:53.459679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.459644    2583 scope.go:117] "RemoveContainer" containerID="37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"
Sep 16 19:08:53.459884 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:53.459866    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": container with ID starting with 37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f not found: ID does not exist" containerID="37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"
Sep 16 19:08:53.459943 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.459893    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"} err="failed to get container status \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": rpc error: code = NotFound desc = could not find container \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": container with ID starting with 37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f not found: ID does not exist"
Sep 16 19:08:53.459943 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.459915    2583 scope.go:117] "RemoveContainer" containerID="d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"
Sep 16 19:08:53.460162 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:53.460146    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": container with ID starting with d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b not found: ID does not exist" containerID="d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"
Sep 16 19:08:53.460211 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.460168    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"} err="failed to get container status \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": rpc error: code = NotFound desc = could not find container \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": container with ID starting with d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b not found: ID does not exist"
Sep 16 19:08:53.460211 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.460182    2583 scope.go:117] "RemoveContainer" containerID="3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"
Sep 16 19:08:53.460448 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:53.460422    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": container with ID starting with 3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64 not found: ID does not exist" containerID="3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"
Sep 16 19:08:53.460523 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.460454    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"} err="failed to get container status \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": rpc error: code = NotFound desc = could not find container \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": container with ID starting with 3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64 not found: ID does not exist"
Sep 16 19:08:53.460523 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.460469    2583 scope.go:117] "RemoveContainer" containerID="b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"
Sep 16 19:08:53.460685 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:53.460669    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": container with ID starting with b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763 not found: ID does not exist" containerID="b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"
Sep 16 19:08:53.460720 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.460690    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"} err="failed to get container status \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": rpc error: code = NotFound desc = could not find container \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": container with ID starting with b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763 not found: ID does not exist"
Sep 16 19:08:53.460720 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.460702    2583 scope.go:117] "RemoveContainer" containerID="9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"
Sep 16 19:08:53.460921 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:53.460906    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": container with ID starting with 9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a not found: ID does not exist" containerID="9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"
Sep 16 19:08:53.460980 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.460924    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"} err="failed to get container status \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": rpc error: code = NotFound desc = could not find container \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": container with ID starting with 9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a not found: ID does not exist"
Sep 16 19:08:53.460980 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.460934    2583 scope.go:117] "RemoveContainer" containerID="deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"
Sep 16 19:08:53.461183 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:08:53.461170    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": container with ID starting with deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06 not found: ID does not exist" containerID="deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"
Sep 16 19:08:53.461224 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.461186    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"} err="failed to get container status \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": rpc error: code = NotFound desc = could not find container \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": container with ID starting with deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06 not found: ID does not exist"
Sep 16 19:08:53.461224 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.461198    2583 scope.go:117] "RemoveContainer" containerID="3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"
Sep 16 19:08:53.461445 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.461396    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"} err="failed to get container status \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": rpc error: code = NotFound desc = could not find container \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": container with ID starting with 3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58 not found: ID does not exist"
Sep 16 19:08:53.461506 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.461471    2583 scope.go:117] "RemoveContainer" containerID="37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"
Sep 16 19:08:53.461682 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.461666    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"} err="failed to get container status \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": rpc error: code = NotFound desc = could not find container \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": container with ID starting with 37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f not found: ID does not exist"
Sep 16 19:08:53.461724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.461682    2583 scope.go:117] "RemoveContainer" containerID="d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"
Sep 16 19:08:53.461869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.461854    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"} err="failed to get container status \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": rpc error: code = NotFound desc = could not find container \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": container with ID starting with d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b not found: ID does not exist"
Sep 16 19:08:53.461869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.461869    2583 scope.go:117] "RemoveContainer" containerID="3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"
Sep 16 19:08:53.462072 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462056    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"} err="failed to get container status \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": rpc error: code = NotFound desc = could not find container \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": container with ID starting with 3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64 not found: ID does not exist"
Sep 16 19:08:53.462121 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462074    2583 scope.go:117] "RemoveContainer" containerID="b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"
Sep 16 19:08:53.462315 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462299    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"} err="failed to get container status \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": rpc error: code = NotFound desc = could not find container \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": container with ID starting with b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763 not found: ID does not exist"
Sep 16 19:08:53.462364 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462314    2583 scope.go:117] "RemoveContainer" containerID="9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"
Sep 16 19:08:53.462545 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462530    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"} err="failed to get container status \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": rpc error: code = NotFound desc = could not find container \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": container with ID starting with 9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a not found: ID does not exist"
Sep 16 19:08:53.462584 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462545    2583 scope.go:117] "RemoveContainer" containerID="deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"
Sep 16 19:08:53.462773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462751    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"} err="failed to get container status \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": rpc error: code = NotFound desc = could not find container \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": container with ID starting with deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06 not found: ID does not exist"
Sep 16 19:08:53.462822 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462774    2583 scope.go:117] "RemoveContainer" containerID="3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"
Sep 16 19:08:53.462967 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462952    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"} err="failed to get container status \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": rpc error: code = NotFound desc = could not find container \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": container with ID starting with 3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58 not found: ID does not exist"
Sep 16 19:08:53.463005 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.462967    2583 scope.go:117] "RemoveContainer" containerID="37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"
Sep 16 19:08:53.463164 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463148    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"} err="failed to get container status \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": rpc error: code = NotFound desc = could not find container \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": container with ID starting with 37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f not found: ID does not exist"
Sep 16 19:08:53.463206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463163    2583 scope.go:117] "RemoveContainer" containerID="d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"
Sep 16 19:08:53.463347 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463332    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"} err="failed to get container status \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": rpc error: code = NotFound desc = could not find container \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": container with ID starting with d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b not found: ID does not exist"
Sep 16 19:08:53.463388 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463347    2583 scope.go:117] "RemoveContainer" containerID="3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"
Sep 16 19:08:53.463564 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463547    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"} err="failed to get container status \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": rpc error: code = NotFound desc = could not find container \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": container with ID starting with 3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64 not found: ID does not exist"
Sep 16 19:08:53.463615 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463564    2583 scope.go:117] "RemoveContainer" containerID="b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"
Sep 16 19:08:53.463770 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463755    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"} err="failed to get container status \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": rpc error: code = NotFound desc = could not find container \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": container with ID starting with b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763 not found: ID does not exist"
Sep 16 19:08:53.463770 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463770    2583 scope.go:117] "RemoveContainer" containerID="9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"
Sep 16 19:08:53.463956 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463941    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"} err="failed to get container status \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": rpc error: code = NotFound desc = could not find container \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": container with ID starting with 9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a not found: ID does not exist"
Sep 16 19:08:53.463994 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.463956    2583 scope.go:117] "RemoveContainer" containerID="deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"
Sep 16 19:08:53.464171 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464155    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"} err="failed to get container status \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": rpc error: code = NotFound desc = could not find container \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": container with ID starting with deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06 not found: ID does not exist"
Sep 16 19:08:53.464171 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464169    2583 scope.go:117] "RemoveContainer" containerID="3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"
Sep 16 19:08:53.464358 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464343    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"} err="failed to get container status \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": rpc error: code = NotFound desc = could not find container \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": container with ID starting with 3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58 not found: ID does not exist"
Sep 16 19:08:53.464400 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464358    2583 scope.go:117] "RemoveContainer" containerID="37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"
Sep 16 19:08:53.464570 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464556    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"} err="failed to get container status \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": rpc error: code = NotFound desc = could not find container \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": container with ID starting with 37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f not found: ID does not exist"
Sep 16 19:08:53.464604 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464570    2583 scope.go:117] "RemoveContainer" containerID="d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"
Sep 16 19:08:53.464773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464758    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"} err="failed to get container status \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": rpc error: code = NotFound desc = could not find container \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": container with ID starting with d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b not found: ID does not exist"
Sep 16 19:08:53.464812 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464773    2583 scope.go:117] "RemoveContainer" containerID="3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"
Sep 16 19:08:53.464960 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464947    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"} err="failed to get container status \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": rpc error: code = NotFound desc = could not find container \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": container with ID starting with 3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64 not found: ID does not exist"
Sep 16 19:08:53.465000 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.464961    2583 scope.go:117] "RemoveContainer" containerID="b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"
Sep 16 19:08:53.465145 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465127    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"} err="failed to get container status \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": rpc error: code = NotFound desc = could not find container \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": container with ID starting with b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763 not found: ID does not exist"
Sep 16 19:08:53.465188 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465145    2583 scope.go:117] "RemoveContainer" containerID="9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"
Sep 16 19:08:53.465355 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465341    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"} err="failed to get container status \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": rpc error: code = NotFound desc = could not find container \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": container with ID starting with 9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a not found: ID does not exist"
Sep 16 19:08:53.465397 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465355    2583 scope.go:117] "RemoveContainer" containerID="deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"
Sep 16 19:08:53.465591 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465573    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"} err="failed to get container status \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": rpc error: code = NotFound desc = could not find container \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": container with ID starting with deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06 not found: ID does not exist"
Sep 16 19:08:53.465634 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465592    2583 scope.go:117] "RemoveContainer" containerID="3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"
Sep 16 19:08:53.465769 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465756    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"} err="failed to get container status \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": rpc error: code = NotFound desc = could not find container \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": container with ID starting with 3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58 not found: ID does not exist"
Sep 16 19:08:53.465807 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465770    2583 scope.go:117] "RemoveContainer" containerID="37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"
Sep 16 19:08:53.465996 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465976    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"} err="failed to get container status \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": rpc error: code = NotFound desc = could not find container \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": container with ID starting with 37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f not found: ID does not exist"
Sep 16 19:08:53.465996 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.465995    2583 scope.go:117] "RemoveContainer" containerID="d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"
Sep 16 19:08:53.466189 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.466172    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"} err="failed to get container status \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": rpc error: code = NotFound desc = could not find container \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": container with ID starting with d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b not found: ID does not exist"
Sep 16 19:08:53.466189 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.466189    2583 scope.go:117] "RemoveContainer" containerID="3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"
Sep 16 19:08:53.466374 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.466353    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"} err="failed to get container status \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": rpc error: code = NotFound desc = could not find container \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": container with ID starting with 3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64 not found: ID does not exist"
Sep 16 19:08:53.466501 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.466375    2583 scope.go:117] "RemoveContainer" containerID="b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"
Sep 16 19:08:53.466617 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.466602    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"} err="failed to get container status \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": rpc error: code = NotFound desc = could not find container \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": container with ID starting with b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763 not found: ID does not exist"
Sep 16 19:08:53.466709 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.466618    2583 scope.go:117] "RemoveContainer" containerID="9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"
Sep 16 19:08:53.466862 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.466842    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"} err="failed to get container status \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": rpc error: code = NotFound desc = could not find container \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": container with ID starting with 9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a not found: ID does not exist"
Sep 16 19:08:53.466905 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.466863    2583 scope.go:117] "RemoveContainer" containerID="deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"
Sep 16 19:08:53.467079 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467064    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"} err="failed to get container status \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": rpc error: code = NotFound desc = could not find container \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": container with ID starting with deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06 not found: ID does not exist"
Sep 16 19:08:53.467123 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467080    2583 scope.go:117] "RemoveContainer" containerID="3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"
Sep 16 19:08:53.467284 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467268    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58"} err="failed to get container status \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": rpc error: code = NotFound desc = could not find container \"3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58\": container with ID starting with 3059b022abeaba74ac9897b7a9cb5d0bdba4c25a64049e9b831c998577142c58 not found: ID does not exist"
Sep 16 19:08:53.467284 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467283    2583 scope.go:117] "RemoveContainer" containerID="37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"
Sep 16 19:08:53.467500 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467485    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f"} err="failed to get container status \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": rpc error: code = NotFound desc = could not find container \"37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f\": container with ID starting with 37d81f3e782894b19968fb55fe91bca5842b6c19a6151486fc2206f2f69c752f not found: ID does not exist"
Sep 16 19:08:53.467539 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467501    2583 scope.go:117] "RemoveContainer" containerID="d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"
Sep 16 19:08:53.467709 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467693    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b"} err="failed to get container status \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": rpc error: code = NotFound desc = could not find container \"d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b\": container with ID starting with d73ca96b8a9c39ee71ad21b065b4be34e122d0c7372a64d02cb7e50d333fb12b not found: ID does not exist"
Sep 16 19:08:53.467746 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467710    2583 scope.go:117] "RemoveContainer" containerID="3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"
Sep 16 19:08:53.467912 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467895    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64"} err="failed to get container status \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": rpc error: code = NotFound desc = could not find container \"3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64\": container with ID starting with 3b22a103bce65900fe3c0cf7da697fc1b122b8c11fa3c46a73f6fdf2ff922c64 not found: ID does not exist"
Sep 16 19:08:53.467912 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.467911    2583 scope.go:117] "RemoveContainer" containerID="b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"
Sep 16 19:08:53.468115 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.468095    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763"} err="failed to get container status \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": rpc error: code = NotFound desc = could not find container \"b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763\": container with ID starting with b2453829162f542e3cd435dd5411db20e7e163d66cc75bf3b06409d540d4b763 not found: ID does not exist"
Sep 16 19:08:53.468158 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.468115    2583 scope.go:117] "RemoveContainer" containerID="9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"
Sep 16 19:08:53.468301 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.468287    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a"} err="failed to get container status \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": rpc error: code = NotFound desc = could not find container \"9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a\": container with ID starting with 9319d70aeb771bfbbbcc3cc9ac7dd40a52fd63d63ad6b9435f587d2a16ffa99a not found: ID does not exist"
Sep 16 19:08:53.468342 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.468301    2583 scope.go:117] "RemoveContainer" containerID="deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"
Sep 16 19:08:53.468530 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.468513    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06"} err="failed to get container status \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": rpc error: code = NotFound desc = could not find container \"deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06\": container with ID starting with deca1a6c7e90a7265d1b04f728d2b3957e029a1cbcf55747ca2f8fc23ce84c06 not found: ID does not exist"
Sep 16 19:08:53.487961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.487937    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:08:53.507643 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.507620    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:08:53.570018 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.569997    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:08:53.570300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570288    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="config-reloader"
Sep 16 19:08:53.570350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570303    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="config-reloader"
Sep 16 19:08:53.570350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570312    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy-thanos"
Sep 16 19:08:53.570350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570318    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy-thanos"
Sep 16 19:08:53.570350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570327    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="thanos-sidecar"
Sep 16 19:08:53.570350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570332    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="thanos-sidecar"
Sep 16 19:08:53.570350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570340    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="prometheus"
Sep 16 19:08:53.570350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570345    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="prometheus"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570356    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="init-config-reloader"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570365    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="init-config-reloader"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570375    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy-web"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570380    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy-web"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570386    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570391    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570461    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy-thanos"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570470    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="prometheus"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570476    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="thanos-sidecar"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570484    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="config-reloader"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570488    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy"
Sep 16 19:08:53.570568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.570494    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71408e6a-93ab-432e-83af-985f5896f931" containerName="kube-rbac-proxy-web"
Sep 16 19:08:53.575542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.575528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.594846 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.594788    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 19:08:53.597554 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.597529    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 19:08:53.600298 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.600283    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f0dted9i830jh\""
Sep 16 19:08:53.601643 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.601629    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gvjwh\""
Sep 16 19:08:53.604305 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.604292    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 19:08:53.615496 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.615476    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 19:08:53.615616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.615478    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 19:08:53.615616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.615478    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 19:08:53.615616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.615517    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 19:08:53.616084 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.616067    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 19:08:53.627790 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.627770    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 19:08:53.641546 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.641526    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 19:08:53.647673 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647714 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647714 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647701    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647782 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647819 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e05c0465-2b7f-448b-895f-23a45971ef52-config-out\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647850 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647882 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz45x\" (UniqueName: \"kubernetes.io/projected/e05c0465-2b7f-448b-895f-23a45971ef52-kube-api-access-mz45x\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647914 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647914 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647976 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647916    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-config\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.647976 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e05c0465-2b7f-448b-895f-23a45971ef52-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.648037 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.647989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-web-config\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.648037 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.648021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.648097 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.648054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.648097 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.648074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.648097 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.648089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.648179 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.648121    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.648179 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.648135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.648179 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.648161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.648266 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.648213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.682808 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.682786    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 19:08:53.685133 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.685120    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 19:08:53.693938 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.693918    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 19:08:53.695375 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.695353    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:08:53.700888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.700872    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 19:08:53.748966 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.748929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.748971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.748990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749141 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749141 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749107    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749141 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749266 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749266 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749266 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749266 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e05c0465-2b7f-448b-895f-23a45971ef52-config-out\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749266 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz45x\" (UniqueName: \"kubernetes.io/projected/e05c0465-2b7f-448b-895f-23a45971ef52-kube-api-access-mz45x\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-config\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e05c0465-2b7f-448b-895f-23a45971ef52-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-web-config\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.749536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.750004 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.750004 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.749851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.750907 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.750879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.751005 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.750984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.751218 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.751193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.751289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.751217    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.753292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.752927    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.753292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.753245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.753292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.753262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.753727 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.753703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e05c0465-2b7f-448b-895f-23a45971ef52-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.754135 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.754101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.754337 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.754317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.754515 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.754496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-config\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.754576 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.754552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.754745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.754724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e05c0465-2b7f-448b-895f-23a45971ef52-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.754814 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.754741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.754909 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.754892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e05c0465-2b7f-448b-895f-23a45971ef52-config-out\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.755083 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.755068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-web-config\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.755598 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.755583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e05c0465-2b7f-448b-895f-23a45971ef52-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.764551 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.764534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz45x\" (UniqueName: \"kubernetes.io/projected/e05c0465-2b7f-448b-895f-23a45971ef52-kube-api-access-mz45x\") pod \"prometheus-k8s-0\" (UID: \"e05c0465-2b7f-448b-895f-23a45971ef52\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:53.889686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:53.889663    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:08:54.195066 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:08:54.195041    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05c0465_2b7f_448b_895f_23a45971ef52.slice/crio-9e70e5ea1668677349611d823e0d0d4606134168aac1b07a6ed886496074b88d WatchSource:0}: Error finding container 9e70e5ea1668677349611d823e0d0d4606134168aac1b07a6ed886496074b88d: Status 404 returned error can't find the container with id 9e70e5ea1668677349611d823e0d0d4606134168aac1b07a6ed886496074b88d
Sep 16 19:08:54.206252 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:54.206231    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 19:08:54.413777 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:54.413750    2583 generic.go:356] "Generic (PLEG): container finished" podID="e05c0465-2b7f-448b-895f-23a45971ef52" containerID="62c54420ef5738ae7b2643f6dcd38a1d631d9541ab4d2550688795b93386eb70" exitCode=0
Sep 16 19:08:54.414145 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:54.413809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e05c0465-2b7f-448b-895f-23a45971ef52","Type":"ContainerDied","Data":"62c54420ef5738ae7b2643f6dcd38a1d631d9541ab4d2550688795b93386eb70"}
Sep 16 19:08:54.414145 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:54.413864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e05c0465-2b7f-448b-895f-23a45971ef52","Type":"ContainerStarted","Data":"9e70e5ea1668677349611d823e0d0d4606134168aac1b07a6ed886496074b88d"}
Sep 16 19:08:55.412557 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:55.412529    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71408e6a-93ab-432e-83af-985f5896f931" path="/var/lib/kubelet/pods/71408e6a-93ab-432e-83af-985f5896f931/volumes"
Sep 16 19:08:55.428919 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:55.428886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e05c0465-2b7f-448b-895f-23a45971ef52","Type":"ContainerStarted","Data":"d3499c30992badb193e0ae561a8961a53c2a3118f67701e2f80597ede34b2416"}
Sep 16 19:08:55.429248 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:55.428973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e05c0465-2b7f-448b-895f-23a45971ef52","Type":"ContainerStarted","Data":"7a0c60f98c8f2de636e2a151fc43286e1c61a0501175334bcc923de4652ea12f"}
Sep 16 19:08:55.429248 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:55.429000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e05c0465-2b7f-448b-895f-23a45971ef52","Type":"ContainerStarted","Data":"40faf7ca070bb039d06572fcfc866710513263c60f9053af40c8caba31ff2d8a"}
Sep 16 19:08:55.429248 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:55.429023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e05c0465-2b7f-448b-895f-23a45971ef52","Type":"ContainerStarted","Data":"56046da522d626771df8fc40a3fa26a11bf9ee2339bd653a8f026dd52b70e3e3"}
Sep 16 19:08:55.429248 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:55.429046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e05c0465-2b7f-448b-895f-23a45971ef52","Type":"ContainerStarted","Data":"5a1c8a5d51ca14b756b82c9685d1c7a5ad2f797ef635f28a42e34db723ee502c"}
Sep 16 19:08:55.429248 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:55.429067    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e05c0465-2b7f-448b-895f-23a45971ef52","Type":"ContainerStarted","Data":"65fa0bb76c7541ec33b1543e568287f87de5c20947e303063333ff1243f46696"}
Sep 16 19:08:55.494634 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:55.494586    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.47672729 podStartE2EDuration="2.494572413s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.845123ms" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:55.489033243 +0000 UTC m=+212.614837633" watchObservedRunningTime="2026-09-16 19:08:55.494572413 +0000 UTC m=+212.620376841"
Sep 16 19:08:58.890508 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:08:58.890478    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:09:53.890176 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:09:53.890078    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:09:53.907877 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:09:53.907848    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:09:54.680044 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:09:54.680016    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 19:10:23.283057 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:10:23.283029    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:10:23.284887 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:10:23.284860    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:10:23.289274 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:10:23.289249    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:10:23.291367 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:10:23.291345    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:10:23.294994 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:10:23.294977    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 19:13:03.426359 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.426332    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-r7qdq"]
Sep 16 19:13:03.429784 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.429764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-r7qdq"
Sep 16 19:13:03.447762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.447742    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 19:13:03.447992 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.447976    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 19:13:03.449456 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.449444    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4jfg9\""
Sep 16 19:13:03.474175 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.474144    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-r7qdq"]
Sep 16 19:13:03.526847 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.526825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f13eec0-c620-4113-8ffa-ed6954906403-bound-sa-token\") pod \"cert-manager-55b4d954d5-r7qdq\" (UID: \"0f13eec0-c620-4113-8ffa-ed6954906403\") " pod="cert-manager/cert-manager-55b4d954d5-r7qdq"
Sep 16 19:13:03.526963 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.526860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2c7r\" (UniqueName: \"kubernetes.io/projected/0f13eec0-c620-4113-8ffa-ed6954906403-kube-api-access-s2c7r\") pod \"cert-manager-55b4d954d5-r7qdq\" (UID: \"0f13eec0-c620-4113-8ffa-ed6954906403\") " pod="cert-manager/cert-manager-55b4d954d5-r7qdq"
Sep 16 19:13:03.628030 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.628012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f13eec0-c620-4113-8ffa-ed6954906403-bound-sa-token\") pod \"cert-manager-55b4d954d5-r7qdq\" (UID: \"0f13eec0-c620-4113-8ffa-ed6954906403\") " pod="cert-manager/cert-manager-55b4d954d5-r7qdq"
Sep 16 19:13:03.628136 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.628042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2c7r\" (UniqueName: \"kubernetes.io/projected/0f13eec0-c620-4113-8ffa-ed6954906403-kube-api-access-s2c7r\") pod \"cert-manager-55b4d954d5-r7qdq\" (UID: \"0f13eec0-c620-4113-8ffa-ed6954906403\") " pod="cert-manager/cert-manager-55b4d954d5-r7qdq"
Sep 16 19:13:03.664070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.664047    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2c7r\" (UniqueName: \"kubernetes.io/projected/0f13eec0-c620-4113-8ffa-ed6954906403-kube-api-access-s2c7r\") pod \"cert-manager-55b4d954d5-r7qdq\" (UID: \"0f13eec0-c620-4113-8ffa-ed6954906403\") " pod="cert-manager/cert-manager-55b4d954d5-r7qdq"
Sep 16 19:13:03.677511 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.677466    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f13eec0-c620-4113-8ffa-ed6954906403-bound-sa-token\") pod \"cert-manager-55b4d954d5-r7qdq\" (UID: \"0f13eec0-c620-4113-8ffa-ed6954906403\") " pod="cert-manager/cert-manager-55b4d954d5-r7qdq"
Sep 16 19:13:03.760502 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:03.760480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-r7qdq"
Sep 16 19:13:04.085287 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:04.085256    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-r7qdq"]
Sep 16 19:13:04.086720 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:13:04.086694    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f13eec0_c620_4113_8ffa_ed6954906403.slice/crio-157d9e520f9abb39d9791e377128e9069077fd4336082bffe6b162f42d5c95a4 WatchSource:0}: Error finding container 157d9e520f9abb39d9791e377128e9069077fd4336082bffe6b162f42d5c95a4: Status 404 returned error can't find the container with id 157d9e520f9abb39d9791e377128e9069077fd4336082bffe6b162f42d5c95a4
Sep 16 19:13:04.088829 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:04.088815    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:13:04.415059 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:04.415027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-r7qdq" event={"ID":"0f13eec0-c620-4113-8ffa-ed6954906403","Type":"ContainerStarted","Data":"157d9e520f9abb39d9791e377128e9069077fd4336082bffe6b162f42d5c95a4"}
Sep 16 19:13:07.430459 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:07.430404    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-r7qdq" event={"ID":"0f13eec0-c620-4113-8ffa-ed6954906403","Type":"ContainerStarted","Data":"861c3409c602fe5f178afe8d72e2342cd33f39135ef3e168904f8430918716a9"}
Sep 16 19:13:33.825648 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.825595    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-r7qdq" podStartSLOduration=27.674805219 podStartE2EDuration="30.825578762s" totalImagesPullingTime="3.150773543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:07.464639285 +0000 UTC m=+464.590443675" watchObservedRunningTime="2026-09-16 19:13:33.825578762 +0000 UTC m=+490.951383190"
Sep 16 19:13:33.826164 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.826027    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"]
Sep 16 19:13:33.831290 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.831268    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:33.839819 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.839797    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:13:33.840228 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.840210    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 19:13:33.840330 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.840269    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 19:13:33.840330 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.840284    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 19:13:33.840517 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.840380    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wqgrr\""
Sep 16 19:13:33.842105 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.842087    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 19:13:33.956090 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.956066    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"]
Sep 16 19:13:33.970669 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.970650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8rd\" (UniqueName: \"kubernetes.io/projected/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-kube-api-access-5q8rd\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:33.970773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.970689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-cert\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:33.970773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.970716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-manager-config\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:33.970773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:33.970746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-metrics-cert\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.071282 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.071258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8rd\" (UniqueName: \"kubernetes.io/projected/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-kube-api-access-5q8rd\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.071385 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.071290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-cert\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.071385 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.071310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-manager-config\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.071385 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.071330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-metrics-cert\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.072022 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.071955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-manager-config\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.073739 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.073721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-cert\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.073902 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.073880    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-metrics-cert\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.090678 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.090629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8rd\" (UniqueName: \"kubernetes.io/projected/1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03-kube-api-access-5q8rd\") pod \"lws-controller-manager-9498697bd-rc2tm\" (UID: \"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03\") " pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.141244 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.141228    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:34.501571 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.501550    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"]
Sep 16 19:13:34.503472 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:13:34.503445    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7bcbfd_1ab7_4cd2_9dd1_a3be26ec5b03.slice/crio-5cee8b1f7559b5d885e1e8188e679a2b37b2ab8e6f13abdaf01e0adc12ab142b WatchSource:0}: Error finding container 5cee8b1f7559b5d885e1e8188e679a2b37b2ab8e6f13abdaf01e0adc12ab142b: Status 404 returned error can't find the container with id 5cee8b1f7559b5d885e1e8188e679a2b37b2ab8e6f13abdaf01e0adc12ab142b
Sep 16 19:13:34.537725 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:34.537700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm" event={"ID":"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03","Type":"ContainerStarted","Data":"5cee8b1f7559b5d885e1e8188e679a2b37b2ab8e6f13abdaf01e0adc12ab142b"}
Sep 16 19:13:37.553161 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:37.553123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm" event={"ID":"1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03","Type":"ContainerStarted","Data":"92661251e8e1f6a5a4139510d2556ace66b74239f7b136d6b36432f7ed798a8d"}
Sep 16 19:13:37.553568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:37.553234    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:37.597310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:37.597262    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm" podStartSLOduration=2.346242101 podStartE2EDuration="4.597248169s" totalImagesPullingTime="2.251006068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:37.595167774 +0000 UTC m=+494.720972162" watchObservedRunningTime="2026-09-16 19:13:37.597248169 +0000 UTC m=+494.723052556"
Sep 16 19:13:45.060998 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.060970    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"]
Sep 16 19:13:45.064283 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.064267    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.068305 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.068290    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 19:13:45.068394 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.068290    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-8rjzn\""
Sep 16 19:13:45.069382 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.069369    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 19:13:45.072024 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.072005    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:13:45.105385 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.105362    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"]
Sep 16 19:13:45.157206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.157175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.157322 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.157215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6klk\" (UniqueName: \"kubernetes.io/projected/8f4d30db-1bc7-409a-9748-62ea5efc867c-kube-api-access-l6klk\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.157322 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.157244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f4d30db-1bc7-409a-9748-62ea5efc867c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.157322 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.157307    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.157507 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.157350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.157507 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.157375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.157507 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.157481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.157616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.157506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.157616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.157536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258378 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258474    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258504 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258771 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6klk\" (UniqueName: \"kubernetes.io/projected/8f4d30db-1bc7-409a-9748-62ea5efc867c-kube-api-access-l6klk\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258771 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f4d30db-1bc7-409a-9748-62ea5efc867c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258771 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258796    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.258935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.258916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.259052 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.259023    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.259287 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.259259    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f4d30db-1bc7-409a-9748-62ea5efc867c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.260943 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.260923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.261172 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.261153    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.274727 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.274706    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f4d30db-1bc7-409a-9748-62ea5efc867c-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.275203 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.275185    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6klk\" (UniqueName: \"kubernetes.io/projected/8f4d30db-1bc7-409a-9748-62ea5efc867c-kube-api-access-l6klk\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4\" (UID: \"8f4d30db-1bc7-409a-9748-62ea5efc867c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.374372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.374352    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:45.680929 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:13:45.680893    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4d30db_1bc7_409a_9748_62ea5efc867c.slice/crio-254fc01b6c10f7e2f4a26751956f67af4f58a5f362e71d34a3c053b2d549aa8b WatchSource:0}: Error finding container 254fc01b6c10f7e2f4a26751956f67af4f58a5f362e71d34a3c053b2d549aa8b: Status 404 returned error can't find the container with id 254fc01b6c10f7e2f4a26751956f67af4f58a5f362e71d34a3c053b2d549aa8b
Sep 16 19:13:45.682628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:45.682557    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"]
Sep 16 19:13:46.594599 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:46.594558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" event={"ID":"8f4d30db-1bc7-409a-9748-62ea5efc867c","Type":"ContainerStarted","Data":"254fc01b6c10f7e2f4a26751956f67af4f58a5f362e71d34a3c053b2d549aa8b"}
Sep 16 19:13:48.457729 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:48.457694    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:13:48.457984 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:48.457768    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:13:48.457984 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:48.457797    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:13:48.559426 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:48.559385    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9498697bd-rc2tm"
Sep 16 19:13:48.607324 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:48.607228    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" event={"ID":"8f4d30db-1bc7-409a-9748-62ea5efc867c","Type":"ContainerStarted","Data":"3853dc257f631cc04e4390df6bf9b2224bed877228c2bfe9baf64b0cd72b46b6"}
Sep 16 19:13:48.655317 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:48.655270    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" podStartSLOduration=0.880674862 podStartE2EDuration="3.655253226s" totalImagesPullingTime="2.774578364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:48.652363283 +0000 UTC m=+505.778167672" watchObservedRunningTime="2026-09-16 19:13:48.655253226 +0000 UTC m=+505.781057615"
Sep 16 19:13:49.375576 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:49.375539    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:49.380265 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:49.380235    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:49.612131 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:49.612098    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:13:49.613441 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:13:49.613380    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:14:19.703656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:19.703578    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"]
Sep 16 19:14:19.706133 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:19.706112    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"
Sep 16 19:14:19.710371 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:19.710354    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hz99t\""
Sep 16 19:14:19.710541 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:19.710521    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 19:14:19.711356 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:19.711335    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 19:14:19.727365 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:19.727341    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"]
Sep 16 19:14:19.831467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:19.831441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxcv\" (UniqueName: \"kubernetes.io/projected/8324512c-72c3-4dcf-b68e-e92739454ff8-kube-api-access-5qxcv\") pod \"authorino-operator-6fc8f864f4-2rdd5\" (UID: \"8324512c-72c3-4dcf-b68e-e92739454ff8\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"
Sep 16 19:14:19.931963 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:19.931938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxcv\" (UniqueName: \"kubernetes.io/projected/8324512c-72c3-4dcf-b68e-e92739454ff8-kube-api-access-5qxcv\") pod \"authorino-operator-6fc8f864f4-2rdd5\" (UID: \"8324512c-72c3-4dcf-b68e-e92739454ff8\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"
Sep 16 19:14:19.956951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:19.956904    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxcv\" (UniqueName: \"kubernetes.io/projected/8324512c-72c3-4dcf-b68e-e92739454ff8-kube-api-access-5qxcv\") pod \"authorino-operator-6fc8f864f4-2rdd5\" (UID: \"8324512c-72c3-4dcf-b68e-e92739454ff8\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"
Sep 16 19:14:20.016836 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:20.016814    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"
Sep 16 19:14:20.311649 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:20.311623    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"]
Sep 16 19:14:20.314376 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:14:20.314352    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8324512c_72c3_4dcf_b68e_e92739454ff8.slice/crio-7dc227fcf49c7222768b81bd246bf37fab9de046fcba00d419519ecc9e807781 WatchSource:0}: Error finding container 7dc227fcf49c7222768b81bd246bf37fab9de046fcba00d419519ecc9e807781: Status 404 returned error can't find the container with id 7dc227fcf49c7222768b81bd246bf37fab9de046fcba00d419519ecc9e807781
Sep 16 19:14:20.742555 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:20.742526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5" event={"ID":"8324512c-72c3-4dcf-b68e-e92739454ff8","Type":"ContainerStarted","Data":"7dc227fcf49c7222768b81bd246bf37fab9de046fcba00d419519ecc9e807781"}
Sep 16 19:14:23.757681 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:23.757645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5" event={"ID":"8324512c-72c3-4dcf-b68e-e92739454ff8","Type":"ContainerStarted","Data":"54fc7eb9a3a738ece7164302644de72570ba304152d992935c02322ee1648667"}
Sep 16 19:14:23.758040 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:23.757827    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"
Sep 16 19:14:23.812013 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:23.811969    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5" podStartSLOduration=2.154565421 podStartE2EDuration="4.811954894s" totalImagesPullingTime="2.657389473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:23.795074625 +0000 UTC m=+540.920879010" watchObservedRunningTime="2026-09-16 19:14:23.811954894 +0000 UTC m=+540.937759281"
Sep 16 19:14:34.763782 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:14:34.763754    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-2rdd5"
Sep 16 19:15:06.529974 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:06.529938    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4bwhz"]
Sep 16 19:15:06.534502 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:06.534483    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz"
Sep 16 19:15:06.539568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:06.539544    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4df5s\""
Sep 16 19:15:06.545574 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:06.545550    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4bwhz"]
Sep 16 19:15:06.613040 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:06.613014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpd8\" (UniqueName: \"kubernetes.io/projected/4269dd83-dd1e-4118-bda5-11d457e5479f-kube-api-access-fqpd8\") pod \"authorino-7bcfd8f9cd-4bwhz\" (UID: \"4269dd83-dd1e-4118-bda5-11d457e5479f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz"
Sep 16 19:15:06.713796 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:06.713771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpd8\" (UniqueName: \"kubernetes.io/projected/4269dd83-dd1e-4118-bda5-11d457e5479f-kube-api-access-fqpd8\") pod \"authorino-7bcfd8f9cd-4bwhz\" (UID: \"4269dd83-dd1e-4118-bda5-11d457e5479f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz"
Sep 16 19:15:06.733054 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:06.733029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpd8\" (UniqueName: \"kubernetes.io/projected/4269dd83-dd1e-4118-bda5-11d457e5479f-kube-api-access-fqpd8\") pod \"authorino-7bcfd8f9cd-4bwhz\" (UID: \"4269dd83-dd1e-4118-bda5-11d457e5479f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz"
Sep 16 19:15:06.843138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:06.843089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz"
Sep 16 19:15:07.131367 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:07.131342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4bwhz"]
Sep 16 19:15:07.133144 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:15:07.133119    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4269dd83_dd1e_4118_bda5_11d457e5479f.slice/crio-f0a93c99c435645ee7f86057c3d5839d29adecabe4074183edfc655b135e83ab WatchSource:0}: Error finding container f0a93c99c435645ee7f86057c3d5839d29adecabe4074183edfc655b135e83ab: Status 404 returned error can't find the container with id f0a93c99c435645ee7f86057c3d5839d29adecabe4074183edfc655b135e83ab
Sep 16 19:15:07.957320 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:07.957286    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz" event={"ID":"4269dd83-dd1e-4118-bda5-11d457e5479f","Type":"ContainerStarted","Data":"f0a93c99c435645ee7f86057c3d5839d29adecabe4074183edfc655b135e83ab"}
Sep 16 19:15:11.257736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:11.257663    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4bwhz"]
Sep 16 19:15:11.975332 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:11.975300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz" event={"ID":"4269dd83-dd1e-4118-bda5-11d457e5479f","Type":"ContainerStarted","Data":"f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a"}
Sep 16 19:15:11.998246 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:11.998200    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz" podStartSLOduration=2.186568242 podStartE2EDuration="5.998185881s" totalImagesPullingTime="3.811617639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:11.997022273 +0000 UTC m=+589.122826660" watchObservedRunningTime="2026-09-16 19:15:11.998185881 +0000 UTC m=+589.123990268"
Sep 16 19:15:12.980548 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:12.980512    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz" podUID="4269dd83-dd1e-4118-bda5-11d457e5479f" containerName="authorino" containerID="cri-o://f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a" gracePeriod=30
Sep 16 19:15:13.222970 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.222948    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz"
Sep 16 19:15:13.371351 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.371327    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqpd8\" (UniqueName: \"kubernetes.io/projected/4269dd83-dd1e-4118-bda5-11d457e5479f-kube-api-access-fqpd8\") pod \"4269dd83-dd1e-4118-bda5-11d457e5479f\" (UID: \"4269dd83-dd1e-4118-bda5-11d457e5479f\") "
Sep 16 19:15:13.373582 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.373560    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4269dd83-dd1e-4118-bda5-11d457e5479f-kube-api-access-fqpd8" (OuterVolumeSpecName: "kube-api-access-fqpd8") pod "4269dd83-dd1e-4118-bda5-11d457e5479f" (UID: "4269dd83-dd1e-4118-bda5-11d457e5479f"). InnerVolumeSpecName "kube-api-access-fqpd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:13.471909 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.471884    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpd8\" (UniqueName: \"kubernetes.io/projected/4269dd83-dd1e-4118-bda5-11d457e5479f-kube-api-access-fqpd8\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:15:13.986707 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.986669    2583 generic.go:356] "Generic (PLEG): container finished" podID="4269dd83-dd1e-4118-bda5-11d457e5479f" containerID="f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a" exitCode=0
Sep 16 19:15:13.987147 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.986720    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz"
Sep 16 19:15:13.987147 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.986747    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz" event={"ID":"4269dd83-dd1e-4118-bda5-11d457e5479f","Type":"ContainerDied","Data":"f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a"}
Sep 16 19:15:13.987147 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.986791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4bwhz" event={"ID":"4269dd83-dd1e-4118-bda5-11d457e5479f","Type":"ContainerDied","Data":"f0a93c99c435645ee7f86057c3d5839d29adecabe4074183edfc655b135e83ab"}
Sep 16 19:15:13.987147 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.986810    2583 scope.go:117] "RemoveContainer" containerID="f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a"
Sep 16 19:15:13.995843 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.995826    2583 scope.go:117] "RemoveContainer" containerID="f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a"
Sep 16 19:15:13.996119 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:15:13.996102    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a\": container with ID starting with f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a not found: ID does not exist" containerID="f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a"
Sep 16 19:15:13.996183 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:13.996125    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a"} err="failed to get container status \"f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a\": rpc error: code = NotFound desc = could not find container \"f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a\": container with ID starting with f41f6ec1d851c43c361d01492ecd3c36fdf6ce8d84c98b3becbcbf16063d8e2a not found: ID does not exist"
Sep 16 19:15:14.013621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:14.013592    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4bwhz"]
Sep 16 19:15:14.021785 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:14.021761    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4bwhz"]
Sep 16 19:15:15.411952 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:15.411923    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4269dd83-dd1e-4118-bda5-11d457e5479f" path="/var/lib/kubelet/pods/4269dd83-dd1e-4118-bda5-11d457e5479f/volumes"
Sep 16 19:15:23.314034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:23.314004    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:15:23.314513 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:23.314437    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:15:23.319288 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:23.319267    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:15:23.319735 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:23.319717    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:15:30.468573 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.468542    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-qw4kr"]
Sep 16 19:15:30.468910 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.468849    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4269dd83-dd1e-4118-bda5-11d457e5479f" containerName="authorino"
Sep 16 19:15:30.468910 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.468859    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4269dd83-dd1e-4118-bda5-11d457e5479f" containerName="authorino"
Sep 16 19:15:30.468979 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.468919    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4269dd83-dd1e-4118-bda5-11d457e5479f" containerName="authorino"
Sep 16 19:15:30.473065 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.473050    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-qw4kr"
Sep 16 19:15:30.479987 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.479965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 19:15:30.480094 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.479968    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4df5s\""
Sep 16 19:15:30.486686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.486657    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-qw4kr"]
Sep 16 19:15:30.522208 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.522179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2gw\" (UniqueName: \"kubernetes.io/projected/c7b07186-8e1b-47bd-9caf-1824cc7889a6-kube-api-access-vj2gw\") pod \"authorino-6d9898bd99-qw4kr\" (UID: \"c7b07186-8e1b-47bd-9caf-1824cc7889a6\") " pod="kuadrant-system/authorino-6d9898bd99-qw4kr"
Sep 16 19:15:30.522302 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.522249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7b07186-8e1b-47bd-9caf-1824cc7889a6-tls-cert\") pod \"authorino-6d9898bd99-qw4kr\" (UID: \"c7b07186-8e1b-47bd-9caf-1824cc7889a6\") " pod="kuadrant-system/authorino-6d9898bd99-qw4kr"
Sep 16 19:15:30.622613 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.622587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7b07186-8e1b-47bd-9caf-1824cc7889a6-tls-cert\") pod \"authorino-6d9898bd99-qw4kr\" (UID: \"c7b07186-8e1b-47bd-9caf-1824cc7889a6\") " pod="kuadrant-system/authorino-6d9898bd99-qw4kr"
Sep 16 19:15:30.622717 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.622642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2gw\" (UniqueName: \"kubernetes.io/projected/c7b07186-8e1b-47bd-9caf-1824cc7889a6-kube-api-access-vj2gw\") pod \"authorino-6d9898bd99-qw4kr\" (UID: \"c7b07186-8e1b-47bd-9caf-1824cc7889a6\") " pod="kuadrant-system/authorino-6d9898bd99-qw4kr"
Sep 16 19:15:30.625141 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.625124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7b07186-8e1b-47bd-9caf-1824cc7889a6-tls-cert\") pod \"authorino-6d9898bd99-qw4kr\" (UID: \"c7b07186-8e1b-47bd-9caf-1824cc7889a6\") " pod="kuadrant-system/authorino-6d9898bd99-qw4kr"
Sep 16 19:15:30.633373 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.633352    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2gw\" (UniqueName: \"kubernetes.io/projected/c7b07186-8e1b-47bd-9caf-1824cc7889a6-kube-api-access-vj2gw\") pod \"authorino-6d9898bd99-qw4kr\" (UID: \"c7b07186-8e1b-47bd-9caf-1824cc7889a6\") " pod="kuadrant-system/authorino-6d9898bd99-qw4kr"
Sep 16 19:15:30.781545 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:30.781494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-qw4kr"
Sep 16 19:15:31.058274 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:31.058246    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-qw4kr"]
Sep 16 19:15:31.060820 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:15:31.060793    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7b07186_8e1b_47bd_9caf_1824cc7889a6.slice/crio-7497f9c8984a7488ea64269194acb77eab927e704b176cd30e46b9c547d1cb3c WatchSource:0}: Error finding container 7497f9c8984a7488ea64269194acb77eab927e704b176cd30e46b9c547d1cb3c: Status 404 returned error can't find the container with id 7497f9c8984a7488ea64269194acb77eab927e704b176cd30e46b9c547d1cb3c
Sep 16 19:15:32.066109 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:32.066080    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-qw4kr" event={"ID":"c7b07186-8e1b-47bd-9caf-1824cc7889a6","Type":"ContainerStarted","Data":"566c54390f9452473bbc64fb5fbdca34ee939b43b3bfeaea93132e219f1a7550"}
Sep 16 19:15:32.066537 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:32.066120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-qw4kr" event={"ID":"c7b07186-8e1b-47bd-9caf-1824cc7889a6","Type":"ContainerStarted","Data":"7497f9c8984a7488ea64269194acb77eab927e704b176cd30e46b9c547d1cb3c"}
Sep 16 19:15:32.091444 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:15:32.091384    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-qw4kr" podStartSLOduration=1.6140851120000002 podStartE2EDuration="2.091367258s" totalImagesPullingTime="477.282146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:32.090625078 +0000 UTC m=+609.216429467" watchObservedRunningTime="2026-09-16 19:15:32.091367258 +0000 UTC m=+609.217171646"
Sep 16 19:19:47.202001 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.201971    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:19:47.205466 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.205446    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.256542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.256517    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 19:19:47.257639 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.257610    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:19:47.257762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.257677    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 19:19:47.257762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.257694    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-gts24\""
Sep 16 19:19:47.257762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.257671    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 19:19:47.257762 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.257738    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-rz87t\""
Sep 16 19:19:47.258571 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.258557    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 19:19:47.271428 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.271392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.271542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.271440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.271542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.271476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.271542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.271509    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21b34306-3b1a-46e8-b400-e85a3a60a4e1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.271542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.271530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.271689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.271600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxx2\" (UniqueName: \"kubernetes.io/projected/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kube-api-access-5xxx2\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.271689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.271649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.271689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.271682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.299208 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.299180    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:19:47.372277 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21b34306-3b1a-46e8-b400-e85a3a60a4e1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.372464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.372464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxx2\" (UniqueName: \"kubernetes.io/projected/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kube-api-access-5xxx2\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.372464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.372464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.372685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.372685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.372685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.372821 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.372926 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.373046 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372935    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21b34306-3b1a-46e8-b400-e85a3a60a4e1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.373046 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.372988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.373046 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.373032    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.374720 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.374696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.374989 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.374974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.419147 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.419119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxx2\" (UniqueName: \"kubernetes.io/projected/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kube-api-access-5xxx2\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.515692 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.515623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:19:47.814691 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.814612    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:19:47.817400 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:19:47.817373    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b34306_3b1a_46e8_b400_e85a3a60a4e1.slice/crio-8387b7f8c78a7807f96e3ca9e06d2e6e824fb8253aa223579f93df6b798d4f24 WatchSource:0}: Error finding container 8387b7f8c78a7807f96e3ca9e06d2e6e824fb8253aa223579f93df6b798d4f24: Status 404 returned error can't find the container with id 8387b7f8c78a7807f96e3ca9e06d2e6e824fb8253aa223579f93df6b798d4f24
Sep 16 19:19:47.819454 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:47.819436    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:19:48.262260 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:48.262216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"21b34306-3b1a-46e8-b400-e85a3a60a4e1","Type":"ContainerStarted","Data":"8387b7f8c78a7807f96e3ca9e06d2e6e824fb8253aa223579f93df6b798d4f24"}
Sep 16 19:19:53.292347 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:53.292307    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"21b34306-3b1a-46e8-b400-e85a3a60a4e1","Type":"ContainerStarted","Data":"148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c"}
Sep 16 19:19:56.308460 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:56.308398    2583 generic.go:356] "Generic (PLEG): container finished" podID="21b34306-3b1a-46e8-b400-e85a3a60a4e1" containerID="148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c" exitCode=0
Sep 16 19:19:56.308848 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:56.308486    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"21b34306-3b1a-46e8-b400-e85a3a60a4e1","Type":"ContainerDied","Data":"148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c"}
Sep 16 19:19:59.345179 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:59.345093    2583 generic.go:356] "Generic (PLEG): container finished" podID="8f4d30db-1bc7-409a-9748-62ea5efc867c" containerID="3853dc257f631cc04e4390df6bf9b2224bed877228c2bfe9baf64b0cd72b46b6" exitCode=0
Sep 16 19:19:59.345179 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:59.345143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" event={"ID":"8f4d30db-1bc7-409a-9748-62ea5efc867c","Type":"ContainerDied","Data":"3853dc257f631cc04e4390df6bf9b2224bed877228c2bfe9baf64b0cd72b46b6"}
Sep 16 19:19:59.345779 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:59.345625    2583 scope.go:117] "RemoveContainer" containerID="3853dc257f631cc04e4390df6bf9b2224bed877228c2bfe9baf64b0cd72b46b6"
Sep 16 19:19:59.346869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:59.346834    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:19:59.346966 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:59.346914    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:19:59.347026 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:19:59.346968    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:20:00.356705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:00.356665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" event={"ID":"8f4d30db-1bc7-409a-9748-62ea5efc867c","Type":"ContainerStarted","Data":"92a202e0703417a58f6e36a53e196fbcdac81b138694a7142614d9102c8abfde"}
Sep 16 19:20:00.375189 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:00.375150    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:20:00.376359 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:00.376330    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 19:20:00.376522 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:00.376389    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" podUID="8f4d30db-1bc7-409a-9748-62ea5efc867c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 19:20:01.375744 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:01.375709    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 19:20:01.376189 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:01.375905    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" podUID="8f4d30db-1bc7-409a-9748-62ea5efc867c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 19:20:02.374793 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:02.374754    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 19:20:02.374956 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:02.374820    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" podUID="8f4d30db-1bc7-409a-9748-62ea5efc867c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 19:20:03.375222 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:03.375190    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 19:20:03.375710 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:03.375263    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" podUID="8f4d30db-1bc7-409a-9748-62ea5efc867c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 19:20:04.375163 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:04.375018    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 19:20:04.375163 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:04.375081    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" podUID="8f4d30db-1bc7-409a-9748-62ea5efc867c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 19:20:04.620459 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:04.617510    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:20:05.375690 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:05.375641    2583 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 19:20:05.376092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:05.375714    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4" podUID="8f4d30db-1bc7-409a-9748-62ea5efc867c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 19:20:06.380611 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:06.380576    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:20:06.381963 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:06.381902    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4"
Sep 16 19:20:18.174139 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.174050    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"]
Sep 16 19:20:18.178684 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.178658    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:20:18.184148 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.184123    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 19:20:18.185436 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.185265    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-km7p5\""
Sep 16 19:20:18.185436 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.185293    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-gk6z8\""
Sep 16 19:20:18.187185 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.186901    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 19:20:18.187185 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.186981    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 19:20:18.194653 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.194627    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"]
Sep 16 19:20:18.306145 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.306084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9z6\" (UniqueName: \"kubernetes.io/projected/26970baa-7309-47a9-9246-e9f916a11d21-kube-api-access-zx9z6\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4\" (UID: \"26970baa-7309-47a9-9246-e9f916a11d21\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:20:18.306349 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.306215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26970baa-7309-47a9-9246-e9f916a11d21-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4\" (UID: \"26970baa-7309-47a9-9246-e9f916a11d21\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:20:18.407831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.407793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9z6\" (UniqueName: \"kubernetes.io/projected/26970baa-7309-47a9-9246-e9f916a11d21-kube-api-access-zx9z6\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4\" (UID: \"26970baa-7309-47a9-9246-e9f916a11d21\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:20:18.408023 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.407904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26970baa-7309-47a9-9246-e9f916a11d21-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4\" (UID: \"26970baa-7309-47a9-9246-e9f916a11d21\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:20:18.411004 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.410949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26970baa-7309-47a9-9246-e9f916a11d21-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4\" (UID: \"26970baa-7309-47a9-9246-e9f916a11d21\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:20:18.420102 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.420070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9z6\" (UniqueName: \"kubernetes.io/projected/26970baa-7309-47a9-9246-e9f916a11d21-kube-api-access-zx9z6\") pod \"fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4\" (UID: \"26970baa-7309-47a9-9246-e9f916a11d21\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:20:18.491371 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:18.491275    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:20:26.719261 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:26.719236    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:20:26.719696 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:26.719302    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:20:26.725265 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:26.725245    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:20:26.725380 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:26.725248    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:20:26.978221 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:26.978139    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"]
Sep 16 19:20:26.988711 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:20:26.988677    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26970baa_7309_47a9_9246_e9f916a11d21.slice/crio-ce41e97cd5718f54ba46ca2eae973d2295ea60fa33583d00f707f8d31b3939e1 WatchSource:0}: Error finding container ce41e97cd5718f54ba46ca2eae973d2295ea60fa33583d00f707f8d31b3939e1: Status 404 returned error can't find the container with id ce41e97cd5718f54ba46ca2eae973d2295ea60fa33583d00f707f8d31b3939e1
Sep 16 19:20:27.511037 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:27.510990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" event={"ID":"26970baa-7309-47a9-9246-e9f916a11d21","Type":"ContainerStarted","Data":"ce41e97cd5718f54ba46ca2eae973d2295ea60fa33583d00f707f8d31b3939e1"}
Sep 16 19:20:27.513962 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:27.513933    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"21b34306-3b1a-46e8-b400-e85a3a60a4e1","Type":"ContainerStarted","Data":"c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988"}
Sep 16 19:20:27.547220 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:27.547158    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.294580995 podStartE2EDuration="40.547141938s" totalImagesPullingTime="35.774856197s" totalInitContainerRuntime="2.477704746s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:27.545862177 +0000 UTC m=+904.671666566" watchObservedRunningTime="2026-09-16 19:20:27.547141938 +0000 UTC m=+904.672946325"
Sep 16 19:20:30.532364 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:30.532321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" event={"ID":"26970baa-7309-47a9-9246-e9f916a11d21","Type":"ContainerStarted","Data":"fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13"}
Sep 16 19:20:30.532881 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:30.532398    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:20:30.573917 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:20:30.573864    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" podStartSLOduration=9.563774832 podStartE2EDuration="12.573848314s" totalImagesPullingTime="3.010073482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:30.571761951 +0000 UTC m=+907.697566338" watchObservedRunningTime="2026-09-16 19:20:30.573848314 +0000 UTC m=+907.699652710"
Sep 16 19:21:01.540380 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:01.540352    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:21:09.191810 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:09.191775    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"]
Sep 16 19:21:11.539908 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:11.539867    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" podUID="26970baa-7309-47a9-9246-e9f916a11d21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:21.540940 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:21.540897    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" podUID="26970baa-7309-47a9-9246-e9f916a11d21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:24.211182 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:24.211136    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" podUID="26970baa-7309-47a9-9246-e9f916a11d21" containerName="main" containerID="cri-o://fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13" gracePeriod=45
Sep 16 19:21:31.540487 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:31.540445    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" podUID="26970baa-7309-47a9-9246-e9f916a11d21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:31.540871 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:31.540577    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:21:41.540090 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:41.540049    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" podUID="26970baa-7309-47a9-9246-e9f916a11d21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:51.539998 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:51.539910    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" podUID="26970baa-7309-47a9-9246-e9f916a11d21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:54.454058 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.454029    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:21:54.578091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.577999    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx9z6\" (UniqueName: \"kubernetes.io/projected/26970baa-7309-47a9-9246-e9f916a11d21-kube-api-access-zx9z6\") pod \"26970baa-7309-47a9-9246-e9f916a11d21\" (UID: \"26970baa-7309-47a9-9246-e9f916a11d21\") "
Sep 16 19:21:54.578091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.578056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26970baa-7309-47a9-9246-e9f916a11d21-tls-certs\") pod \"26970baa-7309-47a9-9246-e9f916a11d21\" (UID: \"26970baa-7309-47a9-9246-e9f916a11d21\") "
Sep 16 19:21:54.580532 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.580500    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26970baa-7309-47a9-9246-e9f916a11d21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26970baa-7309-47a9-9246-e9f916a11d21" (UID: "26970baa-7309-47a9-9246-e9f916a11d21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:54.580650 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.580532    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26970baa-7309-47a9-9246-e9f916a11d21-kube-api-access-zx9z6" (OuterVolumeSpecName: "kube-api-access-zx9z6") pod "26970baa-7309-47a9-9246-e9f916a11d21" (UID: "26970baa-7309-47a9-9246-e9f916a11d21"). InnerVolumeSpecName "kube-api-access-zx9z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:54.679669 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.679641    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26970baa-7309-47a9-9246-e9f916a11d21-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:21:54.679669 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.679666    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx9z6\" (UniqueName: \"kubernetes.io/projected/26970baa-7309-47a9-9246-e9f916a11d21-kube-api-access-zx9z6\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:21:54.965895 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.965859    2583 generic.go:356] "Generic (PLEG): container finished" podID="26970baa-7309-47a9-9246-e9f916a11d21" containerID="fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13" exitCode=0
Sep 16 19:21:54.966056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.965907    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"
Sep 16 19:21:54.966056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.965948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" event={"ID":"26970baa-7309-47a9-9246-e9f916a11d21","Type":"ContainerDied","Data":"fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13"}
Sep 16 19:21:54.966056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.966001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4" event={"ID":"26970baa-7309-47a9-9246-e9f916a11d21","Type":"ContainerDied","Data":"ce41e97cd5718f54ba46ca2eae973d2295ea60fa33583d00f707f8d31b3939e1"}
Sep 16 19:21:54.966056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.966024    2583 scope.go:117] "RemoveContainer" containerID="fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13"
Sep 16 19:21:54.975596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.975582    2583 scope.go:117] "RemoveContainer" containerID="fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13"
Sep 16 19:21:54.975866 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:21:54.975846    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13\": container with ID starting with fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13 not found: ID does not exist" containerID="fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13"
Sep 16 19:21:54.975931 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:54.975873    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13"} err="failed to get container status \"fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13\": rpc error: code = NotFound desc = could not find container \"fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13\": container with ID starting with fe7efe0829ca110c19c07b3238bbd886ccc833c878d3b3055096a0b894986f13 not found: ID does not exist"
Sep 16 19:21:55.020687 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:55.020605    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"]
Sep 16 19:21:55.024742 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:55.024715    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5ff8486764-c9qz4"]
Sep 16 19:21:55.416575 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:21:55.416535    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26970baa-7309-47a9-9246-e9f916a11d21" path="/var/lib/kubelet/pods/26970baa-7309-47a9-9246-e9f916a11d21/volumes"
Sep 16 19:22:04.485515 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.485483    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"]
Sep 16 19:22:04.485858 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.485799    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26970baa-7309-47a9-9246-e9f916a11d21" containerName="main"
Sep 16 19:22:04.485858 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.485810    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26970baa-7309-47a9-9246-e9f916a11d21" containerName="main"
Sep 16 19:22:04.485933 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.485883    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26970baa-7309-47a9-9246-e9f916a11d21" containerName="main"
Sep 16 19:22:04.488833 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.488815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:04.494817 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.494793    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 19:22:04.494913 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.494800    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 19:22:04.496105 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.496087    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-dcns9\""
Sep 16 19:22:04.496195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.496094    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 19:22:04.496195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.496170    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-pmknz\""
Sep 16 19:22:04.514129 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.514104    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"]
Sep 16 19:22:04.664902 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.664862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78jn\" (UniqueName: \"kubernetes.io/projected/76fe1697-592b-4bb5-af14-2b866f7335dc-kube-api-access-h78jn\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4\" (UID: \"76fe1697-592b-4bb5-af14-2b866f7335dc\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:04.665062 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.664920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe1697-592b-4bb5-af14-2b866f7335dc-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4\" (UID: \"76fe1697-592b-4bb5-af14-2b866f7335dc\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:04.766456 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.766347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78jn\" (UniqueName: \"kubernetes.io/projected/76fe1697-592b-4bb5-af14-2b866f7335dc-kube-api-access-h78jn\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4\" (UID: \"76fe1697-592b-4bb5-af14-2b866f7335dc\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:04.766456 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.766392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe1697-592b-4bb5-af14-2b866f7335dc-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4\" (UID: \"76fe1697-592b-4bb5-af14-2b866f7335dc\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:04.768894 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.768869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe1697-592b-4bb5-af14-2b866f7335dc-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4\" (UID: \"76fe1697-592b-4bb5-af14-2b866f7335dc\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:04.777875 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.777850    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78jn\" (UniqueName: \"kubernetes.io/projected/76fe1697-592b-4bb5-af14-2b866f7335dc-kube-api-access-h78jn\") pod \"fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4\" (UID: \"76fe1697-592b-4bb5-af14-2b866f7335dc\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:04.798842 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:04.798817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:05.095929 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:05.095834    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"]
Sep 16 19:22:05.101657 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:22:05.101628    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fe1697_592b_4bb5_af14_2b866f7335dc.slice/crio-79ae04f85aa1e34b7c1bf9ae0cc4e147f15cba3863847e4e72484cf68e5dc4e3 WatchSource:0}: Error finding container 79ae04f85aa1e34b7c1bf9ae0cc4e147f15cba3863847e4e72484cf68e5dc4e3: Status 404 returned error can't find the container with id 79ae04f85aa1e34b7c1bf9ae0cc4e147f15cba3863847e4e72484cf68e5dc4e3
Sep 16 19:22:06.026189 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:06.026156    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" event={"ID":"76fe1697-592b-4bb5-af14-2b866f7335dc","Type":"ContainerStarted","Data":"68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907"}
Sep 16 19:22:06.026590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:06.026219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" event={"ID":"76fe1697-592b-4bb5-af14-2b866f7335dc","Type":"ContainerStarted","Data":"79ae04f85aa1e34b7c1bf9ae0cc4e147f15cba3863847e4e72484cf68e5dc4e3"}
Sep 16 19:22:06.026590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:06.026258    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:06.063593 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:06.063542    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" podStartSLOduration=2.063527445 podStartE2EDuration="2.063527445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:06.060744502 +0000 UTC m=+1003.186548891" watchObservedRunningTime="2026-09-16 19:22:06.063527445 +0000 UTC m=+1003.189331833"
Sep 16 19:22:23.997014 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:23.996981    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:22:37.032710 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:37.032678    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:22:39.020010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.019965    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="21b34306-3b1a-46e8-b400-e85a3a60a4e1" containerName="main" containerID="cri-o://c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988" gracePeriod=45
Sep 16 19:22:39.772741 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.772714    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:22:39.871879 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.871838    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xxx2\" (UniqueName: \"kubernetes.io/projected/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kube-api-access-5xxx2\") pod \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") "
Sep 16 19:22:39.872047 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.871916    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-home\") pod \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") "
Sep 16 19:22:39.872047 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.871966    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kserve-provision-location\") pod \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") "
Sep 16 19:22:39.872047 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.871998    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21b34306-3b1a-46e8-b400-e85a3a60a4e1-cabundle-cert\") pod \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") "
Sep 16 19:22:39.872165 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.872075    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-dshm\") pod \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") "
Sep 16 19:22:39.872165 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.872121    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-model-cache\") pod \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") "
Sep 16 19:22:39.872165 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.872151    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tls-certs\") pod \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") "
Sep 16 19:22:39.872313 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.872172    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tmp-dir\") pod \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\" (UID: \"21b34306-3b1a-46e8-b400-e85a3a60a4e1\") "
Sep 16 19:22:39.872599 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.872466    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b34306-3b1a-46e8-b400-e85a3a60a4e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21b34306-3b1a-46e8-b400-e85a3a60a4e1" (UID: "21b34306-3b1a-46e8-b400-e85a3a60a4e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:39.872599 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.872515    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "21b34306-3b1a-46e8-b400-e85a3a60a4e1" (UID: "21b34306-3b1a-46e8-b400-e85a3a60a4e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:39.872768 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.872745    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:22:39.872832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.872767    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21b34306-3b1a-46e8-b400-e85a3a60a4e1-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:22:39.872832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.872794    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-home" (OuterVolumeSpecName: "home") pod "21b34306-3b1a-46e8-b400-e85a3a60a4e1" (UID: "21b34306-3b1a-46e8-b400-e85a3a60a4e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:39.875080 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.875037    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-dshm" (OuterVolumeSpecName: "dshm") pod "21b34306-3b1a-46e8-b400-e85a3a60a4e1" (UID: "21b34306-3b1a-46e8-b400-e85a3a60a4e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:39.875205 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.875103    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21b34306-3b1a-46e8-b400-e85a3a60a4e1" (UID: "21b34306-3b1a-46e8-b400-e85a3a60a4e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:39.875205 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.875138    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kube-api-access-5xxx2" (OuterVolumeSpecName: "kube-api-access-5xxx2") pod "21b34306-3b1a-46e8-b400-e85a3a60a4e1" (UID: "21b34306-3b1a-46e8-b400-e85a3a60a4e1"). InnerVolumeSpecName "kube-api-access-5xxx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:39.885365 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.885327    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21b34306-3b1a-46e8-b400-e85a3a60a4e1" (UID: "21b34306-3b1a-46e8-b400-e85a3a60a4e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:39.893645 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.893611    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21b34306-3b1a-46e8-b400-e85a3a60a4e1" (UID: "21b34306-3b1a-46e8-b400-e85a3a60a4e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:39.973811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.973775    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:22:39.973811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.973803    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:22:39.973811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.973814    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:22:39.974043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.973822    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xxx2\" (UniqueName: \"kubernetes.io/projected/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kube-api-access-5xxx2\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:22:39.974043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.973832    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:22:39.974043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:39.973840    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b34306-3b1a-46e8-b400-e85a3a60a4e1-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:22:40.194982 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.194892    2583 generic.go:356] "Generic (PLEG): container finished" podID="21b34306-3b1a-46e8-b400-e85a3a60a4e1" containerID="c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988" exitCode=0
Sep 16 19:22:40.194982 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.194968    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:22:40.195524 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.194980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"21b34306-3b1a-46e8-b400-e85a3a60a4e1","Type":"ContainerDied","Data":"c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988"}
Sep 16 19:22:40.195524 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.195062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"21b34306-3b1a-46e8-b400-e85a3a60a4e1","Type":"ContainerDied","Data":"8387b7f8c78a7807f96e3ca9e06d2e6e824fb8253aa223579f93df6b798d4f24"}
Sep 16 19:22:40.195524 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.195087    2583 scope.go:117] "RemoveContainer" containerID="c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988"
Sep 16 19:22:40.203923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.203904    2583 scope.go:117] "RemoveContainer" containerID="148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c"
Sep 16 19:22:40.213809 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.213787    2583 scope.go:117] "RemoveContainer" containerID="c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988"
Sep 16 19:22:40.214107 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:22:40.214086    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988\": container with ID starting with c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988 not found: ID does not exist" containerID="c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988"
Sep 16 19:22:40.214192 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.214120    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988"} err="failed to get container status \"c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988\": rpc error: code = NotFound desc = could not find container \"c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988\": container with ID starting with c73530185fee1dd4608287dca4c735c92d83005e2b06d587c2f081d18377a988 not found: ID does not exist"
Sep 16 19:22:40.214192 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.214145    2583 scope.go:117] "RemoveContainer" containerID="148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c"
Sep 16 19:22:40.214453 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:22:40.214433    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c\": container with ID starting with 148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c not found: ID does not exist" containerID="148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c"
Sep 16 19:22:40.214514 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.214463    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c"} err="failed to get container status \"148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c\": rpc error: code = NotFound desc = could not find container \"148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c\": container with ID starting with 148256a3d40ed7393ca9736c8082da359d5337143234810e3529288ae1f6509c not found: ID does not exist"
Sep 16 19:22:40.229599 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.229566    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:22:40.231138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:40.231115    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:22:41.412092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:41.412058    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b34306-3b1a-46e8-b400-e85a3a60a4e1" path="/var/lib/kubelet/pods/21b34306-3b1a-46e8-b400-e85a3a60a4e1/volumes"
Sep 16 19:22:43.797905 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:43.797875    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"]
Sep 16 19:22:47.033256 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:47.033215    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:57.032664 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:57.032622    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:58.817589 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:22:58.817550    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerName="main" containerID="cri-o://68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907" gracePeriod=45
Sep 16 19:23:07.032779 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:07.032737    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:07.033207 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:07.032863    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:23:13.783150 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.783113    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"]
Sep 16 19:23:13.783546 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.783501    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b34306-3b1a-46e8-b400-e85a3a60a4e1" containerName="storage-initializer"
Sep 16 19:23:13.783546 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.783516    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b34306-3b1a-46e8-b400-e85a3a60a4e1" containerName="storage-initializer"
Sep 16 19:23:13.783546 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.783544    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b34306-3b1a-46e8-b400-e85a3a60a4e1" containerName="main"
Sep 16 19:23:13.783652 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.783549    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b34306-3b1a-46e8-b400-e85a3a60a4e1" containerName="main"
Sep 16 19:23:13.783652 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.783612    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b34306-3b1a-46e8-b400-e85a3a60a4e1" containerName="main"
Sep 16 19:23:13.785713 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.785693    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:13.791833 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.791810    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 19:23:13.791959 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.791880    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-bzbn7\""
Sep 16 19:23:13.793033 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.792845    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 19:23:13.793033 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.792902    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-wn9qw\""
Sep 16 19:23:13.793326 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.793305    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 19:23:13.808909 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.808883    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"]
Sep 16 19:23:13.855615 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.855591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6899ef-035a-42cf-bded-1738e7141618-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs\" (UID: \"6c6899ef-035a-42cf-bded-1738e7141618\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:13.855709 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.855621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltl2d\" (UniqueName: \"kubernetes.io/projected/6c6899ef-035a-42cf-bded-1738e7141618-kube-api-access-ltl2d\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs\" (UID: \"6c6899ef-035a-42cf-bded-1738e7141618\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:13.956387 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.956352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6899ef-035a-42cf-bded-1738e7141618-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs\" (UID: \"6c6899ef-035a-42cf-bded-1738e7141618\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:13.956387 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.956387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltl2d\" (UniqueName: \"kubernetes.io/projected/6c6899ef-035a-42cf-bded-1738e7141618-kube-api-access-ltl2d\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs\" (UID: \"6c6899ef-035a-42cf-bded-1738e7141618\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:13.958859 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.958836    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6899ef-035a-42cf-bded-1738e7141618-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs\" (UID: \"6c6899ef-035a-42cf-bded-1738e7141618\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:13.974529 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:13.974503    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltl2d\" (UniqueName: \"kubernetes.io/projected/6c6899ef-035a-42cf-bded-1738e7141618-kube-api-access-ltl2d\") pod \"scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs\" (UID: \"6c6899ef-035a-42cf-bded-1738e7141618\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:14.096850 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:14.096756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:14.380836 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:14.380806    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"]
Sep 16 19:23:14.385816 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:23:14.385790    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6899ef_035a_42cf_bded_1738e7141618.slice/crio-965362c56a1847e77bfa462948128dbc5a426b2fc031a40bd0dbf563657fc6d0 WatchSource:0}: Error finding container 965362c56a1847e77bfa462948128dbc5a426b2fc031a40bd0dbf563657fc6d0: Status 404 returned error can't find the container with id 965362c56a1847e77bfa462948128dbc5a426b2fc031a40bd0dbf563657fc6d0
Sep 16 19:23:15.370457 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:15.370390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" event={"ID":"6c6899ef-035a-42cf-bded-1738e7141618","Type":"ContainerStarted","Data":"e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf"}
Sep 16 19:23:15.370865 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:15.370476    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:15.370865 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:15.370494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" event={"ID":"6c6899ef-035a-42cf-bded-1738e7141618","Type":"ContainerStarted","Data":"965362c56a1847e77bfa462948128dbc5a426b2fc031a40bd0dbf563657fc6d0"}
Sep 16 19:23:15.397828 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:15.397789    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" podStartSLOduration=2.397775679 podStartE2EDuration="2.397775679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:15.396664867 +0000 UTC m=+1072.522469257" watchObservedRunningTime="2026-09-16 19:23:15.397775679 +0000 UTC m=+1072.523580066"
Sep 16 19:23:17.033764 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:17.033719    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:27.032667 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:27.032625    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:29.059324 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.059300    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:23:29.181824 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.181791    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe1697-592b-4bb5-af14-2b866f7335dc-tls-certs\") pod \"76fe1697-592b-4bb5-af14-2b866f7335dc\" (UID: \"76fe1697-592b-4bb5-af14-2b866f7335dc\") "
Sep 16 19:23:29.181984 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.181910    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h78jn\" (UniqueName: \"kubernetes.io/projected/76fe1697-592b-4bb5-af14-2b866f7335dc-kube-api-access-h78jn\") pod \"76fe1697-592b-4bb5-af14-2b866f7335dc\" (UID: \"76fe1697-592b-4bb5-af14-2b866f7335dc\") "
Sep 16 19:23:29.184152 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.184121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fe1697-592b-4bb5-af14-2b866f7335dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76fe1697-592b-4bb5-af14-2b866f7335dc" (UID: "76fe1697-592b-4bb5-af14-2b866f7335dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:29.184152 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.184141    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fe1697-592b-4bb5-af14-2b866f7335dc-kube-api-access-h78jn" (OuterVolumeSpecName: "kube-api-access-h78jn") pod "76fe1697-592b-4bb5-af14-2b866f7335dc" (UID: "76fe1697-592b-4bb5-af14-2b866f7335dc"). InnerVolumeSpecName "kube-api-access-h78jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:29.282521 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.282490    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78jn\" (UniqueName: \"kubernetes.io/projected/76fe1697-592b-4bb5-af14-2b866f7335dc-kube-api-access-h78jn\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:29.282521 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.282517    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe1697-592b-4bb5-af14-2b866f7335dc-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:29.434124 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.434054    2583 generic.go:356] "Generic (PLEG): container finished" podID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerID="68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907" exitCode=0
Sep 16 19:23:29.434124 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.434102    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"
Sep 16 19:23:29.434317 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.434140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" event={"ID":"76fe1697-592b-4bb5-af14-2b866f7335dc","Type":"ContainerDied","Data":"68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907"}
Sep 16 19:23:29.434317 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.434189    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4" event={"ID":"76fe1697-592b-4bb5-af14-2b866f7335dc","Type":"ContainerDied","Data":"79ae04f85aa1e34b7c1bf9ae0cc4e147f15cba3863847e4e72484cf68e5dc4e3"}
Sep 16 19:23:29.434317 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.434216    2583 scope.go:117] "RemoveContainer" containerID="68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907"
Sep 16 19:23:29.442649 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.442633    2583 scope.go:117] "RemoveContainer" containerID="68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907"
Sep 16 19:23:29.442918 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:23:29.442897    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907\": container with ID starting with 68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907 not found: ID does not exist" containerID="68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907"
Sep 16 19:23:29.442984 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.442925    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907"} err="failed to get container status \"68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907\": rpc error: code = NotFound desc = could not find container \"68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907\": container with ID starting with 68d063c548fb99d1231944aac347f9b4f03d65646051f65bb78a676872037907 not found: ID does not exist"
Sep 16 19:23:29.458622 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.458598    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"]
Sep 16 19:23:29.464590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:29.464562    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-6c7484bc6c-m89h4"]
Sep 16 19:23:31.411421 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:31.411375    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" path="/var/lib/kubelet/pods/76fe1697-592b-4bb5-af14-2b866f7335dc/volumes"
Sep 16 19:23:34.451625 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.451597    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"]
Sep 16 19:23:34.452090 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.452071    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerName="main"
Sep 16 19:23:34.452169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.452093    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerName="main"
Sep 16 19:23:34.452225 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.452189    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fe1697-592b-4bb5-af14-2b866f7335dc" containerName="main"
Sep 16 19:23:34.457278 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.457259    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.461124 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.461087    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 19:23:34.461291 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.461277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 19:23:34.461364 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.461327    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-sh92p\""
Sep 16 19:23:34.461841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.461827    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:23:34.484168 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.484144    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"]
Sep 16 19:23:34.527655 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.527628    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.527759 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.527674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.527759 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.527694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.527759 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.527721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.527899 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.527798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqwl\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-kube-api-access-ngqwl\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.527899 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.527841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.527899 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.527870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.527994 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.527909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.527994 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.527933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.628935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.628908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqwl\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-kube-api-access-ngqwl\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629069 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.628951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629069 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.628977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629457 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629457 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629570 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629523    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629570 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629562    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629672 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629672 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.629934 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.629917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.631481 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.631446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.631891 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.631872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.657404 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.657383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.663598 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.663575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqwl\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-kube-api-access-ngqwl\") pod \"router-gateway-1-openshift-default-5b4d9458bd-75bdz\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:34.767094 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:34.767039    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:35.065351 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:35.065329    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"]
Sep 16 19:23:35.067755 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:23:35.067727    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8c7082_8b2e_4aa8_a81a_f90d743f22d3.slice/crio-bb4d83d3d06b349c1f4d058a496c44455184e3a028270ff08aa78784adf17dcd WatchSource:0}: Error finding container bb4d83d3d06b349c1f4d058a496c44455184e3a028270ff08aa78784adf17dcd: Status 404 returned error can't find the container with id bb4d83d3d06b349c1f4d058a496c44455184e3a028270ff08aa78784adf17dcd
Sep 16 19:23:35.069905 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:35.069876    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:23:35.069980 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:35.069938    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:23:35.069980 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:35.069962    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:23:35.465620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:35.465585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz" event={"ID":"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3","Type":"ContainerStarted","Data":"23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706"}
Sep 16 19:23:35.466076 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:35.465643    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz" event={"ID":"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3","Type":"ContainerStarted","Data":"bb4d83d3d06b349c1f4d058a496c44455184e3a028270ff08aa78784adf17dcd"}
Sep 16 19:23:35.543002 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:35.542959    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz" podStartSLOduration=1.542943715 podStartE2EDuration="1.542943715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:35.539388895 +0000 UTC m=+1092.665193283" watchObservedRunningTime="2026-09-16 19:23:35.542943715 +0000 UTC m=+1092.668748103"
Sep 16 19:23:35.767721 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:35.767650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:36.773473 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:36.773432    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:37.474656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:37.474620    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:37.475596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:37.475576    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:43.393670 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.393639    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"]
Sep 16 19:23:43.409799 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.409741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.424450 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.417908    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 19:23:43.424450 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.418217    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:23:43.424450 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.418345    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-fcnpc\""
Sep 16 19:23:43.424450 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.421575    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 19:23:43.472035 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.471994    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"]
Sep 16 19:23:43.521608 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.521564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.521896 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.521870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17401564-ae27-45a4-99d9-e9cf8b8b1066-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.522030 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.522013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt8mk\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-kube-api-access-dt8mk\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.522146 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.522132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.522266 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.522250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.522427 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.522391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.522647 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.522627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.522808 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.522791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.522964 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.522947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624076 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624375 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17401564-ae27-45a4-99d9-e9cf8b8b1066-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt8mk\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-kube-api-access-dt8mk\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.624801 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.624688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.625170 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.625132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.625452 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.625427    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.625833 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.625807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17401564-ae27-45a4-99d9-e9cf8b8b1066-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.626202 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.626182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.629767 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.629721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.630186 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.630165    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.638681 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.638657    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.640945 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.640917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt8mk\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-kube-api-access-dt8mk\") pod \"router-gateway-1-openshift-default-66bbccc5fb-wwmvp\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:43.739546 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:43.739511    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:44.045710 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:44.045680    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"]
Sep 16 19:23:44.046903 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:23:44.046878    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17401564_ae27_45a4_99d9_e9cf8b8b1066.slice/crio-95796c8e8a7da8a2372a2d1ba74a464c221d56c900c0a8b095f5264c02118ad7 WatchSource:0}: Error finding container 95796c8e8a7da8a2372a2d1ba74a464c221d56c900c0a8b095f5264c02118ad7: Status 404 returned error can't find the container with id 95796c8e8a7da8a2372a2d1ba74a464c221d56c900c0a8b095f5264c02118ad7
Sep 16 19:23:44.048987 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:44.048953    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:23:44.049094 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:44.049032    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:23:44.049094 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:44.049083    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:23:44.521774 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:44.521735    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp" event={"ID":"17401564-ae27-45a4-99d9-e9cf8b8b1066","Type":"ContainerStarted","Data":"ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665"}
Sep 16 19:23:44.522338 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:44.522314    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp" event={"ID":"17401564-ae27-45a4-99d9-e9cf8b8b1066","Type":"ContainerStarted","Data":"95796c8e8a7da8a2372a2d1ba74a464c221d56c900c0a8b095f5264c02118ad7"}
Sep 16 19:23:44.558833 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:44.558783    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp" podStartSLOduration=1.5587661000000002 podStartE2EDuration="1.5587661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:44.55624505 +0000 UTC m=+1101.682049437" watchObservedRunningTime="2026-09-16 19:23:44.5587661 +0000 UTC m=+1101.684570487"
Sep 16 19:23:44.740455 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:44.740423    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:45.745735 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:45.745701    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:46.377388 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:46.377361    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:23:46.535454 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:46.535424    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:46.536865 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:46.536838    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:23:47.868713 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:47.868657    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"]
Sep 16 19:23:47.869299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:47.869261    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz" podUID="ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" containerName="istio-proxy" containerID="cri-o://23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706" gracePeriod=30
Sep 16 19:23:52.113559 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.113499    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:52.224363 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224279    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-data\") pod \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") "
Sep 16 19:23:52.224506 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224402    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-socket\") pod \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") "
Sep 16 19:23:52.224506 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224473    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istiod-ca-cert\") pod \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") "
Sep 16 19:23:52.224506 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224497    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngqwl\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-kube-api-access-ngqwl\") pod \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") "
Sep 16 19:23:52.224685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224517    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" (UID: "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:52.224685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224550    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-certs\") pod \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") "
Sep 16 19:23:52.224685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224568    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-token\") pod \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") "
Sep 16 19:23:52.224685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224589    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-podinfo\") pod \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") "
Sep 16 19:23:52.224685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224647    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-credential-socket\") pod \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") "
Sep 16 19:23:52.224685 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224656    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" (UID: "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:52.224992 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224704    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-envoy\") pod \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\" (UID: \"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3\") "
Sep 16 19:23:52.224992 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224829    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" (UID: "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:52.224992 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224872    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" (UID: "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:52.224992 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.224961    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" (UID: "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:52.225361 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.225027    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-data\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:52.225361 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.225046    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:52.225361 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.225058    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-workload-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:52.226873 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.226850    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-kube-api-access-ngqwl" (OuterVolumeSpecName: "kube-api-access-ngqwl") pod "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" (UID: "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3"). InnerVolumeSpecName "kube-api-access-ngqwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:52.226967 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.226917    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" (UID: "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:52.226967 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.226950    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" (UID: "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:52.227053 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.227031    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" (UID: "ad8c7082-8b2e-4aa8-a81a-f90d743f22d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:23:52.325701 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.325666    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-credential-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:52.325701 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.325701    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-envoy\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:52.325701 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.325711    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istiod-ca-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:52.325869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.325719    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngqwl\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-kube-api-access-ngqwl\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:52.325869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.325727    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-token\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:52.325869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.325735    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3-istio-podinfo\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:23:52.568399 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.568335    2583 generic.go:356] "Generic (PLEG): container finished" podID="ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" containerID="23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706" exitCode=0
Sep 16 19:23:52.568399 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.568380    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"
Sep 16 19:23:52.568572 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.568379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz" event={"ID":"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3","Type":"ContainerDied","Data":"23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706"}
Sep 16 19:23:52.568572 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.568551    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz" event={"ID":"ad8c7082-8b2e-4aa8-a81a-f90d743f22d3","Type":"ContainerDied","Data":"bb4d83d3d06b349c1f4d058a496c44455184e3a028270ff08aa78784adf17dcd"}
Sep 16 19:23:52.568673 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.568587    2583 scope.go:117] "RemoveContainer" containerID="23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706"
Sep 16 19:23:52.578184 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.578168    2583 scope.go:117] "RemoveContainer" containerID="23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706"
Sep 16 19:23:52.578503 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:23:52.578486    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706\": container with ID starting with 23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706 not found: ID does not exist" containerID="23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706"
Sep 16 19:23:52.578563 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.578511    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706"} err="failed to get container status \"23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706\": rpc error: code = NotFound desc = could not find container \"23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706\": container with ID starting with 23651abf71cd0ba9bb647ddd3e47ac7c8d7073cc77a9985b5b4893c6a0253706 not found: ID does not exist"
Sep 16 19:23:52.599436 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.599394    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"]
Sep 16 19:23:52.634111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:52.634090    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-75bdz"]
Sep 16 19:23:53.413484 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:23:53.413452    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" path="/var/lib/kubelet/pods/ad8c7082-8b2e-4aa8-a81a-f90d743f22d3/volumes"
Sep 16 19:24:00.259697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:00.259642    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"]
Sep 16 19:24:00.260174 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:00.259853    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp" podUID="17401564-ae27-45a4-99d9-e9cf8b8b1066" containerName="istio-proxy" containerID="cri-o://ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665" gracePeriod=30
Sep 16 19:24:02.075170 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.075130    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp" podUID="17401564-ae27-45a4-99d9-e9cf8b8b1066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": read tcp 10.134.0.2:58196->10.134.0.34:15021: read: connection reset by peer"
Sep 16 19:24:02.307545 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.307524    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:24:02.416582 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17401564-ae27-45a4-99d9-e9cf8b8b1066-istiod-ca-cert\") pod \"17401564-ae27-45a4-99d9-e9cf8b8b1066\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") "
Sep 16 19:24:02.416688 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416635    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt8mk\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-kube-api-access-dt8mk\") pod \"17401564-ae27-45a4-99d9-e9cf8b8b1066\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") "
Sep 16 19:24:02.416688 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416674    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-token\") pod \"17401564-ae27-45a4-99d9-e9cf8b8b1066\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") "
Sep 16 19:24:02.416764 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416713    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-data\") pod \"17401564-ae27-45a4-99d9-e9cf8b8b1066\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") "
Sep 16 19:24:02.416764 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416751    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-certs\") pod \"17401564-ae27-45a4-99d9-e9cf8b8b1066\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") "
Sep 16 19:24:02.416855 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416779    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-credential-socket\") pod \"17401564-ae27-45a4-99d9-e9cf8b8b1066\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") "
Sep 16 19:24:02.416855 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-envoy\") pod \"17401564-ae27-45a4-99d9-e9cf8b8b1066\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") "
Sep 16 19:24:02.416965 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-podinfo\") pod \"17401564-ae27-45a4-99d9-e9cf8b8b1066\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") "
Sep 16 19:24:02.416965 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416901    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-socket\") pod \"17401564-ae27-45a4-99d9-e9cf8b8b1066\" (UID: \"17401564-ae27-45a4-99d9-e9cf8b8b1066\") "
Sep 16 19:24:02.416965 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.416913    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17401564-ae27-45a4-99d9-e9cf8b8b1066-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17401564-ae27-45a4-99d9-e9cf8b8b1066" (UID: "17401564-ae27-45a4-99d9-e9cf8b8b1066"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:02.417258 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.417136    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-data" (OuterVolumeSpecName: "istio-data") pod "17401564-ae27-45a4-99d9-e9cf8b8b1066" (UID: "17401564-ae27-45a4-99d9-e9cf8b8b1066"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:02.417258 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.417153    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17401564-ae27-45a4-99d9-e9cf8b8b1066" (UID: "17401564-ae27-45a4-99d9-e9cf8b8b1066"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:02.417258 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.417208    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17401564-ae27-45a4-99d9-e9cf8b8b1066-istiod-ca-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.417496 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.417356    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17401564-ae27-45a4-99d9-e9cf8b8b1066" (UID: "17401564-ae27-45a4-99d9-e9cf8b8b1066"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:02.417496 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.417479    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17401564-ae27-45a4-99d9-e9cf8b8b1066" (UID: "17401564-ae27-45a4-99d9-e9cf8b8b1066"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:02.419125 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.419101    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-kube-api-access-dt8mk" (OuterVolumeSpecName: "kube-api-access-dt8mk") pod "17401564-ae27-45a4-99d9-e9cf8b8b1066" (UID: "17401564-ae27-45a4-99d9-e9cf8b8b1066"). InnerVolumeSpecName "kube-api-access-dt8mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:02.419268 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.419229    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-token" (OuterVolumeSpecName: "istio-token") pod "17401564-ae27-45a4-99d9-e9cf8b8b1066" (UID: "17401564-ae27-45a4-99d9-e9cf8b8b1066"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:02.419325 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.419277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17401564-ae27-45a4-99d9-e9cf8b8b1066" (UID: "17401564-ae27-45a4-99d9-e9cf8b8b1066"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:02.419450 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.419406    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17401564-ae27-45a4-99d9-e9cf8b8b1066" (UID: "17401564-ae27-45a4-99d9-e9cf8b8b1066"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:24:02.517863 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.517843    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-envoy\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.517863 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.517865    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-podinfo\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.518010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.517874    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.518010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.517883    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt8mk\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-kube-api-access-dt8mk\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.518010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.517892    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-token\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.518010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.517899    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-istio-data\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.518010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.517906    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-workload-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.518010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.517914    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17401564-ae27-45a4-99d9-e9cf8b8b1066-credential-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.619727 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.619699    2583 generic.go:356] "Generic (PLEG): container finished" podID="17401564-ae27-45a4-99d9-e9cf8b8b1066" containerID="ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665" exitCode=0
Sep 16 19:24:02.619852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.619749    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"
Sep 16 19:24:02.619852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.619751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp" event={"ID":"17401564-ae27-45a4-99d9-e9cf8b8b1066","Type":"ContainerDied","Data":"ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665"}
Sep 16 19:24:02.619852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.619810    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp" event={"ID":"17401564-ae27-45a4-99d9-e9cf8b8b1066","Type":"ContainerDied","Data":"95796c8e8a7da8a2372a2d1ba74a464c221d56c900c0a8b095f5264c02118ad7"}
Sep 16 19:24:02.619852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.619843    2583 scope.go:117] "RemoveContainer" containerID="ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665"
Sep 16 19:24:02.631563 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.630884    2583 scope.go:117] "RemoveContainer" containerID="ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665"
Sep 16 19:24:02.631563 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:24:02.631541    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665\": container with ID starting with ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665 not found: ID does not exist" containerID="ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665"
Sep 16 19:24:02.631666 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.631576    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665"} err="failed to get container status \"ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665\": rpc error: code = NotFound desc = could not find container \"ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665\": container with ID starting with ac1b433cc5185b69aaf280dc7555254186410a80a4135566ccd3988e3280a665 not found: ID does not exist"
Sep 16 19:24:02.657060 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.657035    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"]
Sep 16 19:24:02.662753 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:02.662735    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-wwmvp"]
Sep 16 19:24:03.412709 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:03.412681    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17401564-ae27-45a4-99d9-e9cf8b8b1066" path="/var/lib/kubelet/pods/17401564-ae27-45a4-99d9-e9cf8b8b1066/volumes"
Sep 16 19:24:07.237498 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.237465    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"]
Sep 16 19:24:07.237841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.237787    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17401564-ae27-45a4-99d9-e9cf8b8b1066" containerName="istio-proxy"
Sep 16 19:24:07.237841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.237797    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="17401564-ae27-45a4-99d9-e9cf8b8b1066" containerName="istio-proxy"
Sep 16 19:24:07.237841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.237821    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" containerName="istio-proxy"
Sep 16 19:24:07.237841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.237826    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" containerName="istio-proxy"
Sep 16 19:24:07.237976 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.237876    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="17401564-ae27-45a4-99d9-e9cf8b8b1066" containerName="istio-proxy"
Sep 16 19:24:07.237976 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.237886    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8c7082-8b2e-4aa8-a81a-f90d743f22d3" containerName="istio-proxy"
Sep 16 19:24:07.241234 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.241216    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.244215 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.244192    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 19:24:07.244314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.244224    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:24:07.244520 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.244503    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 19:24:07.244569 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.244514    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-gr6dd\""
Sep 16 19:24:07.245353 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.245337    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 19:24:07.245467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.245449    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 19:24:07.269929 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.269902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"]
Sep 16 19:24:07.349690 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.349665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-kserve-provision-location\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.349795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.349705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmdv\" (UniqueName: \"kubernetes.io/projected/08fac061-8c9f-414c-a062-6e61b4de6bfe-kube-api-access-vhmdv\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.349795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.349745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fac061-8c9f-414c-a062-6e61b4de6bfe-tls-certs\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.349795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.349769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-model-cache\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.349795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.349783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08fac061-8c9f-414c-a062-6e61b4de6bfe-cabundle-cert\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.349962 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.349816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-home\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.349962 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.349836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-tmp-dir\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.349962 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.349853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-dshm\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450363 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmdv\" (UniqueName: \"kubernetes.io/projected/08fac061-8c9f-414c-a062-6e61b4de6bfe-kube-api-access-vhmdv\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450514 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fac061-8c9f-414c-a062-6e61b4de6bfe-tls-certs\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450514 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-model-cache\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450514 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08fac061-8c9f-414c-a062-6e61b4de6bfe-cabundle-cert\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-home\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-tmp-dir\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450767 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-dshm\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450767 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-kserve-provision-location\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450934 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-home\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.450934 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.450897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-model-cache\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.451113 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.451093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-kserve-provision-location\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.451167 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.451114    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-tmp-dir\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.451271 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.451248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08fac061-8c9f-414c-a062-6e61b4de6bfe-cabundle-cert\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.453261 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.453241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fac061-8c9f-414c-a062-6e61b4de6bfe-tls-certs\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.453516 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.453497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-dshm\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.460674 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.460656    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmdv\" (UniqueName: \"kubernetes.io/projected/08fac061-8c9f-414c-a062-6e61b4de6bfe-kube-api-access-vhmdv\") pod \"auth-enabled-test-kserve-68b7f474bf-k79m7\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.551184 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.551118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:07.850889 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:07.850865    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"]
Sep 16 19:24:07.853480 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:24:07.853450    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fac061_8c9f_414c_a062_6e61b4de6bfe.slice/crio-78f8f531e1c69d55208061df1dc336093e00a846a5015beb08685d0df838f4ad WatchSource:0}: Error finding container 78f8f531e1c69d55208061df1dc336093e00a846a5015beb08685d0df838f4ad: Status 404 returned error can't find the container with id 78f8f531e1c69d55208061df1dc336093e00a846a5015beb08685d0df838f4ad
Sep 16 19:24:08.653174 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:08.652297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" event={"ID":"08fac061-8c9f-414c-a062-6e61b4de6bfe","Type":"ContainerStarted","Data":"8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10"}
Sep 16 19:24:08.653174 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:08.652362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" event={"ID":"08fac061-8c9f-414c-a062-6e61b4de6bfe","Type":"ContainerStarted","Data":"78f8f531e1c69d55208061df1dc336093e00a846a5015beb08685d0df838f4ad"}
Sep 16 19:24:10.225162 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:10.225130    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"]
Sep 16 19:24:10.664688 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:10.664660    2583 generic.go:356] "Generic (PLEG): container finished" podID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerID="8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10" exitCode=0
Sep 16 19:24:10.664843 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:10.664697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" event={"ID":"08fac061-8c9f-414c-a062-6e61b4de6bfe","Type":"ContainerDied","Data":"8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10"}
Sep 16 19:24:11.671955 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:11.671912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" event={"ID":"08fac061-8c9f-414c-a062-6e61b4de6bfe","Type":"ContainerStarted","Data":"a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6"}
Sep 16 19:24:11.712555 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:11.712500    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podStartSLOduration=2.53660318 podStartE2EDuration="4.712480585s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.175877405s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:11.709725431 +0000 UTC m=+1128.835529819" watchObservedRunningTime="2026-09-16 19:24:11.712480585 +0000 UTC m=+1128.838284976"
Sep 16 19:24:12.551616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:12.551584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:16.377540 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:16.377493    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" podUID="6c6899ef-035a-42cf-bded-1738e7141618" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:24:17.551613 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:17.551581    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:24:17.552851 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:17.552828    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:24:25.244002 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:25.243940    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" podUID="6c6899ef-035a-42cf-bded-1738e7141618" containerName="main" containerID="cri-o://e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf" gracePeriod=45
Sep 16 19:24:26.376603 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:26.376565    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" podUID="6c6899ef-035a-42cf-bded-1738e7141618" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:24:27.551812 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:27.551769    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:24:36.376639 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:36.376594    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" podUID="6c6899ef-035a-42cf-bded-1738e7141618" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:24:36.377035 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:36.376736    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:24:37.551717 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:37.551664    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:24:46.376890 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:46.376796    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" podUID="6c6899ef-035a-42cf-bded-1738e7141618" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:24:47.551590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:47.551551    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:24:55.499796 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.499764    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:24:55.577497 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.577393    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltl2d\" (UniqueName: \"kubernetes.io/projected/6c6899ef-035a-42cf-bded-1738e7141618-kube-api-access-ltl2d\") pod \"6c6899ef-035a-42cf-bded-1738e7141618\" (UID: \"6c6899ef-035a-42cf-bded-1738e7141618\") "
Sep 16 19:24:55.577497 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.577488    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6899ef-035a-42cf-bded-1738e7141618-tls-certs\") pod \"6c6899ef-035a-42cf-bded-1738e7141618\" (UID: \"6c6899ef-035a-42cf-bded-1738e7141618\") "
Sep 16 19:24:55.579879 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.579848    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6899ef-035a-42cf-bded-1738e7141618-kube-api-access-ltl2d" (OuterVolumeSpecName: "kube-api-access-ltl2d") pod "6c6899ef-035a-42cf-bded-1738e7141618" (UID: "6c6899ef-035a-42cf-bded-1738e7141618"). InnerVolumeSpecName "kube-api-access-ltl2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:55.579879 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.579864    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6899ef-035a-42cf-bded-1738e7141618-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c6899ef-035a-42cf-bded-1738e7141618" (UID: "6c6899ef-035a-42cf-bded-1738e7141618"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:55.678819 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.678777    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltl2d\" (UniqueName: \"kubernetes.io/projected/6c6899ef-035a-42cf-bded-1738e7141618-kube-api-access-ltl2d\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:55.678819 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.678811    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6899ef-035a-42cf-bded-1738e7141618-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:24:55.904655 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.904617    2583 generic.go:356] "Generic (PLEG): container finished" podID="6c6899ef-035a-42cf-bded-1738e7141618" containerID="e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf" exitCode=0
Sep 16 19:24:55.904830 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.904661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" event={"ID":"6c6899ef-035a-42cf-bded-1738e7141618","Type":"ContainerDied","Data":"e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf"}
Sep 16 19:24:55.904830 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.904704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs" event={"ID":"6c6899ef-035a-42cf-bded-1738e7141618","Type":"ContainerDied","Data":"965362c56a1847e77bfa462948128dbc5a426b2fc031a40bd0dbf563657fc6d0"}
Sep 16 19:24:55.904830 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.904726    2583 scope.go:117] "RemoveContainer" containerID="e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf"
Sep 16 19:24:55.904830 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.904664    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"
Sep 16 19:24:55.914742 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.914721    2583 scope.go:117] "RemoveContainer" containerID="e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf"
Sep 16 19:24:55.915047 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:24:55.915030    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf\": container with ID starting with e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf not found: ID does not exist" containerID="e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf"
Sep 16 19:24:55.915100 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.915056    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf"} err="failed to get container status \"e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf\": rpc error: code = NotFound desc = could not find container \"e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf\": container with ID starting with e1e7a365808073e13cacfb5f1f327bf60ad94821233f1cafff653c8d623609cf not found: ID does not exist"
Sep 16 19:24:55.933936 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.933899    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"]
Sep 16 19:24:55.940256 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:55.940229    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-57946nf2zs"]
Sep 16 19:24:57.411667 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:57.411632    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6899ef-035a-42cf-bded-1738e7141618" path="/var/lib/kubelet/pods/6c6899ef-035a-42cf-bded-1738e7141618/volumes"
Sep 16 19:24:57.552012 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:24:57.551973    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:25:03.462529 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.462492    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"]
Sep 16 19:25:03.463043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.463027    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6899ef-035a-42cf-bded-1738e7141618" containerName="main"
Sep 16 19:25:03.463096 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.463047    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6899ef-035a-42cf-bded-1738e7141618" containerName="main"
Sep 16 19:25:03.463147 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.463136    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6899ef-035a-42cf-bded-1738e7141618" containerName="main"
Sep 16 19:25:03.467077 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.467051    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:03.472322 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.472296    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 19:25:03.472322 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.472316    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-l9kb6\""
Sep 16 19:25:03.472553 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.472338    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-mgmtq\""
Sep 16 19:25:03.472553 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.472349    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 19:25:03.472553 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.472448    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 19:25:03.479424 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.479387    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"]
Sep 16 19:25:03.555823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.555788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde04ed5-e377-4d81-b035-ff509129fa40-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55\" (UID: \"cde04ed5-e377-4d81-b035-ff509129fa40\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:03.555994 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.555881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65b68\" (UniqueName: \"kubernetes.io/projected/cde04ed5-e377-4d81-b035-ff509129fa40-kube-api-access-65b68\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55\" (UID: \"cde04ed5-e377-4d81-b035-ff509129fa40\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:03.657373 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.657321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde04ed5-e377-4d81-b035-ff509129fa40-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55\" (UID: \"cde04ed5-e377-4d81-b035-ff509129fa40\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:03.657576 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.657404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65b68\" (UniqueName: \"kubernetes.io/projected/cde04ed5-e377-4d81-b035-ff509129fa40-kube-api-access-65b68\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55\" (UID: \"cde04ed5-e377-4d81-b035-ff509129fa40\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:03.660300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.660265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde04ed5-e377-4d81-b035-ff509129fa40-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55\" (UID: \"cde04ed5-e377-4d81-b035-ff509129fa40\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:03.666920 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.666891    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65b68\" (UniqueName: \"kubernetes.io/projected/cde04ed5-e377-4d81-b035-ff509129fa40-kube-api-access-65b68\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55\" (UID: \"cde04ed5-e377-4d81-b035-ff509129fa40\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:03.777509 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:03.777380    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:04.074967 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:04.074743    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"]
Sep 16 19:25:04.077855 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:25:04.077821    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde04ed5_e377_4d81_b035_ff509129fa40.slice/crio-094fc34fbe68f7e23a4941d644520a2644b3e418c8ad58167591970da964fea4 WatchSource:0}: Error finding container 094fc34fbe68f7e23a4941d644520a2644b3e418c8ad58167591970da964fea4: Status 404 returned error can't find the container with id 094fc34fbe68f7e23a4941d644520a2644b3e418c8ad58167591970da964fea4
Sep 16 19:25:04.951861 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:04.951824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" event={"ID":"cde04ed5-e377-4d81-b035-ff509129fa40","Type":"ContainerStarted","Data":"1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df"}
Sep 16 19:25:04.952280 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:04.951875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" event={"ID":"cde04ed5-e377-4d81-b035-ff509129fa40","Type":"ContainerStarted","Data":"094fc34fbe68f7e23a4941d644520a2644b3e418c8ad58167591970da964fea4"}
Sep 16 19:25:04.952280 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:04.951962    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:04.976735 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:04.976674    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" podStartSLOduration=1.976651554 podStartE2EDuration="1.976651554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:04.974264192 +0000 UTC m=+1182.100068559" watchObservedRunningTime="2026-09-16 19:25:04.976651554 +0000 UTC m=+1182.102455943"
Sep 16 19:25:07.552281 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:07.552229    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:25:17.552345 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:17.552293    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:25:26.751518 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:26.751483    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:25:26.753995 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:26.753972    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:25:26.757492 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:26.757467    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:25:26.759463 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:26.759445    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:25:27.551967 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:27.551920    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:25:35.959012 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:35.958979    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:25:37.551871 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:37.551825    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:25:37.702683 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:37.702002    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"]
Sep 16 19:25:45.958610 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:45.958573    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" podUID="cde04ed5-e377-4d81-b035-ff509129fa40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:47.551681 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:47.551636    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:25:52.728927 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:52.728857    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" podUID="cde04ed5-e377-4d81-b035-ff509129fa40" containerName="main" containerID="cri-o://1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df" gracePeriod=45
Sep 16 19:25:55.958567 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:55.958524    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" podUID="cde04ed5-e377-4d81-b035-ff509129fa40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:57.552049 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:25:57.551989    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:26:05.959225 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:05.959190    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" podUID="cde04ed5-e377-4d81-b035-ff509129fa40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:05.959605 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:05.959294    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:26:07.561606 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:07.561568    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:26:07.569386 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:07.569362    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:26:12.188702 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:12.188667    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"]
Sep 16 19:26:15.958627 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:15.958529    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" podUID="cde04ed5-e377-4d81-b035-ff509129fa40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:22.982815 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:22.982789    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:26:23.075129 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.075069    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65b68\" (UniqueName: \"kubernetes.io/projected/cde04ed5-e377-4d81-b035-ff509129fa40-kube-api-access-65b68\") pod \"cde04ed5-e377-4d81-b035-ff509129fa40\" (UID: \"cde04ed5-e377-4d81-b035-ff509129fa40\") "
Sep 16 19:26:23.075241 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.075141    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde04ed5-e377-4d81-b035-ff509129fa40-tls-certs\") pod \"cde04ed5-e377-4d81-b035-ff509129fa40\" (UID: \"cde04ed5-e377-4d81-b035-ff509129fa40\") "
Sep 16 19:26:23.077342 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.077311    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde04ed5-e377-4d81-b035-ff509129fa40-kube-api-access-65b68" (OuterVolumeSpecName: "kube-api-access-65b68") pod "cde04ed5-e377-4d81-b035-ff509129fa40" (UID: "cde04ed5-e377-4d81-b035-ff509129fa40"). InnerVolumeSpecName "kube-api-access-65b68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:23.077460 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.077351    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde04ed5-e377-4d81-b035-ff509129fa40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cde04ed5-e377-4d81-b035-ff509129fa40" (UID: "cde04ed5-e377-4d81-b035-ff509129fa40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:23.175879 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.175856    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde04ed5-e377-4d81-b035-ff509129fa40-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:23.175879 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.175879    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65b68\" (UniqueName: \"kubernetes.io/projected/cde04ed5-e377-4d81-b035-ff509129fa40-kube-api-access-65b68\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:23.340540 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.340477    2583 generic.go:356] "Generic (PLEG): container finished" podID="cde04ed5-e377-4d81-b035-ff509129fa40" containerID="1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df" exitCode=0
Sep 16 19:26:23.340540 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.340517    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"
Sep 16 19:26:23.340708 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.340569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" event={"ID":"cde04ed5-e377-4d81-b035-ff509129fa40","Type":"ContainerDied","Data":"1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df"}
Sep 16 19:26:23.340708 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.340626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55" event={"ID":"cde04ed5-e377-4d81-b035-ff509129fa40","Type":"ContainerDied","Data":"094fc34fbe68f7e23a4941d644520a2644b3e418c8ad58167591970da964fea4"}
Sep 16 19:26:23.340708 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.340655    2583 scope.go:117] "RemoveContainer" containerID="1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df"
Sep 16 19:26:23.351232 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.351214    2583 scope.go:117] "RemoveContainer" containerID="1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df"
Sep 16 19:26:23.351544 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:26:23.351522    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df\": container with ID starting with 1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df not found: ID does not exist" containerID="1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df"
Sep 16 19:26:23.351644 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.351552    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df"} err="failed to get container status \"1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df\": rpc error: code = NotFound desc = could not find container \"1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df\": container with ID starting with 1e28074928275d5d08083c124d23392feec4494e7bc9b796bedea86549c7c9df not found: ID does not exist"
Sep 16 19:26:23.375492 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.375471    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"]
Sep 16 19:26:23.382813 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.382791    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxx55"]
Sep 16 19:26:23.412665 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:23.412643    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde04ed5-e377-4d81-b035-ff509129fa40" path="/var/lib/kubelet/pods/cde04ed5-e377-4d81-b035-ff509129fa40/volumes"
Sep 16 19:26:27.211838 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:27.211795    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" containerID="cri-o://a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6" gracePeriod=45
Sep 16 19:26:27.562764 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:27.562672    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 16 19:26:27.872503 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:27.872482    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:26:28.018314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018287    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08fac061-8c9f-414c-a062-6e61b4de6bfe-cabundle-cert\") pod \"08fac061-8c9f-414c-a062-6e61b4de6bfe\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") "
Sep 16 19:26:28.018479 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018327    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-home\") pod \"08fac061-8c9f-414c-a062-6e61b4de6bfe\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") "
Sep 16 19:26:28.018479 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018352    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-tmp-dir\") pod \"08fac061-8c9f-414c-a062-6e61b4de6bfe\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") "
Sep 16 19:26:28.018479 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018398    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhmdv\" (UniqueName: \"kubernetes.io/projected/08fac061-8c9f-414c-a062-6e61b4de6bfe-kube-api-access-vhmdv\") pod \"08fac061-8c9f-414c-a062-6e61b4de6bfe\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") "
Sep 16 19:26:28.018479 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018467    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-kserve-provision-location\") pod \"08fac061-8c9f-414c-a062-6e61b4de6bfe\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") "
Sep 16 19:26:28.018702 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018508    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fac061-8c9f-414c-a062-6e61b4de6bfe-tls-certs\") pod \"08fac061-8c9f-414c-a062-6e61b4de6bfe\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") "
Sep 16 19:26:28.018702 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018532    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-model-cache\") pod \"08fac061-8c9f-414c-a062-6e61b4de6bfe\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") "
Sep 16 19:26:28.018702 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018555    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-dshm\") pod \"08fac061-8c9f-414c-a062-6e61b4de6bfe\" (UID: \"08fac061-8c9f-414c-a062-6e61b4de6bfe\") "
Sep 16 19:26:28.018847 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018810    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-model-cache" (OuterVolumeSpecName: "model-cache") pod "08fac061-8c9f-414c-a062-6e61b4de6bfe" (UID: "08fac061-8c9f-414c-a062-6e61b4de6bfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:28.018995 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.018975    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fac061-8c9f-414c-a062-6e61b4de6bfe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08fac061-8c9f-414c-a062-6e61b4de6bfe" (UID: "08fac061-8c9f-414c-a062-6e61b4de6bfe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:28.019093 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.019080    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-home" (OuterVolumeSpecName: "home") pod "08fac061-8c9f-414c-a062-6e61b4de6bfe" (UID: "08fac061-8c9f-414c-a062-6e61b4de6bfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:28.021065 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.021045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-dshm" (OuterVolumeSpecName: "dshm") pod "08fac061-8c9f-414c-a062-6e61b4de6bfe" (UID: "08fac061-8c9f-414c-a062-6e61b4de6bfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:28.021236 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.021217    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fac061-8c9f-414c-a062-6e61b4de6bfe-kube-api-access-vhmdv" (OuterVolumeSpecName: "kube-api-access-vhmdv") pod "08fac061-8c9f-414c-a062-6e61b4de6bfe" (UID: "08fac061-8c9f-414c-a062-6e61b4de6bfe"). InnerVolumeSpecName "kube-api-access-vhmdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:28.021470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.021441    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fac061-8c9f-414c-a062-6e61b4de6bfe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08fac061-8c9f-414c-a062-6e61b4de6bfe" (UID: "08fac061-8c9f-414c-a062-6e61b4de6bfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:28.036668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.036638    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08fac061-8c9f-414c-a062-6e61b4de6bfe" (UID: "08fac061-8c9f-414c-a062-6e61b4de6bfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:28.046495 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.046473    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08fac061-8c9f-414c-a062-6e61b4de6bfe" (UID: "08fac061-8c9f-414c-a062-6e61b4de6bfe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:28.119810 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.119792    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhmdv\" (UniqueName: \"kubernetes.io/projected/08fac061-8c9f-414c-a062-6e61b4de6bfe-kube-api-access-vhmdv\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:28.119897 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.119813    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:28.119897 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.119823    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fac061-8c9f-414c-a062-6e61b4de6bfe-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:28.119897 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.119834    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:28.119897 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.119841    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:28.119897 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.119848    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08fac061-8c9f-414c-a062-6e61b4de6bfe-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:28.119897 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.119856    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:28.119897 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.119864    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fac061-8c9f-414c-a062-6e61b4de6bfe-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:26:28.365467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.365437    2583 generic.go:356] "Generic (PLEG): container finished" podID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerID="a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6" exitCode=0
Sep 16 19:26:28.365869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.365486    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"
Sep 16 19:26:28.365869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.365505    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" event={"ID":"08fac061-8c9f-414c-a062-6e61b4de6bfe","Type":"ContainerDied","Data":"a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6"}
Sep 16 19:26:28.365869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.365565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7" event={"ID":"08fac061-8c9f-414c-a062-6e61b4de6bfe","Type":"ContainerDied","Data":"78f8f531e1c69d55208061df1dc336093e00a846a5015beb08685d0df838f4ad"}
Sep 16 19:26:28.365869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.365590    2583 scope.go:117] "RemoveContainer" containerID="a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6"
Sep 16 19:26:28.374570 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.374554    2583 scope.go:117] "RemoveContainer" containerID="8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10"
Sep 16 19:26:28.382258 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.382221    2583 scope.go:117] "RemoveContainer" containerID="a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6"
Sep 16 19:26:28.382554 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:26:28.382533    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6\": container with ID starting with a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6 not found: ID does not exist" containerID="a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6"
Sep 16 19:26:28.382609 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.382564    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6"} err="failed to get container status \"a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6\": rpc error: code = NotFound desc = could not find container \"a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6\": container with ID starting with a811c2e48c86fa5c706192d7cebb3a58bd7e2d87076f9627ff326c96bef954b6 not found: ID does not exist"
Sep 16 19:26:28.382609 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.382581    2583 scope.go:117] "RemoveContainer" containerID="8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10"
Sep 16 19:26:28.382812 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:26:28.382795    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10\": container with ID starting with 8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10 not found: ID does not exist" containerID="8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10"
Sep 16 19:26:28.382855 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.382816    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10"} err="failed to get container status \"8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10\": rpc error: code = NotFound desc = could not find container \"8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10\": container with ID starting with 8c9e3087a8870ea6f3116b10ba5f981e3439cfc20106a4563edeb8b51615fa10 not found: ID does not exist"
Sep 16 19:26:28.415563 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.415540    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"]
Sep 16 19:26:28.426368 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:28.426347    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68b7f474bf-k79m7"]
Sep 16 19:26:29.411261 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:29.411224    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" path="/var/lib/kubelet/pods/08fac061-8c9f-414c-a062-6e61b4de6bfe/volumes"
Sep 16 19:26:34.071278 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.071245    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"]
Sep 16 19:26:34.071686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.071594    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde04ed5-e377-4d81-b035-ff509129fa40" containerName="main"
Sep 16 19:26:34.071686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.071608    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde04ed5-e377-4d81-b035-ff509129fa40" containerName="main"
Sep 16 19:26:34.071686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.071624    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="storage-initializer"
Sep 16 19:26:34.071686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.071629    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="storage-initializer"
Sep 16 19:26:34.071686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.071651    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main"
Sep 16 19:26:34.071686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.071657    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main"
Sep 16 19:26:34.071893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.071711    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08fac061-8c9f-414c-a062-6e61b4de6bfe" containerName="main"
Sep 16 19:26:34.071893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.071717    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde04ed5-e377-4d81-b035-ff509129fa40" containerName="main"
Sep 16 19:26:34.076380 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.076353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:26:34.080222 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.080196    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 19:26:34.080346 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.080196    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-tgtvg\""
Sep 16 19:26:34.080346 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.080285    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-qwhvm\""
Sep 16 19:26:34.080346 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.080285    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 19:26:34.080589 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.080575    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 19:26:34.093045 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.093026    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"]
Sep 16 19:26:34.169016 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.168993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e7c700-8902-40e2-b407-625affc54f52-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x\" (UID: \"c8e7c700-8902-40e2-b407-625affc54f52\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:26:34.169125 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.169034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbbv\" (UniqueName: \"kubernetes.io/projected/c8e7c700-8902-40e2-b407-625affc54f52-kube-api-access-pqbbv\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x\" (UID: \"c8e7c700-8902-40e2-b407-625affc54f52\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:26:34.270046 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.270020    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbbv\" (UniqueName: \"kubernetes.io/projected/c8e7c700-8902-40e2-b407-625affc54f52-kube-api-access-pqbbv\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x\" (UID: \"c8e7c700-8902-40e2-b407-625affc54f52\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:26:34.270145 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.270097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e7c700-8902-40e2-b407-625affc54f52-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x\" (UID: \"c8e7c700-8902-40e2-b407-625affc54f52\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:26:34.272740 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.272714    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e7c700-8902-40e2-b407-625affc54f52-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x\" (UID: \"c8e7c700-8902-40e2-b407-625affc54f52\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:26:34.280264 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.280243    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbbv\" (UniqueName: \"kubernetes.io/projected/c8e7c700-8902-40e2-b407-625affc54f52-kube-api-access-pqbbv\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x\" (UID: \"c8e7c700-8902-40e2-b407-625affc54f52\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:26:34.386320 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.386296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:26:34.670293 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:34.669971    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"]
Sep 16 19:26:35.400853 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:35.400812    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" event={"ID":"c8e7c700-8902-40e2-b407-625affc54f52","Type":"ContainerStarted","Data":"666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f"}
Sep 16 19:26:35.401227 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:35.400903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" event={"ID":"c8e7c700-8902-40e2-b407-625affc54f52","Type":"ContainerStarted","Data":"f9fa595fb272fbadf33c4fa1e87534af2d6144ab4744edc9fc12de03a48043ba"}
Sep 16 19:26:35.401227 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:35.400933    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:26:35.427365 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:26:35.427317    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" podStartSLOduration=1.427302994 podStartE2EDuration="1.427302994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:35.426719203 +0000 UTC m=+1272.552523596" watchObservedRunningTime="2026-09-16 19:26:35.427302994 +0000 UTC m=+1272.553107381"
Sep 16 19:27:06.408381 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:06.408347    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:27:20.503546 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.503512    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"]
Sep 16 19:27:20.506938 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.506921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.518012 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.517993    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 19:27:20.518392 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.518376    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 19:27:20.518510 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.518395    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 19:27:20.518510 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.518399    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 19:27:20.518711 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.518695    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-bzhlb\""
Sep 16 19:27:20.519332 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.519318    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:27:20.548131 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.548106    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"]
Sep 16 19:27:20.621782 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.621758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8mdt\" (UniqueName: \"kubernetes.io/projected/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kube-api-access-r8mdt\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.621894 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.621792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tls-certs\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.621894 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.621811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.621894 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.621857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-home\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.622002 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.621919    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.622002 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.621952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-dshm\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.622002 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.621967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-model-cache\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.622002 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.621987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tmp-dir\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.722826 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.722799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-home\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.722951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.722846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.722951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.722868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-dshm\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.722951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.722884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-model-cache\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.722951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.722902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tmp-dir\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.722951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.722930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8mdt\" (UniqueName: \"kubernetes.io/projected/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kube-api-access-r8mdt\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.722951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.722952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tls-certs\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.723307 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.722968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.723307 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.723252    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.723307 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.723279    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-home\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.723480 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.723336    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-model-cache\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.723480 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.723469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tmp-dir\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.723799 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.723781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.725281 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.725257    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-dshm\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.725678 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.725660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tls-certs\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.732628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.732608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8mdt\" (UniqueName: \"kubernetes.io/projected/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kube-api-access-r8mdt\") pod \"auth-invalid-token-test-kserve-5df8667c8f-vnnzj\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:20.817283 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:20.817196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:21.104923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:21.104894    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"]
Sep 16 19:27:21.106132 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:27:21.106108    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494f8ed2_c414_4cdb_9e8d_5aa767b8a4ae.slice/crio-4264759ee3a33abe2436fce58397eecfcf131b99f866dd0f201730e4d367f1cf WatchSource:0}: Error finding container 4264759ee3a33abe2436fce58397eecfcf131b99f866dd0f201730e4d367f1cf: Status 404 returned error can't find the container with id 4264759ee3a33abe2436fce58397eecfcf131b99f866dd0f201730e4d367f1cf
Sep 16 19:27:21.621692 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:21.621638    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" event={"ID":"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae","Type":"ContainerStarted","Data":"fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72"}
Sep 16 19:27:21.622209 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:21.621703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" event={"ID":"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae","Type":"ContainerStarted","Data":"4264759ee3a33abe2436fce58397eecfcf131b99f866dd0f201730e4d367f1cf"}
Sep 16 19:27:23.363937 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:23.363895    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"]
Sep 16 19:27:23.634518 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:23.634483    2583 generic.go:356] "Generic (PLEG): container finished" podID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerID="fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72" exitCode=0
Sep 16 19:27:23.634659 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:23.634561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" event={"ID":"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae","Type":"ContainerDied","Data":"fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72"}
Sep 16 19:27:24.641749 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:24.641712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" event={"ID":"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae","Type":"ContainerStarted","Data":"7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94"}
Sep 16 19:27:24.676579 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:24.676517    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podStartSLOduration=2.455182211 podStartE2EDuration="4.676494911s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.2213127s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:24.673818542 +0000 UTC m=+1321.799622930" watchObservedRunningTime="2026-09-16 19:27:24.676494911 +0000 UTC m=+1321.802299300"
Sep 16 19:27:24.818280 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:24.818243    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:26.407960 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:26.407922    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" podUID="c8e7c700-8902-40e2-b407-625affc54f52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:30.817670 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:30.817632    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:27:30.819395 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:30.819363    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:27:36.407472 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:36.407433    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" podUID="c8e7c700-8902-40e2-b407-625affc54f52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:38.383688 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:38.383645    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" podUID="c8e7c700-8902-40e2-b407-625affc54f52" containerName="main" containerID="cri-o://666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f" gracePeriod=45
Sep 16 19:27:40.817893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:40.817852    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:27:46.407532 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:46.407446    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" podUID="c8e7c700-8902-40e2-b407-625affc54f52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:46.407890 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:46.407554    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:27:50.818697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:50.818653    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:27:56.407551 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:27:56.407509    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" podUID="c8e7c700-8902-40e2-b407-625affc54f52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:28:00.818199 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:00.818163    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:28:06.407639 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:06.407592    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" podUID="c8e7c700-8902-40e2-b407-625affc54f52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:28:08.644804 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.644733    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:28:08.739518 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.739479    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e7c700-8902-40e2-b407-625affc54f52-tls-certs\") pod \"c8e7c700-8902-40e2-b407-625affc54f52\" (UID: \"c8e7c700-8902-40e2-b407-625affc54f52\") "
Sep 16 19:28:08.739692 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.739580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqbbv\" (UniqueName: \"kubernetes.io/projected/c8e7c700-8902-40e2-b407-625affc54f52-kube-api-access-pqbbv\") pod \"c8e7c700-8902-40e2-b407-625affc54f52\" (UID: \"c8e7c700-8902-40e2-b407-625affc54f52\") "
Sep 16 19:28:08.741852 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.741816    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e7c700-8902-40e2-b407-625affc54f52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8e7c700-8902-40e2-b407-625affc54f52" (UID: "c8e7c700-8902-40e2-b407-625affc54f52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:08.741961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.741931    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e7c700-8902-40e2-b407-625affc54f52-kube-api-access-pqbbv" (OuterVolumeSpecName: "kube-api-access-pqbbv") pod "c8e7c700-8902-40e2-b407-625affc54f52" (UID: "c8e7c700-8902-40e2-b407-625affc54f52"). InnerVolumeSpecName "kube-api-access-pqbbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:08.840759 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.840714    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e7c700-8902-40e2-b407-625affc54f52-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:28:08.840759 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.840755    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqbbv\" (UniqueName: \"kubernetes.io/projected/c8e7c700-8902-40e2-b407-625affc54f52-kube-api-access-pqbbv\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:28:08.849972 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.849945    2583 generic.go:356] "Generic (PLEG): container finished" podID="c8e7c700-8902-40e2-b407-625affc54f52" containerID="666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f" exitCode=0
Sep 16 19:28:08.850091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.850001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" event={"ID":"c8e7c700-8902-40e2-b407-625affc54f52","Type":"ContainerDied","Data":"666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f"}
Sep 16 19:28:08.850091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.850012    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"
Sep 16 19:28:08.850091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.850043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x" event={"ID":"c8e7c700-8902-40e2-b407-625affc54f52","Type":"ContainerDied","Data":"f9fa595fb272fbadf33c4fa1e87534af2d6144ab4744edc9fc12de03a48043ba"}
Sep 16 19:28:08.850091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.850063    2583 scope.go:117] "RemoveContainer" containerID="666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f"
Sep 16 19:28:08.861823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.861803    2583 scope.go:117] "RemoveContainer" containerID="666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f"
Sep 16 19:28:08.862144 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:28:08.862123    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f\": container with ID starting with 666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f not found: ID does not exist" containerID="666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f"
Sep 16 19:28:08.862240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.862158    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f"} err="failed to get container status \"666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f\": rpc error: code = NotFound desc = could not find container \"666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f\": container with ID starting with 666f973ba46ac903d1c9c27ebb025ef69c4931821a7689f8f11360c9f11d1c7f not found: ID does not exist"
Sep 16 19:28:08.891529 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.890119    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"]
Sep 16 19:28:08.895617 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:08.895555    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-55684xvm9x"]
Sep 16 19:28:09.416144 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:09.416102    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8e7c700-8902-40e2-b407-625affc54f52" path="/var/lib/kubelet/pods/c8e7c700-8902-40e2-b407-625affc54f52/volumes"
Sep 16 19:28:10.818231 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:10.818182    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:28:20.818378 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:20.818331    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:28:23.787328 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.787253    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"]
Sep 16 19:28:23.787682 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.787634    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e7c700-8902-40e2-b407-625affc54f52" containerName="main"
Sep 16 19:28:23.787682 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.787646    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e7c700-8902-40e2-b407-625affc54f52" containerName="main"
Sep 16 19:28:23.787757 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.787704    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e7c700-8902-40e2-b407-625affc54f52" containerName="main"
Sep 16 19:28:23.791583 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.791566    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:23.795722 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.795703    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:28:23.797206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.797048    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-llfn9\""
Sep 16 19:28:23.797206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.797078    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:28:23.797475 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.797455    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:28:23.798212 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.798195    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-chsqs\""
Sep 16 19:28:23.817545 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.817517    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"]
Sep 16 19:28:23.877715 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.877674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d99371-3143-4d9a-b180-1e5edb74a293-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf\" (UID: \"a7d99371-3143-4d9a-b180-1e5edb74a293\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:23.877912 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.877767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrnm\" (UniqueName: \"kubernetes.io/projected/a7d99371-3143-4d9a-b180-1e5edb74a293-kube-api-access-7lrnm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf\" (UID: \"a7d99371-3143-4d9a-b180-1e5edb74a293\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:23.978771 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.978722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrnm\" (UniqueName: \"kubernetes.io/projected/a7d99371-3143-4d9a-b180-1e5edb74a293-kube-api-access-7lrnm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf\" (UID: \"a7d99371-3143-4d9a-b180-1e5edb74a293\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:23.978946 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.978793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d99371-3143-4d9a-b180-1e5edb74a293-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf\" (UID: \"a7d99371-3143-4d9a-b180-1e5edb74a293\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:23.981494 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.981465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d99371-3143-4d9a-b180-1e5edb74a293-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf\" (UID: \"a7d99371-3143-4d9a-b180-1e5edb74a293\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:23.989864 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:23.989826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrnm\" (UniqueName: \"kubernetes.io/projected/a7d99371-3143-4d9a-b180-1e5edb74a293-kube-api-access-7lrnm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf\" (UID: \"a7d99371-3143-4d9a-b180-1e5edb74a293\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:24.102445 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:24.102329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:24.406326 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:24.406294    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"]
Sep 16 19:28:24.409683 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:28:24.409637    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d99371_3143_4d9a_b180_1e5edb74a293.slice/crio-2a54845407aee7c7481f9d476f2e1866d1042efee4264f0c7e46aed0c712cfc1 WatchSource:0}: Error finding container 2a54845407aee7c7481f9d476f2e1866d1042efee4264f0c7e46aed0c712cfc1: Status 404 returned error can't find the container with id 2a54845407aee7c7481f9d476f2e1866d1042efee4264f0c7e46aed0c712cfc1
Sep 16 19:28:24.931655 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:24.931616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf" event={"ID":"a7d99371-3143-4d9a-b180-1e5edb74a293","Type":"ContainerStarted","Data":"b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff"}
Sep 16 19:28:24.932113 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:24.931685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf" event={"ID":"a7d99371-3143-4d9a-b180-1e5edb74a293","Type":"ContainerStarted","Data":"2a54845407aee7c7481f9d476f2e1866d1042efee4264f0c7e46aed0c712cfc1"}
Sep 16 19:28:24.932113 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:24.931754    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:24.953851 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:24.953807    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf" podStartSLOduration=1.953791947 podStartE2EDuration="1.953791947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:24.952843798 +0000 UTC m=+1382.078648187" watchObservedRunningTime="2026-09-16 19:28:24.953791947 +0000 UTC m=+1382.079596335"
Sep 16 19:28:30.818558 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:30.818511    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:28:40.818558 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:40.818516    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:28:50.818156 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:50.818118    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:28:55.939627 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:55.939596    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:28:57.706579 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:28:57.706543    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"]
Sep 16 19:29:00.818175 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:00.818134    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:29:05.939480 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:05.939444    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf" podUID="a7d99371-3143-4d9a-b180-1e5edb74a293" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:08.814945 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:08.814905    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 19:29:08.815310 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:29:08.814949    2583 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf" podUID="a7d99371-3143-4d9a-b180-1e5edb74a293" containerName="main" containerID="cri-o://b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff"
Sep 16 19:29:08.815310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:08.815013    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf" podUID="a7d99371-3143-4d9a-b180-1e5edb74a293" containerName="main" containerID="cri-o://b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff" gracePeriod=49
Sep 16 19:29:08.965059 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:08.965038    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf_a7d99371-3143-4d9a-b180-1e5edb74a293/main/0.log"
Sep 16 19:29:08.968086 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:08.968071    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:29:09.069020 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.068925    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d99371-3143-4d9a-b180-1e5edb74a293-tls-certs\") pod \"a7d99371-3143-4d9a-b180-1e5edb74a293\" (UID: \"a7d99371-3143-4d9a-b180-1e5edb74a293\") "
Sep 16 19:29:09.069020 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.069004    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lrnm\" (UniqueName: \"kubernetes.io/projected/a7d99371-3143-4d9a-b180-1e5edb74a293-kube-api-access-7lrnm\") pod \"a7d99371-3143-4d9a-b180-1e5edb74a293\" (UID: \"a7d99371-3143-4d9a-b180-1e5edb74a293\") "
Sep 16 19:29:09.071369 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.071335    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d99371-3143-4d9a-b180-1e5edb74a293-kube-api-access-7lrnm" (OuterVolumeSpecName: "kube-api-access-7lrnm") pod "a7d99371-3143-4d9a-b180-1e5edb74a293" (UID: "a7d99371-3143-4d9a-b180-1e5edb74a293"). InnerVolumeSpecName "kube-api-access-7lrnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:09.071369 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.071357    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d99371-3143-4d9a-b180-1e5edb74a293-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7d99371-3143-4d9a-b180-1e5edb74a293" (UID: "a7d99371-3143-4d9a-b180-1e5edb74a293"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:09.146395 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.146367    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf_a7d99371-3143-4d9a-b180-1e5edb74a293/main/0.log"
Sep 16 19:29:09.146563 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.146441    2583 generic.go:356] "Generic (PLEG): container finished" podID="a7d99371-3143-4d9a-b180-1e5edb74a293" containerID="b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff" exitCode=1
Sep 16 19:29:09.146563 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.146482    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"
Sep 16 19:29:09.146563 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.146521    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf" event={"ID":"a7d99371-3143-4d9a-b180-1e5edb74a293","Type":"ContainerDied","Data":"b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff"}
Sep 16 19:29:09.146668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.146582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf" event={"ID":"a7d99371-3143-4d9a-b180-1e5edb74a293","Type":"ContainerDied","Data":"2a54845407aee7c7481f9d476f2e1866d1042efee4264f0c7e46aed0c712cfc1"}
Sep 16 19:29:09.146668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.146607    2583 scope.go:117] "RemoveContainer" containerID="b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff"
Sep 16 19:29:09.161364 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.161345    2583 scope.go:117] "RemoveContainer" containerID="b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff"
Sep 16 19:29:09.161681 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:29:09.161654    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff\": container with ID starting with b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff not found: ID does not exist" containerID="b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff"
Sep 16 19:29:09.161767 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.161690    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff"} err="failed to get container status \"b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff\": rpc error: code = NotFound desc = could not find container \"b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff\": container with ID starting with b54ff84bc6bc41b04c0e27581629805cf47a176353e875fda084cdc5a6261bff not found: ID does not exist"
Sep 16 19:29:09.169917 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.169898    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d99371-3143-4d9a-b180-1e5edb74a293-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:09.169993 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.169920    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrnm\" (UniqueName: \"kubernetes.io/projected/a7d99371-3143-4d9a-b180-1e5edb74a293-kube-api-access-7lrnm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:09.176683 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.176661    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"]
Sep 16 19:29:09.180983 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.180962    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67ssppf"]
Sep 16 19:29:09.411539 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:09.411499    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d99371-3143-4d9a-b180-1e5edb74a293" path="/var/lib/kubelet/pods/a7d99371-3143-4d9a-b180-1e5edb74a293/volumes"
Sep 16 19:29:10.831807 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:10.831778    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:29:10.840042 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:10.840015    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:29:12.404982 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:12.404949    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"]
Sep 16 19:29:27.427873 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:27.427812    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" containerID="cri-o://7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94" gracePeriod=45
Sep 16 19:29:27.833002 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:27.832900    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:29:28.060285 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.060261    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:29:28.118859 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.118840    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tls-certs\") pod \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") "
Sep 16 19:29:28.118957 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.118871    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8mdt\" (UniqueName: \"kubernetes.io/projected/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kube-api-access-r8mdt\") pod \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") "
Sep 16 19:29:28.118957 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.118891    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kserve-provision-location\") pod \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") "
Sep 16 19:29:28.118957 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.118909    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-dshm\") pod \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") "
Sep 16 19:29:28.119085 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.119039    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-cabundle-cert\") pod \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") "
Sep 16 19:29:28.119145 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.119083    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-home\") pod \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") "
Sep 16 19:29:28.119145 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.119127    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-model-cache\") pod \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") "
Sep 16 19:29:28.119250 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.119178    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tmp-dir\") pod \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\" (UID: \"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae\") "
Sep 16 19:29:28.119959 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.119857    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" (UID: "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:28.119959 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.119933    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" (UID: "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:28.120389 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.120284    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-home" (OuterVolumeSpecName: "home") pod "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" (UID: "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:28.121806 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.121770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" (UID: "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:28.122121 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.122087    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-dshm" (OuterVolumeSpecName: "dshm") pod "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" (UID: "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:28.122272 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.122135    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kube-api-access-r8mdt" (OuterVolumeSpecName: "kube-api-access-r8mdt") pod "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" (UID: "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae"). InnerVolumeSpecName "kube-api-access-r8mdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:28.133836 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.133810    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" (UID: "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:28.144535 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.144502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" (UID: "494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:28.220715 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.220687    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:28.220715 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.220711    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:28.220831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.220720    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:28.220831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.220728    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:28.220831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.220736    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:28.220831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.220744    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8mdt\" (UniqueName: \"kubernetes.io/projected/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kube-api-access-r8mdt\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:28.220831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.220752    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:28.220831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.220760    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:29:28.242144 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.242122    2583 generic.go:356] "Generic (PLEG): container finished" podID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerID="7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94" exitCode=0
Sep 16 19:29:28.242248 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.242174    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"
Sep 16 19:29:28.242248 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.242222    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" event={"ID":"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae","Type":"ContainerDied","Data":"7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94"}
Sep 16 19:29:28.242328 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.242279    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj" event={"ID":"494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae","Type":"ContainerDied","Data":"4264759ee3a33abe2436fce58397eecfcf131b99f866dd0f201730e4d367f1cf"}
Sep 16 19:29:28.242328 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.242302    2583 scope.go:117] "RemoveContainer" containerID="7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94"
Sep 16 19:29:28.251133 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.251112    2583 scope.go:117] "RemoveContainer" containerID="fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72"
Sep 16 19:29:28.259049 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.259033    2583 scope.go:117] "RemoveContainer" containerID="7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94"
Sep 16 19:29:28.259337 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:29:28.259317    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94\": container with ID starting with 7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94 not found: ID does not exist" containerID="7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94"
Sep 16 19:29:28.259398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.259344    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94"} err="failed to get container status \"7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94\": rpc error: code = NotFound desc = could not find container \"7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94\": container with ID starting with 7a7f21e841ffc1895ed500e4475da395a163b4a1c6a769b5ecedb014eec28e94 not found: ID does not exist"
Sep 16 19:29:28.259398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.259360    2583 scope.go:117] "RemoveContainer" containerID="fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72"
Sep 16 19:29:28.259681 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:29:28.259661    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72\": container with ID starting with fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72 not found: ID does not exist" containerID="fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72"
Sep 16 19:29:28.259723 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.259687    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72"} err="failed to get container status \"fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72\": rpc error: code = NotFound desc = could not find container \"fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72\": container with ID starting with fcf7e19e693544ad3602c15be7d8d2371715966ebf7eef456438c6da0114eb72 not found: ID does not exist"
Sep 16 19:29:28.271813 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.271791    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"]
Sep 16 19:29:28.274748 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:28.274730    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5df8667c8f-vnnzj"]
Sep 16 19:29:29.412092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:29.412049    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" path="/var/lib/kubelet/pods/494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae/volumes"
Sep 16 19:29:50.721469 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.721430    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"]
Sep 16 19:29:50.722051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.721941    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d99371-3143-4d9a-b180-1e5edb74a293" containerName="main"
Sep 16 19:29:50.722051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.721960    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d99371-3143-4d9a-b180-1e5edb74a293" containerName="main"
Sep 16 19:29:50.722051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.721981    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main"
Sep 16 19:29:50.722051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.721989    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main"
Sep 16 19:29:50.722051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.722008    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="storage-initializer"
Sep 16 19:29:50.722051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.722017    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="storage-initializer"
Sep 16 19:29:50.722369 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.722104    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="494f8ed2-c414-4cdb-9e8d-5aa767b8a4ae" containerName="main"
Sep 16 19:29:50.722369 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.722120    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d99371-3143-4d9a-b180-1e5edb74a293" containerName="main"
Sep 16 19:29:50.726876 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.726854    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.730726 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.730704    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-z72jw\""
Sep 16 19:29:50.733515 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.733316    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 19:29:50.733594 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.733578    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-5cwx5\""
Sep 16 19:29:50.734187 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.734167    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 19:29:50.734387 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.734372    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 19:29:50.778996 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.778971    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"]
Sep 16 19:29:50.783511 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.783492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.783597 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.783563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhqjr\" (UniqueName: \"kubernetes.io/projected/26343122-8576-4766-86bb-dde948cee8ee-kube-api-access-mhqjr\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.783597 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.783584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26343122-8576-4766-86bb-dde948cee8ee-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.783689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.783612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.884886 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.884856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.884886 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.884887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.885040 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.884941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhqjr\" (UniqueName: \"kubernetes.io/projected/26343122-8576-4766-86bb-dde948cee8ee-kube-api-access-mhqjr\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.885040 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.884962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26343122-8576-4766-86bb-dde948cee8ee-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.885173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.885154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.885224 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.885208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.887663 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.887636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26343122-8576-4766-86bb-dde948cee8ee-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:50.903181 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:50.903161    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhqjr\" (UniqueName: \"kubernetes.io/projected/26343122-8576-4766-86bb-dde948cee8ee-kube-api-access-mhqjr\") pod \"scheduler-custom-template-test-kserve-router-scheduler-854r5pp4\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:51.037064 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:51.037009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:51.350981 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:51.350950    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"]
Sep 16 19:29:51.353539 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:29:51.353508    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26343122_8576_4766_86bb_dde948cee8ee.slice/crio-efc6d539dd5ac1726b95d3104251807e2b30959b095565e61b3b5ceaeb694b53 WatchSource:0}: Error finding container efc6d539dd5ac1726b95d3104251807e2b30959b095565e61b3b5ceaeb694b53: Status 404 returned error can't find the container with id efc6d539dd5ac1726b95d3104251807e2b30959b095565e61b3b5ceaeb694b53
Sep 16 19:29:52.358203 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:52.358161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" event={"ID":"26343122-8576-4766-86bb-dde948cee8ee","Type":"ContainerStarted","Data":"e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed"}
Sep 16 19:29:52.358619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:52.358235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" event={"ID":"26343122-8576-4766-86bb-dde948cee8ee","Type":"ContainerStarted","Data":"efc6d539dd5ac1726b95d3104251807e2b30959b095565e61b3b5ceaeb694b53"}
Sep 16 19:29:52.358619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:52.358299    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:29:52.420462 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:29:52.420389    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" podStartSLOduration=2.420369456 podStartE2EDuration="2.420369456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:52.41857763 +0000 UTC m=+1469.544382018" watchObservedRunningTime="2026-09-16 19:29:52.420369456 +0000 UTC m=+1469.546173846"
Sep 16 19:30:23.365766 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:23.365734    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:30:24.894975 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:24.894942    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"]
Sep 16 19:30:24.899439 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:24.899400    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:24.904880 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:24.904858    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:30:24.906261 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:24.906238    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:30:24.906588 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:24.906556    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:30:24.906677 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:24.906594    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:30:24.906677 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:24.906597    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-zghdq\""
Sep 16 19:30:24.911523 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:24.911504    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 19:30:24.923745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:24.923724    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"]
Sep 16 19:30:25.057038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.057004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-home\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.057038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.057034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c1268c-a9da-47e5-af00-dc7e85866480-tls-certs\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.057229 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.057108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-model-cache\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.057229 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.057161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-dshm\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.057229 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.057221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c1268c-a9da-47e5-af00-dc7e85866480-cabundle-cert\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.057340 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.057239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9rc\" (UniqueName: \"kubernetes.io/projected/f4c1268c-a9da-47e5-af00-dc7e85866480-kube-api-access-nv9rc\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.057340 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.057264    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-tmp-dir\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.057340 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.057322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-kserve-provision-location\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158099 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-home\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158099 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c1268c-a9da-47e5-af00-dc7e85866480-tls-certs\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-model-cache\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-dshm\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c1268c-a9da-47e5-af00-dc7e85866480-cabundle-cert\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158254    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9rc\" (UniqueName: \"kubernetes.io/projected/f4c1268c-a9da-47e5-af00-dc7e85866480-kube-api-access-nv9rc\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-tmp-dir\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158624 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-kserve-provision-location\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158624 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-home\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158624 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158571    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-model-cache\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-kserve-provision-location\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.158917 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.158879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-tmp-dir\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.159447 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.159396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c1268c-a9da-47e5-af00-dc7e85866480-cabundle-cert\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.160661 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.160633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-dshm\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.161032 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.161013    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c1268c-a9da-47e5-af00-dc7e85866480-tls-certs\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.173747 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.173728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9rc\" (UniqueName: \"kubernetes.io/projected/f4c1268c-a9da-47e5-af00-dc7e85866480-kube-api-access-nv9rc\") pod \"auth-disabled-test-kserve-56bb6c6494-k5n8l\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.209592 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.209570    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:25.497776 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.497749    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"]
Sep 16 19:30:25.499478 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:30:25.499450    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c1268c_a9da_47e5_af00_dc7e85866480.slice/crio-59a2cdc2cb80a8aec3fde242232120c68bb4d4bb76efa87fa20ae611011151ef WatchSource:0}: Error finding container 59a2cdc2cb80a8aec3fde242232120c68bb4d4bb76efa87fa20ae611011151ef: Status 404 returned error can't find the container with id 59a2cdc2cb80a8aec3fde242232120c68bb4d4bb76efa87fa20ae611011151ef
Sep 16 19:30:25.510165 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:25.510141    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" event={"ID":"f4c1268c-a9da-47e5-af00-dc7e85866480","Type":"ContainerStarted","Data":"59a2cdc2cb80a8aec3fde242232120c68bb4d4bb76efa87fa20ae611011151ef"}
Sep 16 19:30:26.517610 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:26.517565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" event={"ID":"f4c1268c-a9da-47e5-af00-dc7e85866480","Type":"ContainerStarted","Data":"9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9"}
Sep 16 19:30:26.785683 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:26.785603    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:30:26.789070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:26.789044    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:30:26.790795 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:26.790778    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:30:26.793920 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:26.793902    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:30:28.532731 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:28.532696    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerID="9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9" exitCode=0
Sep 16 19:30:28.533100 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:28.532769    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" event={"ID":"f4c1268c-a9da-47e5-af00-dc7e85866480","Type":"ContainerDied","Data":"9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9"}
Sep 16 19:30:29.543934 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:29.543873    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" event={"ID":"f4c1268c-a9da-47e5-af00-dc7e85866480","Type":"ContainerStarted","Data":"7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df"}
Sep 16 19:30:29.599681 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:29.599617    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podStartSLOduration=3.325543361 podStartE2EDuration="5.599596979s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.274053618s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:29.597809161 +0000 UTC m=+1506.723613549" watchObservedRunningTime="2026-09-16 19:30:29.599596979 +0000 UTC m=+1506.725401368"
Sep 16 19:30:30.210586 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:30.210549    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:35.210558 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:35.210522    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:30:35.211853 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:35.211815    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:30:38.057335 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:38.057298    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"]
Sep 16 19:30:43.366270 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:43.366232    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" podUID="26343122-8576-4766-86bb-dde948cee8ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:45.210731 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:45.210636    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:30:53.079439 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:53.079363    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" podUID="26343122-8576-4766-86bb-dde948cee8ee" containerName="main" containerID="cri-o://e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed" gracePeriod=45
Sep 16 19:30:53.366138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:53.366061    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" podUID="26343122-8576-4766-86bb-dde948cee8ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:55.210974 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:30:55.210929    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:31:03.365636 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:03.365596    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" podUID="26343122-8576-4766-86bb-dde948cee8ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:03.365993 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:03.365696    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:31:05.210647 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:05.210604    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:31:13.365891 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:13.365853    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" podUID="26343122-8576-4766-86bb-dde948cee8ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:15.210902 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:15.210848    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:31:23.324286 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.324261    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:31:23.377405 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.377377    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-cachi2-cache\") pod \"26343122-8576-4766-86bb-dde948cee8ee\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") "
Sep 16 19:31:23.377601 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.377490    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-tokenizer-cache\") pod \"26343122-8576-4766-86bb-dde948cee8ee\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") "
Sep 16 19:31:23.377601 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.377543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26343122-8576-4766-86bb-dde948cee8ee-tls-certs\") pod \"26343122-8576-4766-86bb-dde948cee8ee\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") "
Sep 16 19:31:23.377601 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.377567    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhqjr\" (UniqueName: \"kubernetes.io/projected/26343122-8576-4766-86bb-dde948cee8ee-kube-api-access-mhqjr\") pod \"26343122-8576-4766-86bb-dde948cee8ee\" (UID: \"26343122-8576-4766-86bb-dde948cee8ee\") "
Sep 16 19:31:23.377771 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.377657    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "26343122-8576-4766-86bb-dde948cee8ee" (UID: "26343122-8576-4766-86bb-dde948cee8ee"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:23.377826 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.377770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "26343122-8576-4766-86bb-dde948cee8ee" (UID: "26343122-8576-4766-86bb-dde948cee8ee"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:23.377870 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.377831    2583 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-cachi2-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.379870 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.379836    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26343122-8576-4766-86bb-dde948cee8ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26343122-8576-4766-86bb-dde948cee8ee" (UID: "26343122-8576-4766-86bb-dde948cee8ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:23.379988 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.379905    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26343122-8576-4766-86bb-dde948cee8ee-kube-api-access-mhqjr" (OuterVolumeSpecName: "kube-api-access-mhqjr") pod "26343122-8576-4766-86bb-dde948cee8ee" (UID: "26343122-8576-4766-86bb-dde948cee8ee"). InnerVolumeSpecName "kube-api-access-mhqjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:23.478954 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.478928    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/26343122-8576-4766-86bb-dde948cee8ee-tokenizer-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.478954 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.478954    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26343122-8576-4766-86bb-dde948cee8ee-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.479116 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.478967    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhqjr\" (UniqueName: \"kubernetes.io/projected/26343122-8576-4766-86bb-dde948cee8ee-kube-api-access-mhqjr\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.803055 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.802971    2583 generic.go:356] "Generic (PLEG): container finished" podID="26343122-8576-4766-86bb-dde948cee8ee" containerID="e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed" exitCode=0
Sep 16 19:31:23.803055 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.803012    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"
Sep 16 19:31:23.803264 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.803012    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" event={"ID":"26343122-8576-4766-86bb-dde948cee8ee","Type":"ContainerDied","Data":"e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed"}
Sep 16 19:31:23.803264 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.803147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4" event={"ID":"26343122-8576-4766-86bb-dde948cee8ee","Type":"ContainerDied","Data":"efc6d539dd5ac1726b95d3104251807e2b30959b095565e61b3b5ceaeb694b53"}
Sep 16 19:31:23.803264 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.803181    2583 scope.go:117] "RemoveContainer" containerID="e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed"
Sep 16 19:31:23.811905 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.811888    2583 scope.go:117] "RemoveContainer" containerID="e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed"
Sep 16 19:31:23.812197 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:31:23.812173    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed\": container with ID starting with e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed not found: ID does not exist" containerID="e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed"
Sep 16 19:31:23.812284 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.812207    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed"} err="failed to get container status \"e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed\": rpc error: code = NotFound desc = could not find container \"e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed\": container with ID starting with e8c46a505b564bbf626c78e9b5095617140a35139c52441f653c86cfe7fd52ed not found: ID does not exist"
Sep 16 19:31:23.837926 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.837900    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"]
Sep 16 19:31:23.843239 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:23.843215    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-854r5pp4"]
Sep 16 19:31:25.210256 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:25.210212    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:31:25.413012 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:25.412981    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26343122-8576-4766-86bb-dde948cee8ee" path="/var/lib/kubelet/pods/26343122-8576-4766-86bb-dde948cee8ee/volumes"
Sep 16 19:31:35.210489 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:35.210449    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:31:36.466080 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.466050    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"]
Sep 16 19:31:36.466466 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.466397    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26343122-8576-4766-86bb-dde948cee8ee" containerName="main"
Sep 16 19:31:36.466466 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.466421    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26343122-8576-4766-86bb-dde948cee8ee" containerName="main"
Sep 16 19:31:36.466542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.466505    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26343122-8576-4766-86bb-dde948cee8ee" containerName="main"
Sep 16 19:31:36.470712 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.470695    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.477502 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.477482    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:31:36.478627 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.478605    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:31:36.478722 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.478635    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-mr2jf\""
Sep 16 19:31:36.478722 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.478646    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:31:36.505457 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.505435    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"]
Sep 16 19:31:36.588001 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.587978    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.588144 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.588014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.588144 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.588097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.588245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.588151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.588245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.588188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c3493b-f83b-41ba-87e4-58345f26fc0b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.588245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.588240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbdlw\" (UniqueName: \"kubernetes.io/projected/04c3493b-f83b-41ba-87e4-58345f26fc0b-kube-api-access-bbdlw\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.689316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.689275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbdlw\" (UniqueName: \"kubernetes.io/projected/04c3493b-f83b-41ba-87e4-58345f26fc0b-kube-api-access-bbdlw\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.689316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.689313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.689564 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.689340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.689564 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.689395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.689564 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.689434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.689564 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.689461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c3493b-f83b-41ba-87e4-58345f26fc0b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.689782 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.689727    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.689846 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.689795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.689910 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.689851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.691958 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.691931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.692085 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.692069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c3493b-f83b-41ba-87e4-58345f26fc0b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.701253 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.701227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbdlw\" (UniqueName: \"kubernetes.io/projected/04c3493b-f83b-41ba-87e4-58345f26fc0b-kube-api-access-bbdlw\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.781048 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.780992    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:36.830529 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.830459    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"]
Sep 16 19:31:36.835139 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.835100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:31:36.840683 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.840660    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-xd97k\""
Sep 16 19:31:36.857438 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.856976    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"]
Sep 16 19:31:36.891425 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.891384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f13e5c65-a04c-489a-9a45-d3165cce2e95-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk\" (UID: \"f13e5c65-a04c-489a-9a45-d3165cce2e95\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:31:36.891538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.891475    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmn7\" (UniqueName: \"kubernetes.io/projected/f13e5c65-a04c-489a-9a45-d3165cce2e95-kube-api-access-dtmn7\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk\" (UID: \"f13e5c65-a04c-489a-9a45-d3165cce2e95\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:31:36.992729 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.992700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f13e5c65-a04c-489a-9a45-d3165cce2e95-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk\" (UID: \"f13e5c65-a04c-489a-9a45-d3165cce2e95\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:31:36.992851 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.992761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmn7\" (UniqueName: \"kubernetes.io/projected/f13e5c65-a04c-489a-9a45-d3165cce2e95-kube-api-access-dtmn7\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk\" (UID: \"f13e5c65-a04c-489a-9a45-d3165cce2e95\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:31:36.995302 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:36.995283    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f13e5c65-a04c-489a-9a45-d3165cce2e95-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk\" (UID: \"f13e5c65-a04c-489a-9a45-d3165cce2e95\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:31:37.022348 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.022329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmn7\" (UniqueName: \"kubernetes.io/projected/f13e5c65-a04c-489a-9a45-d3165cce2e95-kube-api-access-dtmn7\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk\" (UID: \"f13e5c65-a04c-489a-9a45-d3165cce2e95\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:31:37.131598 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.131562    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"]
Sep 16 19:31:37.132842 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:31:37.132816    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c3493b_f83b_41ba_87e4_58345f26fc0b.slice/crio-d01f755fe22686eaf9db3388524f455f318bd3206395262632e22e206d5bcf7f WatchSource:0}: Error finding container d01f755fe22686eaf9db3388524f455f318bd3206395262632e22e206d5bcf7f: Status 404 returned error can't find the container with id d01f755fe22686eaf9db3388524f455f318bd3206395262632e22e206d5bcf7f
Sep 16 19:31:37.134630 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.134616    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:31:37.154402 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.154381    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:31:37.458618 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.458556    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"]
Sep 16 19:31:37.461001 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:31:37.460978    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13e5c65_a04c_489a_9a45_d3165cce2e95.slice/crio-0560d5992ae47cb9b3ba2874d7fcb86cf2d56f9576b6b7af5f5345dcd0b25252 WatchSource:0}: Error finding container 0560d5992ae47cb9b3ba2874d7fcb86cf2d56f9576b6b7af5f5345dcd0b25252: Status 404 returned error can't find the container with id 0560d5992ae47cb9b3ba2874d7fcb86cf2d56f9576b6b7af5f5345dcd0b25252
Sep 16 19:31:37.887438 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.887317    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" event={"ID":"f13e5c65-a04c-489a-9a45-d3165cce2e95","Type":"ContainerStarted","Data":"417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630"}
Sep 16 19:31:37.887438 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.887386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" event={"ID":"f13e5c65-a04c-489a-9a45-d3165cce2e95","Type":"ContainerStarted","Data":"0560d5992ae47cb9b3ba2874d7fcb86cf2d56f9576b6b7af5f5345dcd0b25252"}
Sep 16 19:31:37.888764 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.888707    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:31:37.890341 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.890273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k" event={"ID":"04c3493b-f83b-41ba-87e4-58345f26fc0b","Type":"ContainerStarted","Data":"d01f755fe22686eaf9db3388524f455f318bd3206395262632e22e206d5bcf7f"}
Sep 16 19:31:37.924204 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:37.923726    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" podStartSLOduration=1.923711709 podStartE2EDuration="1.923711709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:37.923331886 +0000 UTC m=+1575.049136275" watchObservedRunningTime="2026-09-16 19:31:37.923711709 +0000 UTC m=+1575.049516096"
Sep 16 19:31:38.898191 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:38.898145    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k" event={"ID":"04c3493b-f83b-41ba-87e4-58345f26fc0b","Type":"ContainerStarted","Data":"83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3"}
Sep 16 19:31:38.946663 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:38.946611    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k" podStartSLOduration=1.2602567 podStartE2EDuration="2.946594613s" totalImagesPullingTime="1.686337913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:38.939018508 +0000 UTC m=+1576.064822896" watchObservedRunningTime="2026-09-16 19:31:38.946594613 +0000 UTC m=+1576.072399000"
Sep 16 19:31:39.781453 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:39.781423    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:45.210667 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:45.210624    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:31:46.781697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:46.781664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:46.794517 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:46.794489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:46.946545 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:46.946512    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:31:55.210841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:31:55.210780    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:32:05.210392 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:05.210350    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:32:09.905274 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:09.905240    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:32:15.210769 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:15.210674    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 19:32:25.220346 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:25.220305    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:32:25.227961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:25.227935    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:32:30.170967 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:30.170925    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"]
Sep 16 19:32:45.194255 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:45.194197    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main" containerID="cri-o://7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df" gracePeriod=45
Sep 16 19:32:45.943635 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:45.943605    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:32:46.030006 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.029912    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-kserve-provision-location\") pod \"f4c1268c-a9da-47e5-af00-dc7e85866480\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") "
Sep 16 19:32:46.030006 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.029954    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-home\") pod \"f4c1268c-a9da-47e5-af00-dc7e85866480\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") "
Sep 16 19:32:46.030006 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.029985    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-model-cache\") pod \"f4c1268c-a9da-47e5-af00-dc7e85866480\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") "
Sep 16 19:32:46.030305 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.030013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-dshm\") pod \"f4c1268c-a9da-47e5-af00-dc7e85866480\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") "
Sep 16 19:32:46.030305 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.030036    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c1268c-a9da-47e5-af00-dc7e85866480-cabundle-cert\") pod \"f4c1268c-a9da-47e5-af00-dc7e85866480\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") "
Sep 16 19:32:46.030305 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.030141    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c1268c-a9da-47e5-af00-dc7e85866480-tls-certs\") pod \"f4c1268c-a9da-47e5-af00-dc7e85866480\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") "
Sep 16 19:32:46.030305 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.030165    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-tmp-dir\") pod \"f4c1268c-a9da-47e5-af00-dc7e85866480\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") "
Sep 16 19:32:46.030536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.030302    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4c1268c-a9da-47e5-af00-dc7e85866480" (UID: "f4c1268c-a9da-47e5-af00-dc7e85866480"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:46.030536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.030496    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c1268c-a9da-47e5-af00-dc7e85866480-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4c1268c-a9da-47e5-af00-dc7e85866480" (UID: "f4c1268c-a9da-47e5-af00-dc7e85866480"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:46.030831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.030792    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-home" (OuterVolumeSpecName: "home") pod "f4c1268c-a9da-47e5-af00-dc7e85866480" (UID: "f4c1268c-a9da-47e5-af00-dc7e85866480"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:46.031005 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.030981    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv9rc\" (UniqueName: \"kubernetes.io/projected/f4c1268c-a9da-47e5-af00-dc7e85866480-kube-api-access-nv9rc\") pod \"f4c1268c-a9da-47e5-af00-dc7e85866480\" (UID: \"f4c1268c-a9da-47e5-af00-dc7e85866480\") "
Sep 16 19:32:46.031329 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.031311    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:32:46.031449 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.031332    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:32:46.031449 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.031348    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c1268c-a9da-47e5-af00-dc7e85866480-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:32:46.032652 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.032625    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-dshm" (OuterVolumeSpecName: "dshm") pod "f4c1268c-a9da-47e5-af00-dc7e85866480" (UID: "f4c1268c-a9da-47e5-af00-dc7e85866480"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:46.032752 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.032662    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c1268c-a9da-47e5-af00-dc7e85866480-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4c1268c-a9da-47e5-af00-dc7e85866480" (UID: "f4c1268c-a9da-47e5-af00-dc7e85866480"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:46.032811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.032770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c1268c-a9da-47e5-af00-dc7e85866480-kube-api-access-nv9rc" (OuterVolumeSpecName: "kube-api-access-nv9rc") pod "f4c1268c-a9da-47e5-af00-dc7e85866480" (UID: "f4c1268c-a9da-47e5-af00-dc7e85866480"). InnerVolumeSpecName "kube-api-access-nv9rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:46.050592 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.050558    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4c1268c-a9da-47e5-af00-dc7e85866480" (UID: "f4c1268c-a9da-47e5-af00-dc7e85866480"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:46.055589 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.055566    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4c1268c-a9da-47e5-af00-dc7e85866480" (UID: "f4c1268c-a9da-47e5-af00-dc7e85866480"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:46.132327 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.132292    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv9rc\" (UniqueName: \"kubernetes.io/projected/f4c1268c-a9da-47e5-af00-dc7e85866480-kube-api-access-nv9rc\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:32:46.132327 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.132321    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:32:46.132327 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.132331    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:32:46.132558 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.132342    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c1268c-a9da-47e5-af00-dc7e85866480-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:32:46.132558 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.132351    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c1268c-a9da-47e5-af00-dc7e85866480-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:32:46.230164 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.230127    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerID="7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df" exitCode=0
Sep 16 19:32:46.230594 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.230180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" event={"ID":"f4c1268c-a9da-47e5-af00-dc7e85866480","Type":"ContainerDied","Data":"7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df"}
Sep 16 19:32:46.230594 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.230181    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"
Sep 16 19:32:46.230594 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.230231    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l" event={"ID":"f4c1268c-a9da-47e5-af00-dc7e85866480","Type":"ContainerDied","Data":"59a2cdc2cb80a8aec3fde242232120c68bb4d4bb76efa87fa20ae611011151ef"}
Sep 16 19:32:46.230594 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.230254    2583 scope.go:117] "RemoveContainer" containerID="7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df"
Sep 16 19:32:46.239343 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.239309    2583 scope.go:117] "RemoveContainer" containerID="9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9"
Sep 16 19:32:46.248051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.248031    2583 scope.go:117] "RemoveContainer" containerID="7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df"
Sep 16 19:32:46.248322 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:32:46.248304    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df\": container with ID starting with 7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df not found: ID does not exist" containerID="7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df"
Sep 16 19:32:46.248398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.248334    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df"} err="failed to get container status \"7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df\": rpc error: code = NotFound desc = could not find container \"7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df\": container with ID starting with 7551e97480a6e42b48d79ab3e1f8ef6e9d5354b2f4b55d9be726ae659b1d94df not found: ID does not exist"
Sep 16 19:32:46.248398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.248360    2583 scope.go:117] "RemoveContainer" containerID="9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9"
Sep 16 19:32:46.248675 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:32:46.248656    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9\": container with ID starting with 9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9 not found: ID does not exist" containerID="9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9"
Sep 16 19:32:46.248723 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.248682    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9"} err="failed to get container status \"9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9\": rpc error: code = NotFound desc = could not find container \"9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9\": container with ID starting with 9c21c71e543b09b5b6b0c0373f6437e6ffdb52cda87a1db5bb051d6ac3eec0b9 not found: ID does not exist"
Sep 16 19:32:46.256041 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.256020    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"]
Sep 16 19:32:46.260174 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:46.260153    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-56bb6c6494-k5n8l"]
Sep 16 19:32:47.411877 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:32:47.411840    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" path="/var/lib/kubelet/pods/f4c1268c-a9da-47e5-af00-dc7e85866480/volumes"
Sep 16 19:33:56.425154 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.425056    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"]
Sep 16 19:33:56.426919 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.426893    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"]
Sep 16 19:33:56.436859 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:33:56.436826    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:33:56.436859 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:33:56.436859 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:33:56.436859 ip-10-0-139-148 kubenswrapper[2583]:  > containerID="83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3" cmd=["/bin/sleep","15"]
Sep 16 19:33:56.437142 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.436881    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k" message="" err=<
Sep 16 19:33:56.437142 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:33:56.437142 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:33:56.437142 ip-10-0-139-148 kubenswrapper[2583]:  >
Sep 16 19:33:56.437142 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:33:56.436898    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:33:56.437142 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:33:56.437142 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:33:56.437142 ip-10-0-139-148 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k" podUID="04c3493b-f83b-41ba-87e4-58345f26fc0b" containerName="main" containerID="cri-o://83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3"
Sep 16 19:33:56.437142 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.436940    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k" podUID="04c3493b-f83b-41ba-87e4-58345f26fc0b" containerName="main" containerID="cri-o://83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3" gracePeriod=60
Sep 16 19:33:56.784824 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.784803    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:33:56.922124 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbdlw\" (UniqueName: \"kubernetes.io/projected/04c3493b-f83b-41ba-87e4-58345f26fc0b-kube-api-access-bbdlw\") pod \"04c3493b-f83b-41ba-87e4-58345f26fc0b\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") "
Sep 16 19:33:56.922279 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c3493b-f83b-41ba-87e4-58345f26fc0b-tls-certs\") pod \"04c3493b-f83b-41ba-87e4-58345f26fc0b\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") "
Sep 16 19:33:56.922279 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922177    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-tmp-dir\") pod \"04c3493b-f83b-41ba-87e4-58345f26fc0b\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") "
Sep 16 19:33:56.922404 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922283    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-home\") pod \"04c3493b-f83b-41ba-87e4-58345f26fc0b\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") "
Sep 16 19:33:56.922404 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922364    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-dshm\") pod \"04c3493b-f83b-41ba-87e4-58345f26fc0b\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") "
Sep 16 19:33:56.922404 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922389    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-model-cache\") pod \"04c3493b-f83b-41ba-87e4-58345f26fc0b\" (UID: \"04c3493b-f83b-41ba-87e4-58345f26fc0b\") "
Sep 16 19:33:56.922404 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922387    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04c3493b-f83b-41ba-87e4-58345f26fc0b" (UID: "04c3493b-f83b-41ba-87e4-58345f26fc0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:56.922649 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922553    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-home" (OuterVolumeSpecName: "home") pod "04c3493b-f83b-41ba-87e4-58345f26fc0b" (UID: "04c3493b-f83b-41ba-87e4-58345f26fc0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:56.922649 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922638    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "04c3493b-f83b-41ba-87e4-58345f26fc0b" (UID: "04c3493b-f83b-41ba-87e4-58345f26fc0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:56.922788 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922769    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:33:56.922826 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922794    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:33:56.922826 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.922809    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:33:56.924560 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.924540    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c3493b-f83b-41ba-87e4-58345f26fc0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04c3493b-f83b-41ba-87e4-58345f26fc0b" (UID: "04c3493b-f83b-41ba-87e4-58345f26fc0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:56.924632 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.924582    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-dshm" (OuterVolumeSpecName: "dshm") pod "04c3493b-f83b-41ba-87e4-58345f26fc0b" (UID: "04c3493b-f83b-41ba-87e4-58345f26fc0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:56.924632 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:56.924624    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c3493b-f83b-41ba-87e4-58345f26fc0b-kube-api-access-bbdlw" (OuterVolumeSpecName: "kube-api-access-bbdlw") pod "04c3493b-f83b-41ba-87e4-58345f26fc0b" (UID: "04c3493b-f83b-41ba-87e4-58345f26fc0b"). InnerVolumeSpecName "kube-api-access-bbdlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:57.023397 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.023331    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c3493b-f83b-41ba-87e4-58345f26fc0b-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:33:57.023397 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.023356    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbdlw\" (UniqueName: \"kubernetes.io/projected/04c3493b-f83b-41ba-87e4-58345f26fc0b-kube-api-access-bbdlw\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:33:57.023397 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.023370    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c3493b-f83b-41ba-87e4-58345f26fc0b-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:33:57.565168 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.565137    2583 generic.go:356] "Generic (PLEG): container finished" podID="04c3493b-f83b-41ba-87e4-58345f26fc0b" containerID="83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3" exitCode=0
Sep 16 19:33:57.565515 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.565176    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"
Sep 16 19:33:57.565515 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.565218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k" event={"ID":"04c3493b-f83b-41ba-87e4-58345f26fc0b","Type":"ContainerDied","Data":"83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3"}
Sep 16 19:33:57.565515 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.565269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k" event={"ID":"04c3493b-f83b-41ba-87e4-58345f26fc0b","Type":"ContainerDied","Data":"d01f755fe22686eaf9db3388524f455f318bd3206395262632e22e206d5bcf7f"}
Sep 16 19:33:57.565515 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.565291    2583 scope.go:117] "RemoveContainer" containerID="83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3"
Sep 16 19:33:57.574381 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.574363    2583 scope.go:117] "RemoveContainer" containerID="83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3"
Sep 16 19:33:57.574700 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:33:57.574683    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3\": container with ID starting with 83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3 not found: ID does not exist" containerID="83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3"
Sep 16 19:33:57.574749 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.574706    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3"} err="failed to get container status \"83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3\": rpc error: code = NotFound desc = could not find container \"83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3\": container with ID starting with 83c8cbbed712c76f7e70de403e5d72667ab3d2fff348175ecc9d1dd845f8e4b3 not found: ID does not exist"
Sep 16 19:33:57.585901 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.585873    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"]
Sep 16 19:33:57.592660 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:57.592639    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-fvf6k"]
Sep 16 19:33:59.412651 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:59.412615    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04c3493b-f83b-41ba-87e4-58345f26fc0b" path="/var/lib/kubelet/pods/04c3493b-f83b-41ba-87e4-58345f26fc0b/volumes"
Sep 16 19:33:59.905798 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:33:59.905759    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:09.905360 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:09.905320    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:11.448952 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:11.448911    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerName="main" containerID="cri-o://417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630" gracePeriod=45
Sep 16 19:34:19.905208 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:19.905172    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:19.905649 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:19.905310    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:34:29.905453 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:29.905382    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:39.905718 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:39.905674    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:41.696841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.696819    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:34:41.782101 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.782042    2583 generic.go:356] "Generic (PLEG): container finished" podID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerID="417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630" exitCode=0
Sep 16 19:34:41.782101 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.782082    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"
Sep 16 19:34:41.782101 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.782096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" event={"ID":"f13e5c65-a04c-489a-9a45-d3165cce2e95","Type":"ContainerDied","Data":"417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630"}
Sep 16 19:34:41.782311 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.782137    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk" event={"ID":"f13e5c65-a04c-489a-9a45-d3165cce2e95","Type":"ContainerDied","Data":"0560d5992ae47cb9b3ba2874d7fcb86cf2d56f9576b6b7af5f5345dcd0b25252"}
Sep 16 19:34:41.782311 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.782160    2583 scope.go:117] "RemoveContainer" containerID="417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630"
Sep 16 19:34:41.790179 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.790164    2583 scope.go:117] "RemoveContainer" containerID="417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630"
Sep 16 19:34:41.790493 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:34:41.790475    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630\": container with ID starting with 417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630 not found: ID does not exist" containerID="417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630"
Sep 16 19:34:41.790556 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.790500    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630"} err="failed to get container status \"417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630\": rpc error: code = NotFound desc = could not find container \"417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630\": container with ID starting with 417b6a9ed9fd28082be8ab8e10a27ff6df86cb8bfaa4442eac16834e1d2a4630 not found: ID does not exist"
Sep 16 19:34:41.871827 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.871795    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f13e5c65-a04c-489a-9a45-d3165cce2e95-tls-certs\") pod \"f13e5c65-a04c-489a-9a45-d3165cce2e95\" (UID: \"f13e5c65-a04c-489a-9a45-d3165cce2e95\") "
Sep 16 19:34:41.871929 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.871916    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtmn7\" (UniqueName: \"kubernetes.io/projected/f13e5c65-a04c-489a-9a45-d3165cce2e95-kube-api-access-dtmn7\") pod \"f13e5c65-a04c-489a-9a45-d3165cce2e95\" (UID: \"f13e5c65-a04c-489a-9a45-d3165cce2e95\") "
Sep 16 19:34:41.873882 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.873856    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f13e5c65-a04c-489a-9a45-d3165cce2e95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f13e5c65-a04c-489a-9a45-d3165cce2e95" (UID: "f13e5c65-a04c-489a-9a45-d3165cce2e95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:41.873979 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.873964    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13e5c65-a04c-489a-9a45-d3165cce2e95-kube-api-access-dtmn7" (OuterVolumeSpecName: "kube-api-access-dtmn7") pod "f13e5c65-a04c-489a-9a45-d3165cce2e95" (UID: "f13e5c65-a04c-489a-9a45-d3165cce2e95"). InnerVolumeSpecName "kube-api-access-dtmn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:41.973004 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.972981    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtmn7\" (UniqueName: \"kubernetes.io/projected/f13e5c65-a04c-489a-9a45-d3165cce2e95-kube-api-access-dtmn7\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:34:41.973004 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:41.973004    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f13e5c65-a04c-489a-9a45-d3165cce2e95-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:34:42.107637 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:42.107604    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"]
Sep 16 19:34:42.110996 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:42.110972    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-55z67gk"]
Sep 16 19:34:43.410926 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:43.410897    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" path="/var/lib/kubelet/pods/f13e5c65-a04c-489a-9a45-d3165cce2e95/volumes"
Sep 16 19:34:56.784929 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.784894    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"]
Sep 16 19:34:56.785289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785206    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main"
Sep 16 19:34:56.785289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785218    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main"
Sep 16 19:34:56.785289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785230    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c3493b-f83b-41ba-87e4-58345f26fc0b" containerName="main"
Sep 16 19:34:56.785289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785235    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c3493b-f83b-41ba-87e4-58345f26fc0b" containerName="main"
Sep 16 19:34:56.785289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785245    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerName="main"
Sep 16 19:34:56.785289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785250    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerName="main"
Sep 16 19:34:56.785289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785267    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="storage-initializer"
Sep 16 19:34:56.785289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785273    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="storage-initializer"
Sep 16 19:34:56.785573 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785325    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c1268c-a9da-47e5-af00-dc7e85866480" containerName="main"
Sep 16 19:34:56.785573 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785333    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c3493b-f83b-41ba-87e4-58345f26fc0b" containerName="main"
Sep 16 19:34:56.785573 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.785343    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f13e5c65-a04c-489a-9a45-d3165cce2e95" containerName="main"
Sep 16 19:34:56.788041 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.788020    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.793184 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.793161    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:34:56.793786 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.793767    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:34:56.794586 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.794289    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:34:56.794586 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.794300    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-fkqtl\""
Sep 16 19:34:56.808819 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.808799    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"]
Sep 16 19:34:56.884081 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.884055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.884190 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.884094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk66q\" (UniqueName: \"kubernetes.io/projected/83d6c6fc-ee1a-4c36-a344-6463f2786d63-kube-api-access-lk66q\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.884190 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.884154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.884190 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.884180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.884324 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.884209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.884324 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.884249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.921720 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.921696    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"]
Sep 16 19:34:56.925043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.925027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:34:56.928246 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.928231    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-zfwls\""
Sep 16 19:34:56.936096 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.936073    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"]
Sep 16 19:34:56.985053 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.985182 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.985182 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.985182 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.985182 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf\" (UID: \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:34:56.985442 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985185    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrx99\" (UniqueName: \"kubernetes.io/projected/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-kube-api-access-zrx99\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf\" (UID: \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:34:56.985442 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.985442 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk66q\" (UniqueName: \"kubernetes.io/projected/83d6c6fc-ee1a-4c36-a344-6463f2786d63-kube-api-access-lk66q\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.985614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.985614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.985703 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.985655    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.987571 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.987550    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.987881 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.987862    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:56.995283 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:56.995262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk66q\" (UniqueName: \"kubernetes.io/projected/83d6c6fc-ee1a-4c36-a344-6463f2786d63-kube-api-access-lk66q\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:57.086185 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.086127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf\" (UID: \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:34:57.086185 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.086170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrx99\" (UniqueName: \"kubernetes.io/projected/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-kube-api-access-zrx99\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf\" (UID: \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:34:57.088610 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.088589    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf\" (UID: \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:34:57.098114 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.098096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:34:57.098912 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.098886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrx99\" (UniqueName: \"kubernetes.io/projected/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-kube-api-access-zrx99\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf\" (UID: \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:34:57.235647 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.235617    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:34:57.376700 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.376678    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"]
Sep 16 19:34:57.378286 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:34:57.378260    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d6c6fc_ee1a_4c36_a344_6463f2786d63.slice/crio-bb49cb5db023fcc781f198990ca927f710ed1bac491d5b8234d490e1b1b3b147 WatchSource:0}: Error finding container bb49cb5db023fcc781f198990ca927f710ed1bac491d5b8234d490e1b1b3b147: Status 404 returned error can't find the container with id bb49cb5db023fcc781f198990ca927f710ed1bac491d5b8234d490e1b1b3b147
Sep 16 19:34:57.518075 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.518053    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"]
Sep 16 19:34:57.859372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.859277    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" event={"ID":"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83","Type":"ContainerStarted","Data":"0630b6ec7f86b9e4f5774ea940e9b16e8b40806da1c4f820a78389167dd480a8"}
Sep 16 19:34:57.859372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.859341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" event={"ID":"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83","Type":"ContainerStarted","Data":"d06b53d6caa759dc6c659af795a3ffacfbef8ea8d6b09a22a25063945cef0747"}
Sep 16 19:34:57.859372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.859365    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:34:57.860944 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.860915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd" event={"ID":"83d6c6fc-ee1a-4c36-a344-6463f2786d63","Type":"ContainerStarted","Data":"34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf"}
Sep 16 19:34:57.861078 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.860960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd" event={"ID":"83d6c6fc-ee1a-4c36-a344-6463f2786d63","Type":"ContainerStarted","Data":"bb49cb5db023fcc781f198990ca927f710ed1bac491d5b8234d490e1b1b3b147"}
Sep 16 19:34:57.959183 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.959125    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" podStartSLOduration=1.959091533 podStartE2EDuration="1.959091533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:57.914759148 +0000 UTC m=+1775.040563536" watchObservedRunningTime="2026-09-16 19:34:57.959091533 +0000 UTC m=+1775.084895924"
Sep 16 19:34:57.959835 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:57.959785    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd" podStartSLOduration=1.959769891 podStartE2EDuration="1.959769891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:57.955598246 +0000 UTC m=+1775.081402631" watchObservedRunningTime="2026-09-16 19:34:57.959769891 +0000 UTC m=+1775.085574280"
Sep 16 19:34:58.098505 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:34:58.098470    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:35:07.099179 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:07.099134    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:35:07.112372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:07.112345    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:35:07.920426 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:07.920387    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:35:26.818470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:26.818435    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:35:26.824470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:26.824443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:35:26.824809 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:26.824786    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:35:26.830688 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:26.830667    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:35:28.867700 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:28.867671    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:35:32.088107 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.088047    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"]
Sep 16 19:35:32.097830 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:35:32.097785    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:35:32.097830 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:35:32.097830 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:35:32.097830 ip-10-0-139-148 kubenswrapper[2583]:  > containerID="34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf" cmd=["/bin/sleep","15"]
Sep 16 19:35:32.098252 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.097856    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd" message="" err=<
Sep 16 19:35:32.098252 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:35:32.098252 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:35:32.098252 ip-10-0-139-148 kubenswrapper[2583]:  >
Sep 16 19:35:32.098252 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:35:32.097873    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:35:32.098252 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:35:32.098252 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:35:32.098252 ip-10-0-139-148 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd" podUID="83d6c6fc-ee1a-4c36-a344-6463f2786d63" containerName="main" containerID="cri-o://34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf"
Sep 16 19:35:32.098252 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.097930    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd" podUID="83d6c6fc-ee1a-4c36-a344-6463f2786d63" containerName="main" containerID="cri-o://34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf" gracePeriod=60
Sep 16 19:35:32.098739 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.098707    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"]
Sep 16 19:35:32.846754 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.846730    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:35:32.974868 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.974806    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk66q\" (UniqueName: \"kubernetes.io/projected/83d6c6fc-ee1a-4c36-a344-6463f2786d63-kube-api-access-lk66q\") pod \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") "
Sep 16 19:35:32.974868 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.974836    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-dshm\") pod \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") "
Sep 16 19:35:32.975054 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.974899    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-model-cache\") pod \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") "
Sep 16 19:35:32.975054 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.974935    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tls-certs\") pod \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") "
Sep 16 19:35:32.975054 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.974957    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-home\") pod \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") "
Sep 16 19:35:32.975054 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.975027    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tmp-dir\") pod \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\" (UID: \"83d6c6fc-ee1a-4c36-a344-6463f2786d63\") "
Sep 16 19:35:32.975262 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.975229    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-model-cache" (OuterVolumeSpecName: "model-cache") pod "83d6c6fc-ee1a-4c36-a344-6463f2786d63" (UID: "83d6c6fc-ee1a-4c36-a344-6463f2786d63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:32.975329 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.975278    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-home" (OuterVolumeSpecName: "home") pod "83d6c6fc-ee1a-4c36-a344-6463f2786d63" (UID: "83d6c6fc-ee1a-4c36-a344-6463f2786d63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:32.975394 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.975330    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83d6c6fc-ee1a-4c36-a344-6463f2786d63" (UID: "83d6c6fc-ee1a-4c36-a344-6463f2786d63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:32.977200 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.977178    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d6c6fc-ee1a-4c36-a344-6463f2786d63-kube-api-access-lk66q" (OuterVolumeSpecName: "kube-api-access-lk66q") pod "83d6c6fc-ee1a-4c36-a344-6463f2786d63" (UID: "83d6c6fc-ee1a-4c36-a344-6463f2786d63"). InnerVolumeSpecName "kube-api-access-lk66q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:32.977299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.977216    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83d6c6fc-ee1a-4c36-a344-6463f2786d63" (UID: "83d6c6fc-ee1a-4c36-a344-6463f2786d63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:32.977299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:32.977272    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-dshm" (OuterVolumeSpecName: "dshm") pod "83d6c6fc-ee1a-4c36-a344-6463f2786d63" (UID: "83d6c6fc-ee1a-4c36-a344-6463f2786d63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:33.036883 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.036854    2583 generic.go:356] "Generic (PLEG): container finished" podID="83d6c6fc-ee1a-4c36-a344-6463f2786d63" containerID="34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf" exitCode=0
Sep 16 19:35:33.036979 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.036901    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"
Sep 16 19:35:33.036979 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.036944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd" event={"ID":"83d6c6fc-ee1a-4c36-a344-6463f2786d63","Type":"ContainerDied","Data":"34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf"}
Sep 16 19:35:33.037068 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.036999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd" event={"ID":"83d6c6fc-ee1a-4c36-a344-6463f2786d63","Type":"ContainerDied","Data":"bb49cb5db023fcc781f198990ca927f710ed1bac491d5b8234d490e1b1b3b147"}
Sep 16 19:35:33.037068 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.037023    2583 scope.go:117] "RemoveContainer" containerID="34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf"
Sep 16 19:35:33.046160 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.046139    2583 scope.go:117] "RemoveContainer" containerID="34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf"
Sep 16 19:35:33.046476 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:35:33.046459    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf\": container with ID starting with 34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf not found: ID does not exist" containerID="34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf"
Sep 16 19:35:33.046546 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.046485    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf"} err="failed to get container status \"34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf\": rpc error: code = NotFound desc = could not find container \"34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf\": container with ID starting with 34585abf1584c8996a22261fe719f2558eac882f8346774cb9761080760dbcaf not found: ID does not exist"
Sep 16 19:35:33.063627 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.063594    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"]
Sep 16 19:35:33.065378 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.065359    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444dd7bd"]
Sep 16 19:35:33.076538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.076505    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:35:33.076538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.076535    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk66q\" (UniqueName: \"kubernetes.io/projected/83d6c6fc-ee1a-4c36-a344-6463f2786d63-kube-api-access-lk66q\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:35:33.076538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.076544    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:35:33.076697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.076553    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:35:33.076697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.076562    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d6c6fc-ee1a-4c36-a344-6463f2786d63-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:35:33.076697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.076569    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d6c6fc-ee1a-4c36-a344-6463f2786d63-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:35:33.411095 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:33.411066    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d6c6fc-ee1a-4c36-a344-6463f2786d63" path="/var/lib/kubelet/pods/83d6c6fc-ee1a-4c36-a344-6463f2786d63/volumes"
Sep 16 19:35:38.867670 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:38.867629    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" podUID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:47.117538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:47.117486    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" podUID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" containerName="main" containerID="cri-o://0630b6ec7f86b9e4f5774ea940e9b16e8b40806da1c4f820a78389167dd480a8" gracePeriod=45
Sep 16 19:35:48.867224 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:48.867179    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" podUID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:58.867678 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:58.867633    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" podUID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:58.868042 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:35:58.867732    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:36:08.867858 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:08.867809    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" podUID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:17.256067 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:17.256031    2583 generic.go:356] "Generic (PLEG): container finished" podID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" containerID="0630b6ec7f86b9e4f5774ea940e9b16e8b40806da1c4f820a78389167dd480a8" exitCode=0
Sep 16 19:36:17.256439 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:17.256097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" event={"ID":"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83","Type":"ContainerDied","Data":"0630b6ec7f86b9e4f5774ea940e9b16e8b40806da1c4f820a78389167dd480a8"}
Sep 16 19:36:17.367590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:17.367566    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:36:17.417183 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:17.417130    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrx99\" (UniqueName: \"kubernetes.io/projected/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-kube-api-access-zrx99\") pod \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\" (UID: \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\") "
Sep 16 19:36:17.417288 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:17.417231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-tls-certs\") pod \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\" (UID: \"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83\") "
Sep 16 19:36:17.419382 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:17.419350    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-kube-api-access-zrx99" (OuterVolumeSpecName: "kube-api-access-zrx99") pod "a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" (UID: "a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83"). InnerVolumeSpecName "kube-api-access-zrx99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:17.419553 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:17.419532    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" (UID: "a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:17.518538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:17.518510    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:36:17.518668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:17.518542    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrx99\" (UniqueName: \"kubernetes.io/projected/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83-kube-api-access-zrx99\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.263032 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:18.262992    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"
Sep 16 19:36:18.263544 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:18.263029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf" event={"ID":"a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83","Type":"ContainerDied","Data":"d06b53d6caa759dc6c659af795a3ffacfbef8ea8d6b09a22a25063945cef0747"}
Sep 16 19:36:18.263544 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:18.263105    2583 scope.go:117] "RemoveContainer" containerID="0630b6ec7f86b9e4f5774ea940e9b16e8b40806da1c4f820a78389167dd480a8"
Sep 16 19:36:18.288825 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:18.288800    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"]
Sep 16 19:36:18.290308 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:18.290284    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt8tdf"]
Sep 16 19:36:19.411652 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:19.411620    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" path="/var/lib/kubelet/pods/a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83/volumes"
Sep 16 19:36:41.188383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.188347    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"]
Sep 16 19:36:41.188923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.188732    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" containerName="main"
Sep 16 19:36:41.188923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.188744    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" containerName="main"
Sep 16 19:36:41.188923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.188754    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d6c6fc-ee1a-4c36-a344-6463f2786d63" containerName="main"
Sep 16 19:36:41.188923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.188759    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d6c6fc-ee1a-4c36-a344-6463f2786d63" containerName="main"
Sep 16 19:36:41.188923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.188818    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d6c6fc-ee1a-4c36-a344-6463f2786d63" containerName="main"
Sep 16 19:36:41.188923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.188827    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f0a35b-5f7e-4bdc-9a66-d2951bba8d83" containerName="main"
Sep 16 19:36:41.196711 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.196686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.201975 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.201950    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:36:41.202119 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.202009    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:36:41.202930 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.202912    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-4vqpk\""
Sep 16 19:36:41.203077 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.202929    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:36:41.203077 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.202934    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:36:41.209338 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.209321    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:36:41.243317 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.243287    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"]
Sep 16 19:36:41.322160 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.322123    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.322330 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.322175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133da96d-a135-4e88-b845-19e76d205612-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.322330 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.322237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-model-cache\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.322330 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.322288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-home\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.322330 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.322313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133da96d-a135-4e88-b845-19e76d205612-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.322503 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.322381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnfmh\" (UniqueName: \"kubernetes.io/projected/133da96d-a135-4e88-b845-19e76d205612-kube-api-access-mnfmh\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.322503 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.322446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.322503 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.322466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-dshm\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.423567 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-model-cache\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.423760 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-home\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.423760 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133da96d-a135-4e88-b845-19e76d205612-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.423760 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnfmh\" (UniqueName: \"kubernetes.io/projected/133da96d-a135-4e88-b845-19e76d205612-kube-api-access-mnfmh\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.423760 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.423760 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-dshm\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.423760 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.424082 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133da96d-a135-4e88-b845-19e76d205612-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.424082 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-model-cache\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.424082 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.423977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-home\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.424244 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.424096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.424244 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.424144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.424528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.424508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133da96d-a135-4e88-b845-19e76d205612-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.426131 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.426109    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-dshm\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.426273 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.426258    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133da96d-a135-4e88-b845-19e76d205612-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.433854 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.433832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnfmh\" (UniqueName: \"kubernetes.io/projected/133da96d-a135-4e88-b845-19e76d205612-kube-api-access-mnfmh\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.507720 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.507658    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:41.732557 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.732526    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"]
Sep 16 19:36:41.736832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.736815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:36:41.740140 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.740117    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-47k88\""
Sep 16 19:36:41.750344 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.750311    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"]
Sep 16 19:36:41.797823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.797747    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"]
Sep 16 19:36:41.801921 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:36:41.801897    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133da96d_a135_4e88_b845_19e76d205612.slice/crio-8456f98164edf3072d514cc44c2b3884c8ad6ce5470881df47066be8b736f99c WatchSource:0}: Error finding container 8456f98164edf3072d514cc44c2b3884c8ad6ce5470881df47066be8b736f99c: Status 404 returned error can't find the container with id 8456f98164edf3072d514cc44c2b3884c8ad6ce5470881df47066be8b736f99c
Sep 16 19:36:41.827831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.827805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf\" (UID: \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:36:41.827935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.827866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkbd\" (UniqueName: \"kubernetes.io/projected/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-kube-api-access-djkbd\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf\" (UID: \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:36:41.929058 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.929021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf\" (UID: \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:36:41.929240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.929108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkbd\" (UniqueName: \"kubernetes.io/projected/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-kube-api-access-djkbd\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf\" (UID: \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:36:41.931845 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.931815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf\" (UID: \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:36:41.942270 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:41.942249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkbd\" (UniqueName: \"kubernetes.io/projected/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-kube-api-access-djkbd\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf\" (UID: \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:36:42.055926 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:42.055845    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:36:42.337700 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:42.337673    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"]
Sep 16 19:36:42.340026 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:36:42.339995    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0404b27_1d2b_41a7_b9f9_9e3ba8102b6a.slice/crio-2d134a635ebe823b9252e9dbaff60cf33fb4dfd14480839aeeaae800b314d2e3 WatchSource:0}: Error finding container 2d134a635ebe823b9252e9dbaff60cf33fb4dfd14480839aeeaae800b314d2e3: Status 404 returned error can't find the container with id 2d134a635ebe823b9252e9dbaff60cf33fb4dfd14480839aeeaae800b314d2e3
Sep 16 19:36:42.376463 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:42.376432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" event={"ID":"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a","Type":"ContainerStarted","Data":"2d134a635ebe823b9252e9dbaff60cf33fb4dfd14480839aeeaae800b314d2e3"}
Sep 16 19:36:42.377939 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:42.377909    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" event={"ID":"133da96d-a135-4e88-b845-19e76d205612","Type":"ContainerStarted","Data":"9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7"}
Sep 16 19:36:42.378057 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:42.377958    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" event={"ID":"133da96d-a135-4e88-b845-19e76d205612","Type":"ContainerStarted","Data":"8456f98164edf3072d514cc44c2b3884c8ad6ce5470881df47066be8b736f99c"}
Sep 16 19:36:43.385073 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:43.385021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" event={"ID":"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a","Type":"ContainerStarted","Data":"637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f"}
Sep 16 19:36:43.385473 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:43.385100    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:36:43.405658 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:43.405591    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" podStartSLOduration=2.405571524 podStartE2EDuration="2.405571524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:43.40404001 +0000 UTC m=+1880.529844398" watchObservedRunningTime="2026-09-16 19:36:43.405571524 +0000 UTC m=+1880.531375924"
Sep 16 19:36:44.391825 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:44.391786    2583 generic.go:356] "Generic (PLEG): container finished" podID="133da96d-a135-4e88-b845-19e76d205612" containerID="9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7" exitCode=0
Sep 16 19:36:44.392196 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:44.391861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" event={"ID":"133da96d-a135-4e88-b845-19e76d205612","Type":"ContainerDied","Data":"9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7"}
Sep 16 19:36:45.398356 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:45.398326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" event={"ID":"133da96d-a135-4e88-b845-19e76d205612","Type":"ContainerStarted","Data":"6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a"}
Sep 16 19:36:45.426989 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:45.426928    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" podStartSLOduration=2.090372332 podStartE2EDuration="4.42690968s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.336537348s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:45.426471725 +0000 UTC m=+1882.552276114" watchObservedRunningTime="2026-09-16 19:36:45.42690968 +0000 UTC m=+1882.552714068"
Sep 16 19:36:45.508486 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:45.508454    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:51.508250 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:51.508210    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:51.520822 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:51.520795    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:36:52.442602 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:36:52.442574    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:37:14.394181 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:37:14.394149    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:38:20.757822 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.757736    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"]
Sep 16 19:38:20.760696 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.760674    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.768554 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.768531    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-7twkp\""
Sep 16 19:38:20.769797 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.769777    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:38:20.769922 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.769813    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:38:20.769922 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.769776    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 16 19:38:20.780439 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.780399    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"]
Sep 16 19:38:20.838496 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.838476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422dc654-a772-4802-aa0a-1447c8510c9e-tls-certs\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.838607 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.838504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxh9v\" (UniqueName: \"kubernetes.io/projected/422dc654-a772-4802-aa0a-1447c8510c9e-kube-api-access-lxh9v\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.838607 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.838525    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-model-cache\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.838736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.838624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.838736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.838674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-dshm\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.838736 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.838707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-home\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.939114 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.939092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422dc654-a772-4802-aa0a-1447c8510c9e-tls-certs\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.939240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.939123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxh9v\" (UniqueName: \"kubernetes.io/projected/422dc654-a772-4802-aa0a-1447c8510c9e-kube-api-access-lxh9v\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.939240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.939148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-model-cache\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.939240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.939192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.939240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.939216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-dshm\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.939536 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.939240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-home\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.939815 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.939788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-home\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.939935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.939843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-model-cache\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.939935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.939874    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.942090 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.942059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422dc654-a772-4802-aa0a-1447c8510c9e-tls-certs\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.942351 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.942324    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-dshm\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:20.949312 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:20.949289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxh9v\" (UniqueName: \"kubernetes.io/projected/422dc654-a772-4802-aa0a-1447c8510c9e-kube-api-access-lxh9v\") pod \"stop-v1-kserve-5657b88c95-8jcgk\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:21.071521 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:21.071459    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:21.371111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:21.371086    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"]
Sep 16 19:38:21.374659 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:38:21.374629    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422dc654_a772_4802_aa0a_1447c8510c9e.slice/crio-16c1a63933d3ea2e84308f2fceae8c712bc8c8e47585aa138c45b1474afd48c4 WatchSource:0}: Error finding container 16c1a63933d3ea2e84308f2fceae8c712bc8c8e47585aa138c45b1474afd48c4: Status 404 returned error can't find the container with id 16c1a63933d3ea2e84308f2fceae8c712bc8c8e47585aa138c45b1474afd48c4
Sep 16 19:38:21.376664 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:21.376644    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:38:21.870961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:21.870912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk" event={"ID":"422dc654-a772-4802-aa0a-1447c8510c9e","Type":"ContainerStarted","Data":"16c1a63933d3ea2e84308f2fceae8c712bc8c8e47585aa138c45b1474afd48c4"}
Sep 16 19:38:22.877902 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:22.877867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk" event={"ID":"422dc654-a772-4802-aa0a-1447c8510c9e","Type":"ContainerStarted","Data":"49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003"}
Sep 16 19:38:22.902539 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:22.902484    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk" podStartSLOduration=1.849366955 podStartE2EDuration="2.902469334s" totalImagesPullingTime="1.053102379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:22.900154033 +0000 UTC m=+1980.025958422" watchObservedRunningTime="2026-09-16 19:38:22.902469334 +0000 UTC m=+1980.028273721"
Sep 16 19:38:23.071691 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:23.071664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:31.072517 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:31.072475    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:31.092446 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:31.092392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:31.936147 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:31.936110    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:38:48.253869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.253837    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"]
Sep 16 19:38:48.263732 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:38:48.263695    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:38:48.263732 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:38:48.263732 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:38:48.263732 ip-10-0-139-148 kubenswrapper[2583]:  > containerID="6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a" cmd=["/bin/sleep","15"]
Sep 16 19:38:48.264055 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.263748    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" message="" err=<
Sep 16 19:38:48.264055 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:38:48.264055 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:38:48.264055 ip-10-0-139-148 kubenswrapper[2583]:  >
Sep 16 19:38:48.264055 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:38:48.263761    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:38:48.264055 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:38:48.264055 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:38:48.264055 ip-10-0-139-148 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" podUID="133da96d-a135-4e88-b845-19e76d205612" containerName="main" containerID="cri-o://6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a"
Sep 16 19:38:48.264055 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.263795    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" podUID="133da96d-a135-4e88-b845-19e76d205612" containerName="main" containerID="cri-o://6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a" gracePeriod=60
Sep 16 19:38:48.264540 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.264208    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"]
Sep 16 19:38:48.509758 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.509694    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:38:48.572755 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.572729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133da96d-a135-4e88-b845-19e76d205612-tls-certs\") pod \"133da96d-a135-4e88-b845-19e76d205612\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") "
Sep 16 19:38:48.572881 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.572759    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-home\") pod \"133da96d-a135-4e88-b845-19e76d205612\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") "
Sep 16 19:38:48.572928 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.572890    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-tmp-dir\") pod \"133da96d-a135-4e88-b845-19e76d205612\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") "
Sep 16 19:38:48.573017 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.572941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-model-cache\") pod \"133da96d-a135-4e88-b845-19e76d205612\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") "
Sep 16 19:38:48.573017 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.572977    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133da96d-a135-4e88-b845-19e76d205612-cabundle-cert\") pod \"133da96d-a135-4e88-b845-19e76d205612\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") "
Sep 16 19:38:48.573017 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.572995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnfmh\" (UniqueName: \"kubernetes.io/projected/133da96d-a135-4e88-b845-19e76d205612-kube-api-access-mnfmh\") pod \"133da96d-a135-4e88-b845-19e76d205612\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") "
Sep 16 19:38:48.573017 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.572995    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-home" (OuterVolumeSpecName: "home") pod "133da96d-a135-4e88-b845-19e76d205612" (UID: "133da96d-a135-4e88-b845-19e76d205612"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:48.573226 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.573021    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-kserve-provision-location\") pod \"133da96d-a135-4e88-b845-19e76d205612\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") "
Sep 16 19:38:48.573226 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.573044    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-dshm\") pod \"133da96d-a135-4e88-b845-19e76d205612\" (UID: \"133da96d-a135-4e88-b845-19e76d205612\") "
Sep 16 19:38:48.573226 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.573170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "133da96d-a135-4e88-b845-19e76d205612" (UID: "133da96d-a135-4e88-b845-19e76d205612"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:48.573391 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.573292    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-model-cache" (OuterVolumeSpecName: "model-cache") pod "133da96d-a135-4e88-b845-19e76d205612" (UID: "133da96d-a135-4e88-b845-19e76d205612"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:48.573391 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.573362    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133da96d-a135-4e88-b845-19e76d205612-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "133da96d-a135-4e88-b845-19e76d205612" (UID: "133da96d-a135-4e88-b845-19e76d205612"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:38:48.573391 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.573367    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.573571 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.573400    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.575369 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.575345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133da96d-a135-4e88-b845-19e76d205612-kube-api-access-mnfmh" (OuterVolumeSpecName: "kube-api-access-mnfmh") pod "133da96d-a135-4e88-b845-19e76d205612" (UID: "133da96d-a135-4e88-b845-19e76d205612"). InnerVolumeSpecName "kube-api-access-mnfmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:48.575523 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.575438    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-dshm" (OuterVolumeSpecName: "dshm") pod "133da96d-a135-4e88-b845-19e76d205612" (UID: "133da96d-a135-4e88-b845-19e76d205612"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:48.575690 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.575661    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133da96d-a135-4e88-b845-19e76d205612-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "133da96d-a135-4e88-b845-19e76d205612" (UID: "133da96d-a135-4e88-b845-19e76d205612"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:48.601559 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.601532    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "133da96d-a135-4e88-b845-19e76d205612" (UID: "133da96d-a135-4e88-b845-19e76d205612"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:48.674755 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.674724    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.674755 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.674751    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133da96d-a135-4e88-b845-19e76d205612-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.674891 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.674764    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnfmh\" (UniqueName: \"kubernetes.io/projected/133da96d-a135-4e88-b845-19e76d205612-kube-api-access-mnfmh\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.674891 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.674777    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.674891 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.674789    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133da96d-a135-4e88-b845-19e76d205612-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:38:48.674891 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:48.674802    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133da96d-a135-4e88-b845-19e76d205612-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:38:49.013599 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.013563    2583 generic.go:356] "Generic (PLEG): container finished" podID="133da96d-a135-4e88-b845-19e76d205612" containerID="6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a" exitCode=0
Sep 16 19:38:49.013739 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.013617    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"
Sep 16 19:38:49.013739 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.013635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" event={"ID":"133da96d-a135-4e88-b845-19e76d205612","Type":"ContainerDied","Data":"6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a"}
Sep 16 19:38:49.013739 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.013689    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" event={"ID":"133da96d-a135-4e88-b845-19e76d205612","Type":"ContainerDied","Data":"8456f98164edf3072d514cc44c2b3884c8ad6ce5470881df47066be8b736f99c"}
Sep 16 19:38:49.013739 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.013719    2583 scope.go:117] "RemoveContainer" containerID="6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a"
Sep 16 19:38:49.023828 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.023812    2583 scope.go:117] "RemoveContainer" containerID="9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7"
Sep 16 19:38:49.031273 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.031255    2583 scope.go:117] "RemoveContainer" containerID="6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a"
Sep 16 19:38:49.031578 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:38:49.031559    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a\": container with ID starting with 6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a not found: ID does not exist" containerID="6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a"
Sep 16 19:38:49.031653 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.031585    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a"} err="failed to get container status \"6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a\": rpc error: code = NotFound desc = could not find container \"6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a\": container with ID starting with 6004f3537cf7f584e2c20a6d546de0e46568562856b0906f2aecfa288f0e3a5a not found: ID does not exist"
Sep 16 19:38:49.031653 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.031602    2583 scope.go:117] "RemoveContainer" containerID="9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7"
Sep 16 19:38:49.031837 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:38:49.031822    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7\": container with ID starting with 9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7 not found: ID does not exist" containerID="9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7"
Sep 16 19:38:49.031879 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.031840    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7"} err="failed to get container status \"9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7\": rpc error: code = NotFound desc = could not find container \"9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7\": container with ID starting with 9ef225f4b138fd7c1cc1aabd843562af853f21ec942e982a5465395ef56466d7 not found: ID does not exist"
Sep 16 19:38:49.038533 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.038512    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"]
Sep 16 19:38:49.044319 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.044298    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k"]
Sep 16 19:38:49.411657 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.411615    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="133da96d-a135-4e88-b845-19e76d205612" path="/var/lib/kubelet/pods/133da96d-a135-4e88-b845-19e76d205612/volumes"
Sep 16 19:38:49.432644 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:49.432613    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-2rc7k" podUID="133da96d-a135-4e88-b845-19e76d205612" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:38:54.393741 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:38:54.393698    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" podUID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:03.282720 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:03.282667    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" podUID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" containerName="main" containerID="cri-o://637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f" gracePeriod=45
Sep 16 19:39:04.400505 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:04.400460    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" podUID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:14.394052 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:14.394011    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" podUID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:14.394482 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:14.394138    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:39:24.394182 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:24.394139    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" podUID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:25.199672 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:25.199641    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"]
Sep 16 19:39:33.539495 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:33.539470    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:39:33.654011 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:33.653986    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-tls-certs\") pod \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\" (UID: \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\") "
Sep 16 19:39:33.654165 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:33.654045    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djkbd\" (UniqueName: \"kubernetes.io/projected/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-kube-api-access-djkbd\") pod \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\" (UID: \"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a\") "
Sep 16 19:39:33.656186 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:33.656157    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" (UID: "d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:39:33.656304 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:33.656256    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-kube-api-access-djkbd" (OuterVolumeSpecName: "kube-api-access-djkbd") pod "d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" (UID: "d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a"). InnerVolumeSpecName "kube-api-access-djkbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:39:33.755467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:33.755444    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:39:33.755467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:33.755463    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djkbd\" (UniqueName: \"kubernetes.io/projected/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a-kube-api-access-djkbd\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:39:34.232109 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:34.232079    2583 generic.go:356] "Generic (PLEG): container finished" podID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" containerID="637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f" exitCode=0
Sep 16 19:39:34.232233 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:34.232126    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"
Sep 16 19:39:34.232233 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:34.232168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" event={"ID":"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a","Type":"ContainerDied","Data":"637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f"}
Sep 16 19:39:34.232233 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:34.232224    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf" event={"ID":"d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a","Type":"ContainerDied","Data":"2d134a635ebe823b9252e9dbaff60cf33fb4dfd14480839aeeaae800b314d2e3"}
Sep 16 19:39:34.232342 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:34.232253    2583 scope.go:117] "RemoveContainer" containerID="637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f"
Sep 16 19:39:34.241172 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:34.241156    2583 scope.go:117] "RemoveContainer" containerID="637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f"
Sep 16 19:39:34.241456 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:39:34.241438    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f\": container with ID starting with 637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f not found: ID does not exist" containerID="637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f"
Sep 16 19:39:34.241514 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:34.241464    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f"} err="failed to get container status \"637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f\": rpc error: code = NotFound desc = could not find container \"637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f\": container with ID starting with 637c7e11d9a3a1cadc28ccf109bf140a2097577c6d3a3b14b708f4293718d84f not found: ID does not exist"
Sep 16 19:39:34.258337 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:34.258314    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"]
Sep 16 19:39:34.263111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:34.263090    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-6856648b96xnf"]
Sep 16 19:39:35.410941 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:35.410913    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" path="/var/lib/kubelet/pods/d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a/volumes"
Sep 16 19:39:40.219028 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.218978    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk" podUID="422dc654-a772-4802-aa0a-1447c8510c9e" containerName="main" containerID="cri-o://49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003" gracePeriod=45
Sep 16 19:39:40.466966 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.466942    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:39:40.504227 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.504164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-dshm\") pod \"422dc654-a772-4802-aa0a-1447c8510c9e\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") "
Sep 16 19:39:40.504227 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.504211    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxh9v\" (UniqueName: \"kubernetes.io/projected/422dc654-a772-4802-aa0a-1447c8510c9e-kube-api-access-lxh9v\") pod \"422dc654-a772-4802-aa0a-1447c8510c9e\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") "
Sep 16 19:39:40.504372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.504272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-tmp-dir\") pod \"422dc654-a772-4802-aa0a-1447c8510c9e\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") "
Sep 16 19:39:40.504372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.504293    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422dc654-a772-4802-aa0a-1447c8510c9e-tls-certs\") pod \"422dc654-a772-4802-aa0a-1447c8510c9e\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") "
Sep 16 19:39:40.504372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.504317    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-model-cache\") pod \"422dc654-a772-4802-aa0a-1447c8510c9e\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") "
Sep 16 19:39:40.504372 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.504338    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-home\") pod \"422dc654-a772-4802-aa0a-1447c8510c9e\" (UID: \"422dc654-a772-4802-aa0a-1447c8510c9e\") "
Sep 16 19:39:40.504621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.504599    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "422dc654-a772-4802-aa0a-1447c8510c9e" (UID: "422dc654-a772-4802-aa0a-1447c8510c9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:40.504702 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.504675    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "422dc654-a772-4802-aa0a-1447c8510c9e" (UID: "422dc654-a772-4802-aa0a-1447c8510c9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:40.504818 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.504792    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-home" (OuterVolumeSpecName: "home") pod "422dc654-a772-4802-aa0a-1447c8510c9e" (UID: "422dc654-a772-4802-aa0a-1447c8510c9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:40.506768 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.506742    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422dc654-a772-4802-aa0a-1447c8510c9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "422dc654-a772-4802-aa0a-1447c8510c9e" (UID: "422dc654-a772-4802-aa0a-1447c8510c9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:39:40.506934 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.506913    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-dshm" (OuterVolumeSpecName: "dshm") pod "422dc654-a772-4802-aa0a-1447c8510c9e" (UID: "422dc654-a772-4802-aa0a-1447c8510c9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:40.507000 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.506932    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422dc654-a772-4802-aa0a-1447c8510c9e-kube-api-access-lxh9v" (OuterVolumeSpecName: "kube-api-access-lxh9v") pod "422dc654-a772-4802-aa0a-1447c8510c9e" (UID: "422dc654-a772-4802-aa0a-1447c8510c9e"). InnerVolumeSpecName "kube-api-access-lxh9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:39:40.605420 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.605382    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:39:40.605538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.605450    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxh9v\" (UniqueName: \"kubernetes.io/projected/422dc654-a772-4802-aa0a-1447c8510c9e-kube-api-access-lxh9v\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:39:40.605538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.605466    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:39:40.605538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.605480    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422dc654-a772-4802-aa0a-1447c8510c9e-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:39:40.605538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.605492    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:39:40.605538 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:40.605503    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422dc654-a772-4802-aa0a-1447c8510c9e-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:39:41.273869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.273831    2583 generic.go:356] "Generic (PLEG): container finished" podID="422dc654-a772-4802-aa0a-1447c8510c9e" containerID="49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003" exitCode=0
Sep 16 19:39:41.274306 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.273880    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"
Sep 16 19:39:41.274306 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.273912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk" event={"ID":"422dc654-a772-4802-aa0a-1447c8510c9e","Type":"ContainerDied","Data":"49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003"}
Sep 16 19:39:41.274306 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.273966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk" event={"ID":"422dc654-a772-4802-aa0a-1447c8510c9e","Type":"ContainerDied","Data":"16c1a63933d3ea2e84308f2fceae8c712bc8c8e47585aa138c45b1474afd48c4"}
Sep 16 19:39:41.274306 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.273989    2583 scope.go:117] "RemoveContainer" containerID="49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003"
Sep 16 19:39:41.283095 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.283074    2583 scope.go:117] "RemoveContainer" containerID="49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003"
Sep 16 19:39:41.283355 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:39:41.283336    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003\": container with ID starting with 49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003 not found: ID does not exist" containerID="49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003"
Sep 16 19:39:41.283397 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.283366    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003"} err="failed to get container status \"49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003\": rpc error: code = NotFound desc = could not find container \"49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003\": container with ID starting with 49c30879be3fe406ef286996d3f9c62bdaa36936ccbc96d5fce82cd3c10ff003 not found: ID does not exist"
Sep 16 19:39:41.297335 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.297315    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"]
Sep 16 19:39:41.302489 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.302469    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-8jcgk"]
Sep 16 19:39:41.412605 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:39:41.412575    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="422dc654-a772-4802-aa0a-1447c8510c9e" path="/var/lib/kubelet/pods/422dc654-a772-4802-aa0a-1447c8510c9e/volumes"
Sep 16 19:40:02.867651 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.867610    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"]
Sep 16 19:40:02.868038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.867983    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" containerName="main"
Sep 16 19:40:02.868038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.867995    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" containerName="main"
Sep 16 19:40:02.868038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.868008    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133da96d-a135-4e88-b845-19e76d205612" containerName="main"
Sep 16 19:40:02.868038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.868013    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="133da96d-a135-4e88-b845-19e76d205612" containerName="main"
Sep 16 19:40:02.868038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.868034    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133da96d-a135-4e88-b845-19e76d205612" containerName="storage-initializer"
Sep 16 19:40:02.868038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.868040    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="133da96d-a135-4e88-b845-19e76d205612" containerName="storage-initializer"
Sep 16 19:40:02.868299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.868047    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422dc654-a772-4802-aa0a-1447c8510c9e" containerName="main"
Sep 16 19:40:02.868299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.868053    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="422dc654-a772-4802-aa0a-1447c8510c9e" containerName="main"
Sep 16 19:40:02.868299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.868110    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="133da96d-a135-4e88-b845-19e76d205612" containerName="main"
Sep 16 19:40:02.868299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.868122    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0404b27-1d2b-41a7-b9f9-9e3ba8102b6a" containerName="main"
Sep 16 19:40:02.868299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.868129    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="422dc654-a772-4802-aa0a-1447c8510c9e" containerName="main"
Sep 16 19:40:02.870402 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.870380    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:02.873468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.873432    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 19:40:02.874679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.874652    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-bjzjg\""
Sep 16 19:40:02.874812 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.874728    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 19:40:02.874812 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.874734    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 19:40:02.884113 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.884082    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"]
Sep 16 19:40:02.979578 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.979507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-dshm\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:02.979578 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.979547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-model-cache\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:02.979774 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.979632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hpls\" (UniqueName: \"kubernetes.io/projected/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-kube-api-access-8hpls\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:02.979774 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.979674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-home\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:02.979774 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.979697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:02.979905 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:02.979771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.080488 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.080462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-dshm\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.080628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.080500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-model-cache\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.080628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.080546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hpls\" (UniqueName: \"kubernetes.io/projected/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-kube-api-access-8hpls\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.080628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.080566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-home\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.080628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.080582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.080628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.080613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.081035 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.081000    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.081166 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.081037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-model-cache\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.081166 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.081068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-home\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.083174 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.083148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-dshm\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.083489 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.083461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.090961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.090937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hpls\" (UniqueName: \"kubernetes.io/projected/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-kube-api-access-8hpls\") pod \"decom-v1-kserve-6cb797fb75-6k62l\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.182532 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.182508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:03.473099 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:03.473063    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"]
Sep 16 19:40:03.475104 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:40:03.475074    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91e1951_1e7c_4006_adc1_eaa3ebb56e78.slice/crio-14e20c310ba69148ea3419f7fe4993ed9e8faeced6e5a7d7cec341b470b8232f WatchSource:0}: Error finding container 14e20c310ba69148ea3419f7fe4993ed9e8faeced6e5a7d7cec341b470b8232f: Status 404 returned error can't find the container with id 14e20c310ba69148ea3419f7fe4993ed9e8faeced6e5a7d7cec341b470b8232f
Sep 16 19:40:04.387887 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:04.387849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l" event={"ID":"b91e1951-1e7c-4006-adc1-eaa3ebb56e78","Type":"ContainerStarted","Data":"69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc"}
Sep 16 19:40:04.388460 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:04.387914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l" event={"ID":"b91e1951-1e7c-4006-adc1-eaa3ebb56e78","Type":"ContainerStarted","Data":"14e20c310ba69148ea3419f7fe4993ed9e8faeced6e5a7d7cec341b470b8232f"}
Sep 16 19:40:04.407934 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:04.407893    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l" podStartSLOduration=2.407878456 podStartE2EDuration="2.407878456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:04.405994529 +0000 UTC m=+2081.531798917" watchObservedRunningTime="2026-09-16 19:40:04.407878456 +0000 UTC m=+2081.533682843"
Sep 16 19:40:05.183586 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:05.183539    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:13.183457 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:13.183398    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:13.196233 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:13.196210    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:13.443643 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:13.443567    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:40:26.853643 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:26.853605    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:40:26.860277 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:26.860239    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:40:26.861070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:26.861051    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:40:26.867075 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:26.867056    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:40:31.049724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.049684    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"]
Sep 16 19:40:31.052595 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.052574    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.055687 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.055661    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 16 19:40:31.063609 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.063585    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"]
Sep 16 19:40:31.207482 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.207439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-home\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.207620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.207501    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tls-certs\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.207620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.207543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-dshm\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.207620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.207568    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrxm\" (UniqueName: \"kubernetes.io/projected/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-kube-api-access-rtrxm\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.207784 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.207653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tmp-dir\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.207784 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.207685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-model-cache\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.308438 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.308355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-dshm\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.308438 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.308385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrxm\" (UniqueName: \"kubernetes.io/projected/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-kube-api-access-rtrxm\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.308621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.308463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tmp-dir\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.308621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.308485    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-model-cache\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.308621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.308511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-home\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.308621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.308549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tls-certs\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.308895 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.308870    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tmp-dir\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.308961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.308905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-model-cache\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.309018 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.308961    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-home\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.310933 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.310910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-dshm\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.311099 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.311080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tls-certs\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.317649 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.317627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrxm\" (UniqueName: \"kubernetes.io/projected/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-kube-api-access-rtrxm\") pod \"decom-v2-kserve-77f867d787-hflt7\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.363807 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.363786    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:31.680043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:31.680011    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"]
Sep 16 19:40:31.694565 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:40:31.694533    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec3be6d_6cdd_4ebe_a26a_cfd191097ad4.slice/crio-e5721e9d6e5030d573bc8a0e7cd2068ea7f18cc3de1036c1c04ff6531ed91227 WatchSource:0}: Error finding container e5721e9d6e5030d573bc8a0e7cd2068ea7f18cc3de1036c1c04ff6531ed91227: Status 404 returned error can't find the container with id e5721e9d6e5030d573bc8a0e7cd2068ea7f18cc3de1036c1c04ff6531ed91227
Sep 16 19:40:32.521363 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:32.521329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7" event={"ID":"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4","Type":"ContainerStarted","Data":"a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a"}
Sep 16 19:40:32.521847 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:32.521382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7" event={"ID":"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4","Type":"ContainerStarted","Data":"e5721e9d6e5030d573bc8a0e7cd2068ea7f18cc3de1036c1c04ff6531ed91227"}
Sep 16 19:40:32.548652 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:32.548598    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7" podStartSLOduration=1.548584008 podStartE2EDuration="1.548584008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:32.545775051 +0000 UTC m=+2109.671579439" watchObservedRunningTime="2026-09-16 19:40:32.548584008 +0000 UTC m=+2109.674388396"
Sep 16 19:40:33.364826 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:33.364781    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:39.561890 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.561849    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"]
Sep 16 19:40:39.565660 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.565643    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.569190 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.569165    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:40:39.569330 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.569250    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-p5tq5\""
Sep 16 19:40:39.569330 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.569296    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:40:39.570302 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.570271    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:40:39.570389 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.570302    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:40:39.571033 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.571001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:40:39.572093 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.572058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-dshm\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.572093 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.572092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.572284 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.572147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.572284 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.572251    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-model-cache\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.572284 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.572278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-home\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.572482 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.572294    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.572482 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.572321    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.572482 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.572352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rrb\" (UniqueName: \"kubernetes.io/projected/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kube-api-access-f5rrb\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.586521 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.586496    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"]
Sep 16 19:40:39.673040 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-home\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673040 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673038    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673202 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673202 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rrb\" (UniqueName: \"kubernetes.io/projected/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kube-api-access-f5rrb\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673327 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-dshm\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673385 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673458 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673521 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-home\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673579 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-model-cache\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673778 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.673778 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673774    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.674083 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.674083 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.673886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-model-cache\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.675491 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.675469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-dshm\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.675639 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.675626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.684327 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.684295    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rrb\" (UniqueName: \"kubernetes.io/projected/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kube-api-access-f5rrb\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.877834 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.877805    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:39.935589 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.935222    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"]
Sep 16 19:40:39.940379 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.940289    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:40:39.943494 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.943471    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-nsfjk\""
Sep 16 19:40:39.954174 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.954149    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"]
Sep 16 19:40:39.976345 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.976308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk\" (UID: \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:40:39.976500 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:39.976473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgzdx\" (UniqueName: \"kubernetes.io/projected/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-kube-api-access-zgzdx\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk\" (UID: \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:40:40.077730 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.077706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgzdx\" (UniqueName: \"kubernetes.io/projected/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-kube-api-access-zgzdx\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk\" (UID: \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:40:40.077835 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.077744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk\" (UID: \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:40:40.080269 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.080251    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk\" (UID: \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:40:40.088787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.088764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgzdx\" (UniqueName: \"kubernetes.io/projected/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-kube-api-access-zgzdx\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk\" (UID: \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:40:40.167868 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.167843    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"]
Sep 16 19:40:40.169522 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:40:40.169497    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f14f92a_9fad_4b3a_a082_d6aef5fe5a66.slice/crio-add3fed3a0fc5ef21b6162dda187b4fb76347e816884a750c264e6fb2e0d30cf WatchSource:0}: Error finding container add3fed3a0fc5ef21b6162dda187b4fb76347e816884a750c264e6fb2e0d30cf: Status 404 returned error can't find the container with id add3fed3a0fc5ef21b6162dda187b4fb76347e816884a750c264e6fb2e0d30cf
Sep 16 19:40:40.255701 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.255675    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:40:40.531641 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.531551    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"]
Sep 16 19:40:40.534631 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:40:40.534598    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a10940e_7e2c_4117_b997_e3d9e9eb3e50.slice/crio-ffbd7c25366cb944dc5090273cb8d9e2bd8bdaf7138efe55ea105be86f56a5c3 WatchSource:0}: Error finding container ffbd7c25366cb944dc5090273cb8d9e2bd8bdaf7138efe55ea105be86f56a5c3: Status 404 returned error can't find the container with id ffbd7c25366cb944dc5090273cb8d9e2bd8bdaf7138efe55ea105be86f56a5c3
Sep 16 19:40:40.564124 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.564101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" event={"ID":"1a10940e-7e2c-4117-b997-e3d9e9eb3e50","Type":"ContainerStarted","Data":"ffbd7c25366cb944dc5090273cb8d9e2bd8bdaf7138efe55ea105be86f56a5c3"}
Sep 16 19:40:40.568460 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.566214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" event={"ID":"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66","Type":"ContainerStarted","Data":"8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7"}
Sep 16 19:40:40.568460 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:40.566327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" event={"ID":"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66","Type":"ContainerStarted","Data":"add3fed3a0fc5ef21b6162dda187b4fb76347e816884a750c264e6fb2e0d30cf"}
Sep 16 19:40:41.364566 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:41.364526    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:41.377561 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:41.377529    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:41.572523 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:41.572476    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" event={"ID":"1a10940e-7e2c-4117-b997-e3d9e9eb3e50","Type":"ContainerStarted","Data":"171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50"}
Sep 16 19:40:41.572988 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:41.572688    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:40:41.584469 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:41.584446    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:40:41.593678 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:41.593638    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" podStartSLOduration=2.593624271 podStartE2EDuration="2.593624271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:41.592587136 +0000 UTC m=+2118.718391520" watchObservedRunningTime="2026-09-16 19:40:41.593624271 +0000 UTC m=+2118.719428658"
Sep 16 19:40:42.580452 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:42.580405    2583 generic.go:356] "Generic (PLEG): container finished" podID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" containerID="8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7" exitCode=0
Sep 16 19:40:42.580822 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:42.580450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" event={"ID":"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66","Type":"ContainerDied","Data":"8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7"}
Sep 16 19:40:43.587599 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:43.587569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" event={"ID":"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66","Type":"ContainerStarted","Data":"9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad"}
Sep 16 19:40:43.620116 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:43.620057    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" podStartSLOduration=2.317455588 podStartE2EDuration="4.620037823s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.302582235s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:43.616614942 +0000 UTC m=+2120.742419329" watchObservedRunningTime="2026-09-16 19:40:43.620037823 +0000 UTC m=+2120.745842213"
Sep 16 19:40:43.878888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:43.878855    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:49.878935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:49.878892    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:49.891989 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:49.891959    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:40:50.632545 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:40:50.632516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:41:12.581960 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:12.581926    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:41:18.624765 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:18.624669    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"]
Sep 16 19:41:18.631552 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:18.631524    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"]
Sep 16 19:41:18.635601 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:41:18.635570    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:41:18.635601 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:41:18.635601 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:41:18.635601 ip-10-0-139-148 kubenswrapper[2583]:  > containerID="9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad" cmd=["/bin/sleep","15"]
Sep 16 19:41:18.635875 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:18.635625    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" message="" err=<
Sep 16 19:41:18.635875 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:41:18.635875 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:41:18.635875 ip-10-0-139-148 kubenswrapper[2583]:  >
Sep 16 19:41:18.635875 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:41:18.635641    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:41:18.635875 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:41:18.635875 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:41:18.635875 ip-10-0-139-148 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" podUID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" containerName="main" containerID="cri-o://9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad"
Sep 16 19:41:18.635875 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:18.635683    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" podUID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" containerName="main" containerID="cri-o://9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad" gracePeriod=60
Sep 16 19:41:19.384333 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.384306    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:41:19.453281 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.453253    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"]
Sep 16 19:41:19.498079 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498054    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tls-certs\") pod \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") "
Sep 16 19:41:19.498253 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498103    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-cabundle-cert\") pod \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") "
Sep 16 19:41:19.498253 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tmp-dir\") pod \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") "
Sep 16 19:41:19.498253 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498151    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-dshm\") pod \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") "
Sep 16 19:41:19.498253 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498172    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-home\") pod \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") "
Sep 16 19:41:19.498253 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kserve-provision-location\") pod \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") "
Sep 16 19:41:19.498600 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498278    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-model-cache\") pod \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") "
Sep 16 19:41:19.498600 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498333    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5rrb\" (UniqueName: \"kubernetes.io/projected/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kube-api-access-f5rrb\") pod \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\" (UID: \"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66\") "
Sep 16 19:41:19.498600 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498500    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-home" (OuterVolumeSpecName: "home") pod "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" (UID: "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:19.498600 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498488    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" (UID: "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:19.498600 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498565    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" (UID: "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:19.498863 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498597    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" (UID: "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:41:19.498863 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498851    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:19.499008 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498871    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:19.499008 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498885    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:19.499008 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.498898    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:19.500574 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.500532    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-dshm" (OuterVolumeSpecName: "dshm") pod "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" (UID: "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:19.500674 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.500600    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" (UID: "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:41:19.500836 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.500817    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kube-api-access-f5rrb" (OuterVolumeSpecName: "kube-api-access-f5rrb") pod "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" (UID: "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66"). InnerVolumeSpecName "kube-api-access-f5rrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:41:19.520078 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.520048    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" (UID: "7f14f92a-9fad-4b3a-a082-d6aef5fe5a66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:19.599676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.599651    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:19.599676 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.599674    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:19.599817 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.599684    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5rrb\" (UniqueName: \"kubernetes.io/projected/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-kube-api-access-f5rrb\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:19.599817 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.599693    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:19.758173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.758106    2583 generic.go:356] "Generic (PLEG): container finished" podID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" containerID="9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad" exitCode=0
Sep 16 19:41:19.758173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.758157    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"
Sep 16 19:41:19.758609 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.758177    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" event={"ID":"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66","Type":"ContainerDied","Data":"9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad"}
Sep 16 19:41:19.758609 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.758248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp" event={"ID":"7f14f92a-9fad-4b3a-a082-d6aef5fe5a66","Type":"ContainerDied","Data":"add3fed3a0fc5ef21b6162dda187b4fb76347e816884a750c264e6fb2e0d30cf"}
Sep 16 19:41:19.758609 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.758272    2583 scope.go:117] "RemoveContainer" containerID="9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad"
Sep 16 19:41:19.767351 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.767333    2583 scope.go:117] "RemoveContainer" containerID="8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7"
Sep 16 19:41:19.774721 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.774704    2583 scope.go:117] "RemoveContainer" containerID="9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad"
Sep 16 19:41:19.774963 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:41:19.774945    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad\": container with ID starting with 9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad not found: ID does not exist" containerID="9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad"
Sep 16 19:41:19.775039 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.774974    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad"} err="failed to get container status \"9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad\": rpc error: code = NotFound desc = could not find container \"9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad\": container with ID starting with 9c9a9b1c51bb9c94bbc91853e0649c9e97eb01ed6460ec275004479a20b40cad not found: ID does not exist"
Sep 16 19:41:19.775039 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.774998    2583 scope.go:117] "RemoveContainer" containerID="8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7"
Sep 16 19:41:19.775250 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:41:19.775228    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7\": container with ID starting with 8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7 not found: ID does not exist" containerID="8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7"
Sep 16 19:41:19.775292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.775261    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7"} err="failed to get container status \"8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7\": rpc error: code = NotFound desc = could not find container \"8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7\": container with ID starting with 8f4944c8437f47027b090ff6a5383a3d7ba7b99cbdc7811ff4f9b25ba63f28e7 not found: ID does not exist"
Sep 16 19:41:19.785426 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.785389    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"]
Sep 16 19:41:19.792140 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:19.792120    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-pcgqp"]
Sep 16 19:41:21.411589 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:21.411552    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" path="/var/lib/kubelet/pods/7f14f92a-9fad-4b3a-a082-d6aef5fe5a66/volumes"
Sep 16 19:41:22.582119 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:22.582076    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:32.582873 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:32.582828    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:33.650865 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:33.650822    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerName="main" containerID="cri-o://171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50" gracePeriod=45
Sep 16 19:41:34.472364 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.472328    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7" podUID="6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" containerName="main" containerID="cri-o://a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a" gracePeriod=45
Sep 16 19:41:34.574138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.574102    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7" podUID="6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 16 19:41:34.725374 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.725319    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:41:34.845265 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.845236    2583 generic.go:356] "Generic (PLEG): container finished" podID="6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" containerID="a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a" exitCode=0
Sep 16 19:41:34.845402 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.845271    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"
Sep 16 19:41:34.845402 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.845320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7" event={"ID":"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4","Type":"ContainerDied","Data":"a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a"}
Sep 16 19:41:34.845402 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.845373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7" event={"ID":"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4","Type":"ContainerDied","Data":"e5721e9d6e5030d573bc8a0e7cd2068ea7f18cc3de1036c1c04ff6531ed91227"}
Sep 16 19:41:34.845402 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.845396    2583 scope.go:117] "RemoveContainer" containerID="a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a"
Sep 16 19:41:34.855993 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.855973    2583 scope.go:117] "RemoveContainer" containerID="a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a"
Sep 16 19:41:34.856266 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:41:34.856247    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a\": container with ID starting with a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a not found: ID does not exist" containerID="a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a"
Sep 16 19:41:34.856320 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.856273    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a"} err="failed to get container status \"a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a\": rpc error: code = NotFound desc = could not find container \"a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a\": container with ID starting with a5f3c068a29fd505b6092bda989b7f8ccf1ac59c1c9fa0356b65ccd0b6a6e11a not found: ID does not exist"
Sep 16 19:41:34.917660 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.917634    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tmp-dir\") pod \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") "
Sep 16 19:41:34.917786 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.917662    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tls-certs\") pod \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") "
Sep 16 19:41:34.917786 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.917686    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-home\") pod \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") "
Sep 16 19:41:34.917786 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.917705    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-dshm\") pod \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") "
Sep 16 19:41:34.917786 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.917760    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtrxm\" (UniqueName: \"kubernetes.io/projected/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-kube-api-access-rtrxm\") pod \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") "
Sep 16 19:41:34.918010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.917798    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-model-cache\") pod \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\" (UID: \"6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4\") "
Sep 16 19:41:34.918010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.917925    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-home" (OuterVolumeSpecName: "home") pod "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" (UID: "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:34.918010 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.917940    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" (UID: "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:34.918151 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.918110    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" (UID: "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:34.918151 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.918130    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:34.918151 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.918149    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:34.920135 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.920114    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-dshm" (OuterVolumeSpecName: "dshm") pod "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" (UID: "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:34.920135 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.920119    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" (UID: "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:41:34.920260 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:34.920112    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-kube-api-access-rtrxm" (OuterVolumeSpecName: "kube-api-access-rtrxm") pod "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" (UID: "6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4"). InnerVolumeSpecName "kube-api-access-rtrxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:41:35.025477 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:35.025386    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtrxm\" (UniqueName: \"kubernetes.io/projected/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-kube-api-access-rtrxm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:35.025590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:35.025568    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:35.025628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:35.025595    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:35.025628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:35.025607    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:35.169918 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:35.169894    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"]
Sep 16 19:41:35.174827 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:35.174799    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hflt7"]
Sep 16 19:41:35.411361 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:35.411325    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" path="/var/lib/kubelet/pods/6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4/volumes"
Sep 16 19:41:35.631920 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:35.631888    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"]
Sep 16 19:41:42.582162 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:42.582123    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:42.582568 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:42.582239    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:41:50.650550 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.650514    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l" podUID="b91e1951-1e7c-4006-adc1-eaa3ebb56e78" containerName="main" containerID="cri-o://69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc" gracePeriod=45
Sep 16 19:41:50.896614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.896585    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:41:50.915740 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.915679    2583 generic.go:356] "Generic (PLEG): container finished" podID="b91e1951-1e7c-4006-adc1-eaa3ebb56e78" containerID="69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc" exitCode=0
Sep 16 19:41:50.915740 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.915732    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"
Sep 16 19:41:50.915914 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.915762    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l" event={"ID":"b91e1951-1e7c-4006-adc1-eaa3ebb56e78","Type":"ContainerDied","Data":"69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc"}
Sep 16 19:41:50.915914 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.915816    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l" event={"ID":"b91e1951-1e7c-4006-adc1-eaa3ebb56e78","Type":"ContainerDied","Data":"14e20c310ba69148ea3419f7fe4993ed9e8faeced6e5a7d7cec341b470b8232f"}
Sep 16 19:41:50.915914 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.915837    2583 scope.go:117] "RemoveContainer" containerID="69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc"
Sep 16 19:41:50.924983 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.924649    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-model-cache\") pod \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") "
Sep 16 19:41:50.924983 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.924932    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-model-cache" (OuterVolumeSpecName: "model-cache") pod "b91e1951-1e7c-4006-adc1-eaa3ebb56e78" (UID: "b91e1951-1e7c-4006-adc1-eaa3ebb56e78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:50.925165 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.925049    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:50.926485 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.926468    2583 scope.go:117] "RemoveContainer" containerID="69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc"
Sep 16 19:41:50.926816 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:41:50.926796    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc\": container with ID starting with 69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc not found: ID does not exist" containerID="69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc"
Sep 16 19:41:50.926908 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:50.926822    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc"} err="failed to get container status \"69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc\": rpc error: code = NotFound desc = could not find container \"69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc\": container with ID starting with 69a88640ddeeed977df9ea1639d2b944d0c065bd25d0dc2011129a04aa74cffc not found: ID does not exist"
Sep 16 19:41:51.025316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.025287    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tls-certs\") pod \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") "
Sep 16 19:41:51.025467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.025330    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tmp-dir\") pod \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") "
Sep 16 19:41:51.025467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.025370    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-dshm\") pod \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") "
Sep 16 19:41:51.025467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.025438    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hpls\" (UniqueName: \"kubernetes.io/projected/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-kube-api-access-8hpls\") pod \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") "
Sep 16 19:41:51.025644 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.025504    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-home\") pod \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\" (UID: \"b91e1951-1e7c-4006-adc1-eaa3ebb56e78\") "
Sep 16 19:41:51.025644 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.025621    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b91e1951-1e7c-4006-adc1-eaa3ebb56e78" (UID: "b91e1951-1e7c-4006-adc1-eaa3ebb56e78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:51.025787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.025771    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:51.025847 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.025808    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-home" (OuterVolumeSpecName: "home") pod "b91e1951-1e7c-4006-adc1-eaa3ebb56e78" (UID: "b91e1951-1e7c-4006-adc1-eaa3ebb56e78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:51.027454 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.027431    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b91e1951-1e7c-4006-adc1-eaa3ebb56e78" (UID: "b91e1951-1e7c-4006-adc1-eaa3ebb56e78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:41:51.027575 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.027556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-kube-api-access-8hpls" (OuterVolumeSpecName: "kube-api-access-8hpls") pod "b91e1951-1e7c-4006-adc1-eaa3ebb56e78" (UID: "b91e1951-1e7c-4006-adc1-eaa3ebb56e78"). InnerVolumeSpecName "kube-api-access-8hpls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:41:51.027621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.027571    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-dshm" (OuterVolumeSpecName: "dshm") pod "b91e1951-1e7c-4006-adc1-eaa3ebb56e78" (UID: "b91e1951-1e7c-4006-adc1-eaa3ebb56e78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:51.126178 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.126156    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:51.126178 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.126175    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hpls\" (UniqueName: \"kubernetes.io/projected/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-kube-api-access-8hpls\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:51.126288 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.126185    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:51.126288 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.126194    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b91e1951-1e7c-4006-adc1-eaa3ebb56e78-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:41:51.255177 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.255153    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"]
Sep 16 19:41:51.263469 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.263448    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6k62l"]
Sep 16 19:41:51.411993 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:51.411964    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b91e1951-1e7c-4006-adc1-eaa3ebb56e78" path="/var/lib/kubelet/pods/b91e1951-1e7c-4006-adc1-eaa3ebb56e78/volumes"
Sep 16 19:41:52.582173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:41:52.582133    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:02.583038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:02.582983    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:03.898896 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.898871    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:42:03.908788 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.908723    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-tls-certs\") pod \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\" (UID: \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\") "
Sep 16 19:42:03.908911 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.908791    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgzdx\" (UniqueName: \"kubernetes.io/projected/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-kube-api-access-zgzdx\") pod \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\" (UID: \"1a10940e-7e2c-4117-b997-e3d9e9eb3e50\") "
Sep 16 19:42:03.911043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.911007    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-kube-api-access-zgzdx" (OuterVolumeSpecName: "kube-api-access-zgzdx") pod "1a10940e-7e2c-4117-b997-e3d9e9eb3e50" (UID: "1a10940e-7e2c-4117-b997-e3d9e9eb3e50"). InnerVolumeSpecName "kube-api-access-zgzdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:03.911043 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.911023    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a10940e-7e2c-4117-b997-e3d9e9eb3e50" (UID: "1a10940e-7e2c-4117-b997-e3d9e9eb3e50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:03.978238 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.978201    2583 generic.go:356] "Generic (PLEG): container finished" podID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerID="171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50" exitCode=0
Sep 16 19:42:03.978358 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.978252    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"
Sep 16 19:42:03.978358 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.978286    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" event={"ID":"1a10940e-7e2c-4117-b997-e3d9e9eb3e50","Type":"ContainerDied","Data":"171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50"}
Sep 16 19:42:03.978358 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.978338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk" event={"ID":"1a10940e-7e2c-4117-b997-e3d9e9eb3e50","Type":"ContainerDied","Data":"ffbd7c25366cb944dc5090273cb8d9e2bd8bdaf7138efe55ea105be86f56a5c3"}
Sep 16 19:42:03.978526 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.978360    2583 scope.go:117] "RemoveContainer" containerID="171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50"
Sep 16 19:42:03.986647 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.986628    2583 scope.go:117] "RemoveContainer" containerID="171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50"
Sep 16 19:42:03.986893 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:42:03.986875    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50\": container with ID starting with 171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50 not found: ID does not exist" containerID="171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50"
Sep 16 19:42:03.986951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:03.986901    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50"} err="failed to get container status \"171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50\": rpc error: code = NotFound desc = could not find container \"171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50\": container with ID starting with 171dde88e2dc8687c4836c7c0fad1b3586c75547b173d11171baa6f8dc8a9b50 not found: ID does not exist"
Sep 16 19:42:04.001431 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:04.001393    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"]
Sep 16 19:42:04.004062 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:04.004041    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-789djqjdk"]
Sep 16 19:42:04.009735 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:04.009713    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgzdx\" (UniqueName: \"kubernetes.io/projected/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-kube-api-access-zgzdx\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:42:04.009816 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:04.009739    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a10940e-7e2c-4117-b997-e3d9e9eb3e50-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:42:05.411633 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:05.411606    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" path="/var/lib/kubelet/pods/1a10940e-7e2c-4117-b997-e3d9e9eb3e50/volumes"
Sep 16 19:42:32.569896 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.569863    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"]
Sep 16 19:42:32.570400 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570346    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91e1951-1e7c-4006-adc1-eaa3ebb56e78" containerName="main"
Sep 16 19:42:32.570400 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570365    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91e1951-1e7c-4006-adc1-eaa3ebb56e78" containerName="main"
Sep 16 19:42:32.570400 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570377    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" containerName="main"
Sep 16 19:42:32.570400 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570385    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" containerName="main"
Sep 16 19:42:32.570400 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570403    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" containerName="main"
Sep 16 19:42:32.570695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570430    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" containerName="main"
Sep 16 19:42:32.570695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570456    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" containerName="storage-initializer"
Sep 16 19:42:32.570695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570465    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" containerName="storage-initializer"
Sep 16 19:42:32.570695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570482    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerName="main"
Sep 16 19:42:32.570695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570491    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerName="main"
Sep 16 19:42:32.570695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570583    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f14f92a-9fad-4b3a-a082-d6aef5fe5a66" containerName="main"
Sep 16 19:42:32.570695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570598    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a10940e-7e2c-4117-b997-e3d9e9eb3e50" containerName="main"
Sep 16 19:42:32.570695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570608    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91e1951-1e7c-4006-adc1-eaa3ebb56e78" containerName="main"
Sep 16 19:42:32.570695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.570616    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec3be6d-6cdd-4ebe-a26a-cfd191097ad4" containerName="main"
Sep 16 19:42:32.573212 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.573193    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.576471 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.576448    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 19:42:32.578021 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.577993    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 16 19:42:32.578021 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.578004    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-tmsl4\""
Sep 16 19:42:32.578192 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.578059    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 19:42:32.591094 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.591066    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"]
Sep 16 19:42:32.642467 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.642445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-model-cache\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.642578 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.642488    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.642578 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.642522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-home\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.642672 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.642588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.642672 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.642637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-dshm\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.642672 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.642657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9kwz\" (UniqueName: \"kubernetes.io/projected/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-kube-api-access-c9kwz\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.743815 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.743789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.743910 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.743832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-dshm\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.743910 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.743849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kwz\" (UniqueName: \"kubernetes.io/projected/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-kube-api-access-c9kwz\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.744020 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.743962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-model-cache\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.744020 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.744002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.744123 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.744035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-home\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.744239 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.744220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.744299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.744276    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-model-cache\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.744362 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.744345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-home\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.746193 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.746165    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-dshm\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.746535 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.746520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.753639 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.753619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kwz\" (UniqueName: \"kubernetes.io/projected/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-kube-api-access-c9kwz\") pod \"leave-v2-kserve-556b4dddf9-s5bkq\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:32.883457 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:32.883435    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:33.172623 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:33.172589    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"]
Sep 16 19:42:33.180090 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:42:33.180043    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f718e17_4207_4b9a_ae4a_7ea95cb6c0fb.slice/crio-741bee443f3871b2debd50369c4a98c7c03f7123adf63514c6d0cc72798c3595 WatchSource:0}: Error finding container 741bee443f3871b2debd50369c4a98c7c03f7123adf63514c6d0cc72798c3595: Status 404 returned error can't find the container with id 741bee443f3871b2debd50369c4a98c7c03f7123adf63514c6d0cc72798c3595
Sep 16 19:42:34.117239 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:34.117204    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq" event={"ID":"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb","Type":"ContainerStarted","Data":"4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8"}
Sep 16 19:42:34.117652 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:34.117259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq" event={"ID":"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb","Type":"ContainerStarted","Data":"741bee443f3871b2debd50369c4a98c7c03f7123adf63514c6d0cc72798c3595"}
Sep 16 19:42:34.883991 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:34.883959    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:42.884109 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:42.884062    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:42.896690 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:42.896664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:42.920121 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:42.920068    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq" podStartSLOduration=10.920050783 podStartE2EDuration="10.920050783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:34.156660383 +0000 UTC m=+2231.282464772" watchObservedRunningTime="2026-09-16 19:42:42.920050783 +0000 UTC m=+2240.045855170"
Sep 16 19:42:43.175334 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:43.175259    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:42:53.459114 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.459036    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"]
Sep 16 19:42:53.462598 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.462581    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.467560 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.467535    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:42:53.467681 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.467633    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:42:53.468000 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.467978    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:42:53.468067 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.468024    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:42:53.468106 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.468083    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-czpmt\""
Sep 16 19:42:53.468641 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.468621    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:42:53.481919 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.481894    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"]
Sep 16 19:42:53.617197 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.617163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-home\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.617406 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.617200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.617406 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.617277    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-model-cache\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.617406 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.617295    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tls-certs\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.617406 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.617317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.617406 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.617390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.617619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.617437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-dshm\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.617619 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.617458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnzb\" (UniqueName: \"kubernetes.io/projected/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kube-api-access-7jnzb\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718377 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718377 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-dshm\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718377 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnzb\" (UniqueName: \"kubernetes.io/projected/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kube-api-access-7jnzb\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718663 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-home\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718663 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718663 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-model-cache\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718663 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tls-certs\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718663 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718925 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718773    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718925 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718830    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-home\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.718925 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.719098 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.718924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-model-cache\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.719278 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.719257    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.720714 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.720695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-dshm\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.721268 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.721238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tls-certs\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.738508 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.738475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnzb\" (UniqueName: \"kubernetes.io/projected/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kube-api-access-7jnzb\") pod \"tokenizer-template-args-test-kserve-84f74d478f-g8fdh\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.743983 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.743953    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"]
Sep 16 19:42:53.748596 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.748569    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:42:53.751791 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.751765    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-g7cc2\""
Sep 16 19:42:53.763383 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.763361    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"]
Sep 16 19:42:53.774560 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.774518    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:42:53.920628 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.920587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de5432-c01e-4873-93f8-1dc0fd371a52-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn\" (UID: \"23de5432-c01e-4873-93f8-1dc0fd371a52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:42:53.920811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:53.920706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2t6\" (UniqueName: \"kubernetes.io/projected/23de5432-c01e-4873-93f8-1dc0fd371a52-kube-api-access-zh2t6\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn\" (UID: \"23de5432-c01e-4873-93f8-1dc0fd371a52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:42:54.021429 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:54.021314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2t6\" (UniqueName: \"kubernetes.io/projected/23de5432-c01e-4873-93f8-1dc0fd371a52-kube-api-access-zh2t6\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn\" (UID: \"23de5432-c01e-4873-93f8-1dc0fd371a52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:42:54.021597 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:54.021444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de5432-c01e-4873-93f8-1dc0fd371a52-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn\" (UID: \"23de5432-c01e-4873-93f8-1dc0fd371a52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:42:54.024070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:54.024047    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de5432-c01e-4873-93f8-1dc0fd371a52-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn\" (UID: \"23de5432-c01e-4873-93f8-1dc0fd371a52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:42:54.031076 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:54.031047    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2t6\" (UniqueName: \"kubernetes.io/projected/23de5432-c01e-4873-93f8-1dc0fd371a52-kube-api-access-zh2t6\") pod \"tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn\" (UID: \"23de5432-c01e-4873-93f8-1dc0fd371a52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:42:54.059298 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:54.059266    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:42:54.066272 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:54.066244    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"]
Sep 16 19:42:54.069732 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:42:54.069709    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc93d0a_f0fe_495f_a0a2_218d41a3b1d2.slice/crio-436668cd0b1d6bc9c60a0a6268b4f756b6adc0ce38067abd62a0afa7f278ed17 WatchSource:0}: Error finding container 436668cd0b1d6bc9c60a0a6268b4f756b6adc0ce38067abd62a0afa7f278ed17: Status 404 returned error can't find the container with id 436668cd0b1d6bc9c60a0a6268b4f756b6adc0ce38067abd62a0afa7f278ed17
Sep 16 19:42:54.217136 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:54.217106    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" event={"ID":"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2","Type":"ContainerStarted","Data":"083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d"}
Sep 16 19:42:54.217275 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:54.217160    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" event={"ID":"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2","Type":"ContainerStarted","Data":"436668cd0b1d6bc9c60a0a6268b4f756b6adc0ce38067abd62a0afa7f278ed17"}
Sep 16 19:42:54.363497 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:54.363355    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"]
Sep 16 19:42:54.367053 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:42:54.366988    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23de5432_c01e_4873_93f8_1dc0fd371a52.slice/crio-44fda2811bb7a318be36162c8493ec2ed5f665268f14a4ea2f4d488e9d9af920 WatchSource:0}: Error finding container 44fda2811bb7a318be36162c8493ec2ed5f665268f14a4ea2f4d488e9d9af920: Status 404 returned error can't find the container with id 44fda2811bb7a318be36162c8493ec2ed5f665268f14a4ea2f4d488e9d9af920
Sep 16 19:42:55.225081 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:55.225048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" event={"ID":"23de5432-c01e-4873-93f8-1dc0fd371a52","Type":"ContainerStarted","Data":"2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39"}
Sep 16 19:42:55.225506 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:55.225111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" event={"ID":"23de5432-c01e-4873-93f8-1dc0fd371a52","Type":"ContainerStarted","Data":"44fda2811bb7a318be36162c8493ec2ed5f665268f14a4ea2f4d488e9d9af920"}
Sep 16 19:42:55.225506 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:55.225404    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:42:55.248134 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:55.248088    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" podStartSLOduration=2.248072849 podStartE2EDuration="2.248072849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:55.245860373 +0000 UTC m=+2252.371664761" watchObservedRunningTime="2026-09-16 19:42:55.248072849 +0000 UTC m=+2252.373877236"
Sep 16 19:42:57.237051 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:57.237016    2583 generic.go:356] "Generic (PLEG): container finished" podID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" containerID="083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d" exitCode=0
Sep 16 19:42:57.237449 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:57.237071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" event={"ID":"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2","Type":"ContainerDied","Data":"083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d"}
Sep 16 19:42:58.244473 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:58.244440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" event={"ID":"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2","Type":"ContainerStarted","Data":"ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b"}
Sep 16 19:42:58.774991 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:42:58.774964    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:43:03.775091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:03.775058    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:43:03.787418 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:03.787389    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:43:03.817979 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:03.817930    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" podStartSLOduration=8.527155065 podStartE2EDuration="10.817915671s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.290760606s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:58.277983659 +0000 UTC m=+2255.403788046" watchObservedRunningTime="2026-09-16 19:43:03.817915671 +0000 UTC m=+2260.943720059"
Sep 16 19:43:04.285338 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:04.285308    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:43:12.470131 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:12.470082    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"]
Sep 16 19:43:26.232321 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:26.232288    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:43:27.489461 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.489405    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq" podUID="5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" containerName="main" containerID="cri-o://4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8" gracePeriod=45
Sep 16 19:43:27.736439 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.736394    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:43:27.823634 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.823573    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-home\") pod \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") "
Sep 16 19:43:27.823634 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.823610    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-model-cache\") pod \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") "
Sep 16 19:43:27.823634 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.823626    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tmp-dir\") pod \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") "
Sep 16 19:43:27.823849 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.823660    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9kwz\" (UniqueName: \"kubernetes.io/projected/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-kube-api-access-c9kwz\") pod \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") "
Sep 16 19:43:27.823849 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.823749    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tls-certs\") pod \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") "
Sep 16 19:43:27.823849 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.823778    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-dshm\") pod \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\" (UID: \"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb\") "
Sep 16 19:43:27.824015 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.823860    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" (UID: "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:27.824015 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.823875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" (UID: "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:27.824015 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.823893    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-home" (OuterVolumeSpecName: "home") pod "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" (UID: "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:27.824138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.824058    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:43:27.824138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.824075    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:43:27.824138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.824088    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:43:27.826109 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.826073    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-kube-api-access-c9kwz" (OuterVolumeSpecName: "kube-api-access-c9kwz") pod "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" (UID: "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb"). InnerVolumeSpecName "kube-api-access-c9kwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:27.826222 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.826111    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-dshm" (OuterVolumeSpecName: "dshm") pod "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" (UID: "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:27.826222 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.826150    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" (UID: "5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:27.924573 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.924546    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:43:27.924573 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.924570    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:43:27.924742 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:27.924580    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9kwz\" (UniqueName: \"kubernetes.io/projected/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb-kube-api-access-c9kwz\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:43:28.386984 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:28.386950    2583 generic.go:356] "Generic (PLEG): container finished" podID="5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" containerID="4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8" exitCode=0
Sep 16 19:43:28.387136 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:28.387000    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"
Sep 16 19:43:28.387136 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:28.387023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq" event={"ID":"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb","Type":"ContainerDied","Data":"4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8"}
Sep 16 19:43:28.387136 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:28.387092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq" event={"ID":"5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb","Type":"ContainerDied","Data":"741bee443f3871b2debd50369c4a98c7c03f7123adf63514c6d0cc72798c3595"}
Sep 16 19:43:28.387136 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:28.387117    2583 scope.go:117] "RemoveContainer" containerID="4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8"
Sep 16 19:43:28.396923 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:28.396907    2583 scope.go:117] "RemoveContainer" containerID="4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8"
Sep 16 19:43:28.397201 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:43:28.397179    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8\": container with ID starting with 4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8 not found: ID does not exist" containerID="4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8"
Sep 16 19:43:28.397270 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:28.397213    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8"} err="failed to get container status \"4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8\": rpc error: code = NotFound desc = could not find container \"4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8\": container with ID starting with 4d7dbef0476dd99d2f5e96ad920a29072ef51ff14b1f0781687785ed7225c2e8 not found: ID does not exist"
Sep 16 19:43:28.411561 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:28.411535    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"]
Sep 16 19:43:28.418069 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:28.418046    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-s5bkq"]
Sep 16 19:43:29.412185 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:43:29.412151    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" path="/var/lib/kubelet/pods/5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb/volumes"
Sep 16 19:44:01.160653 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.160615    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"]
Sep 16 19:44:01.161159 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.161010    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" containerName="main"
Sep 16 19:44:01.161159 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.161022    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" containerName="main"
Sep 16 19:44:01.161159 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.161112    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f718e17-4207-4b9a-ae4a-7ea95cb6c0fb" containerName="main"
Sep 16 19:44:01.164116 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.164098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.167167 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.167144    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-xjmqm\""
Sep 16 19:44:01.168376 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.168351    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 19:44:01.168511 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.168381    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 19:44:01.168511 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.168350    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 16 19:44:01.177187 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.177168    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"]
Sep 16 19:44:01.191727 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.191705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-model-cache\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.191850 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.191739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d82113c-7636-4e12-8a64-e2843dea2f3f-tls-certs\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.191850 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.191809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.191937 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.191851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rvln\" (UniqueName: \"kubernetes.io/projected/3d82113c-7636-4e12-8a64-e2843dea2f3f-kube-api-access-6rvln\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.191937 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.191872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-dshm\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.191937 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.191900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-home\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.292833 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.292799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.293023 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.292844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rvln\" (UniqueName: \"kubernetes.io/projected/3d82113c-7636-4e12-8a64-e2843dea2f3f-kube-api-access-6rvln\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.293023 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.292866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-dshm\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.293023 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.292890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-home\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.293198 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.293059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-model-cache\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.293198 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.293112    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d82113c-7636-4e12-8a64-e2843dea2f3f-tls-certs\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.293198 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.293187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.293325 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.293233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-home\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.293427 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.293396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-model-cache\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.295350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.295327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-dshm\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.295668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.295650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d82113c-7636-4e12-8a64-e2843dea2f3f-tls-certs\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.301056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.301035    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rvln\" (UniqueName: \"kubernetes.io/projected/3d82113c-7636-4e12-8a64-e2843dea2f3f-kube-api-access-6rvln\") pod \"tri-v3-kserve-798c88c66c-bdn7v\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.474825 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.474742    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:01.753920 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:01.753843    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"]
Sep 16 19:44:01.758258 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:44:01.758228    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d82113c_7636_4e12_8a64_e2843dea2f3f.slice/crio-3e6e1753ba07c2fa540c27ddd0edfc8fafe3e58aa393936ebca1c25064e76fdb WatchSource:0}: Error finding container 3e6e1753ba07c2fa540c27ddd0edfc8fafe3e58aa393936ebca1c25064e76fdb: Status 404 returned error can't find the container with id 3e6e1753ba07c2fa540c27ddd0edfc8fafe3e58aa393936ebca1c25064e76fdb
Sep 16 19:44:02.552767 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:02.552729    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v" event={"ID":"3d82113c-7636-4e12-8a64-e2843dea2f3f","Type":"ContainerStarted","Data":"f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156"}
Sep 16 19:44:02.553142 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:02.552789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v" event={"ID":"3d82113c-7636-4e12-8a64-e2843dea2f3f","Type":"ContainerStarted","Data":"3e6e1753ba07c2fa540c27ddd0edfc8fafe3e58aa393936ebca1c25064e76fdb"}
Sep 16 19:44:02.574152 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:02.574103    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v" podStartSLOduration=1.574088854 podStartE2EDuration="1.574088854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:44:02.57230236 +0000 UTC m=+2319.698106751" watchObservedRunningTime="2026-09-16 19:44:02.574088854 +0000 UTC m=+2319.699893242"
Sep 16 19:44:03.475506 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:03.475472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:11.475162 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:11.475123    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:11.487885 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:11.487858    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:11.607246 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:11.607217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:44:39.288841 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.288804    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"]
Sep 16 19:44:39.304049 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.304017    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"]
Sep 16 19:44:39.315744 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:44:39.315699    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:44:39.315744 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:44:39.315744 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:44:39.315744 ip-10-0-139-148 kubenswrapper[2583]:  > containerID="ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b" cmd=["/bin/sleep","15"]
Sep 16 19:44:39.316077 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.315769    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" message="" err=<
Sep 16 19:44:39.316077 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:44:39.316077 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:44:39.316077 ip-10-0-139-148 kubenswrapper[2583]:  >
Sep 16 19:44:39.316077 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:44:39.315786    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:44:39.316077 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:44:39.316077 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:44:39.316077 ip-10-0-139-148 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" podUID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" containerName="main" containerID="cri-o://ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b"
Sep 16 19:44:39.316077 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.315846    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" podUID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" containerName="main" containerID="cri-o://ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b" gracePeriod=60
Sep 16 19:44:39.669439 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.669401    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:44:39.732518 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.732482    2583 generic.go:356] "Generic (PLEG): container finished" podID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" containerID="ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b" exitCode=0
Sep 16 19:44:39.732691 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.732542    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"
Sep 16 19:44:39.732691 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.732572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" event={"ID":"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2","Type":"ContainerDied","Data":"ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b"}
Sep 16 19:44:39.732691 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.732629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh" event={"ID":"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2","Type":"ContainerDied","Data":"436668cd0b1d6bc9c60a0a6268b4f756b6adc0ce38067abd62a0afa7f278ed17"}
Sep 16 19:44:39.732691 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.732658    2583 scope.go:117] "RemoveContainer" containerID="ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b"
Sep 16 19:44:39.741549 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.741526    2583 scope.go:117] "RemoveContainer" containerID="083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d"
Sep 16 19:44:39.749194 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.749176    2583 scope.go:117] "RemoveContainer" containerID="ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b"
Sep 16 19:44:39.749510 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:44:39.749491    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b\": container with ID starting with ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b not found: ID does not exist" containerID="ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b"
Sep 16 19:44:39.749573 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.749522    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b"} err="failed to get container status \"ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b\": rpc error: code = NotFound desc = could not find container \"ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b\": container with ID starting with ac3f48bbdbc87dbd0f80ab5358369528d6c048c1a5049bdc9ec794548d74ee8b not found: ID does not exist"
Sep 16 19:44:39.749573 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.749540    2583 scope.go:117] "RemoveContainer" containerID="083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d"
Sep 16 19:44:39.749775 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:44:39.749758    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d\": container with ID starting with 083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d not found: ID does not exist" containerID="083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d"
Sep 16 19:44:39.749832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.749786    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d"} err="failed to get container status \"083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d\": rpc error: code = NotFound desc = could not find container \"083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d\": container with ID starting with 083a4f591a14986d86fb70338bd63983491c5c47b9c972265f319c5bde854a2d not found: ID does not exist"
Sep 16 19:44:39.756079 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-cabundle-cert\") pod \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") "
Sep 16 19:44:39.756169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756097    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jnzb\" (UniqueName: \"kubernetes.io/projected/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kube-api-access-7jnzb\") pod \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") "
Sep 16 19:44:39.756169 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756127    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kserve-provision-location\") pod \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") "
Sep 16 19:44:39.756282 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756203    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tls-certs\") pod \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") "
Sep 16 19:44:39.756282 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-dshm\") pod \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") "
Sep 16 19:44:39.756384 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756301    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-home\") pod \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") "
Sep 16 19:44:39.756384 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756347    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tmp-dir\") pod \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") "
Sep 16 19:44:39.756513 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756385    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-model-cache\") pod \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\" (UID: \"8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2\") "
Sep 16 19:44:39.756566 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756386    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" (UID: "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:44:39.756741 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756706    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:44:39.756741 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756734    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" (UID: "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:39.757066 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756755    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" (UID: "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:39.757066 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.756830    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-home" (OuterVolumeSpecName: "home") pod "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" (UID: "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:39.758622 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.758591    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" (UID: "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:44:39.758790 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.758769    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kube-api-access-7jnzb" (OuterVolumeSpecName: "kube-api-access-7jnzb") pod "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" (UID: "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2"). InnerVolumeSpecName "kube-api-access-7jnzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:44:39.758854 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.758831    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-dshm" (OuterVolumeSpecName: "dshm") pod "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" (UID: "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:39.778620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.778592    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" (UID: "8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:39.858111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.858023    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:44:39.858111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.858052    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:44:39.858111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.858059    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:44:39.858111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.858067    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:44:39.858111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.858075    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:44:39.858111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.858083    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jnzb\" (UniqueName: \"kubernetes.io/projected/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kube-api-access-7jnzb\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:44:39.858111 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:39.858093    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:44:40.056973 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:40.056939    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"]
Sep 16 19:44:40.061976 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:40.061947    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-g8fdh"]
Sep 16 19:44:41.412021 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:41.411985    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" path="/var/lib/kubelet/pods/8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2/volumes"
Sep 16 19:44:46.232158 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:46.232120    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" podUID="23de5432-c01e-4873-93f8-1dc0fd371a52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:44:54.313132 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:54.313086    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" podUID="23de5432-c01e-4873-93f8-1dc0fd371a52" containerName="main" containerID="cri-o://2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39" gracePeriod=45
Sep 16 19:44:56.232001 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:44:56.231963    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" podUID="23de5432-c01e-4873-93f8-1dc0fd371a52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:06.232590 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:06.232548    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" podUID="23de5432-c01e-4873-93f8-1dc0fd371a52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:06.232999 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:06.232666    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:45:16.232555 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:16.232511    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" podUID="23de5432-c01e-4873-93f8-1dc0fd371a52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:24.564155 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.564130    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:45:24.737663 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.737627    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de5432-c01e-4873-93f8-1dc0fd371a52-tls-certs\") pod \"23de5432-c01e-4873-93f8-1dc0fd371a52\" (UID: \"23de5432-c01e-4873-93f8-1dc0fd371a52\") "
Sep 16 19:45:24.737859 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.737753    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh2t6\" (UniqueName: \"kubernetes.io/projected/23de5432-c01e-4873-93f8-1dc0fd371a52-kube-api-access-zh2t6\") pod \"23de5432-c01e-4873-93f8-1dc0fd371a52\" (UID: \"23de5432-c01e-4873-93f8-1dc0fd371a52\") "
Sep 16 19:45:24.739858 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.739831    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23de5432-c01e-4873-93f8-1dc0fd371a52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23de5432-c01e-4873-93f8-1dc0fd371a52" (UID: "23de5432-c01e-4873-93f8-1dc0fd371a52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:24.740026 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.740000    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23de5432-c01e-4873-93f8-1dc0fd371a52-kube-api-access-zh2t6" (OuterVolumeSpecName: "kube-api-access-zh2t6") pod "23de5432-c01e-4873-93f8-1dc0fd371a52" (UID: "23de5432-c01e-4873-93f8-1dc0fd371a52"). InnerVolumeSpecName "kube-api-access-zh2t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:24.838787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.838752    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh2t6\" (UniqueName: \"kubernetes.io/projected/23de5432-c01e-4873-93f8-1dc0fd371a52-kube-api-access-zh2t6\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:45:24.838787 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.838782    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de5432-c01e-4873-93f8-1dc0fd371a52-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:45:24.943229 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.943197    2583 generic.go:356] "Generic (PLEG): container finished" podID="23de5432-c01e-4873-93f8-1dc0fd371a52" containerID="2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39" exitCode=0
Sep 16 19:45:24.943399 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.943243    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"
Sep 16 19:45:24.943399 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.943288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" event={"ID":"23de5432-c01e-4873-93f8-1dc0fd371a52","Type":"ContainerDied","Data":"2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39"}
Sep 16 19:45:24.943399 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.943349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn" event={"ID":"23de5432-c01e-4873-93f8-1dc0fd371a52","Type":"ContainerDied","Data":"44fda2811bb7a318be36162c8493ec2ed5f665268f14a4ea2f4d488e9d9af920"}
Sep 16 19:45:24.943399 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.943377    2583 scope.go:117] "RemoveContainer" containerID="2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39"
Sep 16 19:45:24.952854 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.952839    2583 scope.go:117] "RemoveContainer" containerID="2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39"
Sep 16 19:45:24.953106 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:45:24.953091    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39\": container with ID starting with 2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39 not found: ID does not exist" containerID="2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39"
Sep 16 19:45:24.953154 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.953113    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39"} err="failed to get container status \"2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39\": rpc error: code = NotFound desc = could not find container \"2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39\": container with ID starting with 2bc14e0a269a63ec9525ce170c25f460e20abb898e76f99fe1e4964429b3ec39 not found: ID does not exist"
Sep 16 19:45:24.966302 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.966281    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"]
Sep 16 19:45:24.970485 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:24.970462    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-57fb9pt5jn"]
Sep 16 19:45:25.411947 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:25.411916    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23de5432-c01e-4873-93f8-1dc0fd371a52" path="/var/lib/kubelet/pods/23de5432-c01e-4873-93f8-1dc0fd371a52/volumes"
Sep 16 19:45:26.889453 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:26.889403    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:45:26.901883 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:26.901856    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:45:26.902550 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:26.902523    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:45:26.908283 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:26.908264    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:45:32.004185 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:32.004144    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"]
Sep 16 19:45:43.401294 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.401257    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"]
Sep 16 19:45:43.401888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.401606    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23de5432-c01e-4873-93f8-1dc0fd371a52" containerName="main"
Sep 16 19:45:43.401888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.401619    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="23de5432-c01e-4873-93f8-1dc0fd371a52" containerName="main"
Sep 16 19:45:43.401888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.401634    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" containerName="main"
Sep 16 19:45:43.401888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.401639    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" containerName="main"
Sep 16 19:45:43.401888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.401649    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" containerName="storage-initializer"
Sep 16 19:45:43.401888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.401654    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" containerName="storage-initializer"
Sep 16 19:45:43.401888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.401711    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc93d0a-f0fe-495f-a0a2-218d41a3b1d2" containerName="main"
Sep 16 19:45:43.401888 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.401724    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="23de5432-c01e-4873-93f8-1dc0fd371a52" containerName="main"
Sep 16 19:45:43.404789 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.404769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:45:43.408118 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.408097    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:45:43.409508 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.409486    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-9p7tz\""
Sep 16 19:45:43.409508 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.409499    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:45:43.409695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.409506    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:45:43.409695 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.409563    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-ch7d4\""
Sep 16 19:45:43.416215 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.416191    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"]
Sep 16 19:45:43.497374 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.497347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncm7d\" (UniqueName: \"kubernetes.io/projected/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-kube-api-access-ncm7d\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z\" (UID: \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:45:43.497509 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.497376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z\" (UID: \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:45:43.597856 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.597833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncm7d\" (UniqueName: \"kubernetes.io/projected/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-kube-api-access-ncm7d\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z\" (UID: \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:45:43.597975 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.597862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z\" (UID: \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:45:43.600512 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.600492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z\" (UID: \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:45:43.608886 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.608859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncm7d\" (UniqueName: \"kubernetes.io/projected/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-kube-api-access-ncm7d\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z\" (UID: \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:45:43.715330 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.715270    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:45:43.992438 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:43.992331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"]
Sep 16 19:45:43.995964 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:45:43.995931    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f9e75c_19e2_4b4e_bdae_fb43e1dad0f1.slice/crio-7fd1af1e2b61e9e8c63331af2b6e4b1ed2b710a8a49e4ac855277c37454930d0 WatchSource:0}: Error finding container 7fd1af1e2b61e9e8c63331af2b6e4b1ed2b710a8a49e4ac855277c37454930d0: Status 404 returned error can't find the container with id 7fd1af1e2b61e9e8c63331af2b6e4b1ed2b710a8a49e4ac855277c37454930d0
Sep 16 19:45:44.040461 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:44.040436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" event={"ID":"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1","Type":"ContainerStarted","Data":"7fd1af1e2b61e9e8c63331af2b6e4b1ed2b710a8a49e4ac855277c37454930d0"}
Sep 16 19:45:45.047397 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:45.047357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" event={"ID":"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1","Type":"ContainerStarted","Data":"4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa"}
Sep 16 19:45:45.047826 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:45.047455    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:45:45.090357 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:45.090309    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" podStartSLOduration=2.090293394 podStartE2EDuration="2.090293394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:45.084983322 +0000 UTC m=+2422.210787710" watchObservedRunningTime="2026-09-16 19:45:45.090293394 +0000 UTC m=+2422.216097886"
Sep 16 19:45:47.023808 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.023762    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v" podUID="3d82113c-7636-4e12-8a64-e2843dea2f3f" containerName="main" containerID="cri-o://f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156" gracePeriod=45
Sep 16 19:45:47.273549 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.273525    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:45:47.330015 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.329951    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-model-cache\") pod \"3d82113c-7636-4e12-8a64-e2843dea2f3f\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") "
Sep 16 19:45:47.330015 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330008    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-home\") pod \"3d82113c-7636-4e12-8a64-e2843dea2f3f\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") "
Sep 16 19:45:47.330234 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330032    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d82113c-7636-4e12-8a64-e2843dea2f3f-tls-certs\") pod \"3d82113c-7636-4e12-8a64-e2843dea2f3f\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") "
Sep 16 19:45:47.330234 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330083    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rvln\" (UniqueName: \"kubernetes.io/projected/3d82113c-7636-4e12-8a64-e2843dea2f3f-kube-api-access-6rvln\") pod \"3d82113c-7636-4e12-8a64-e2843dea2f3f\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") "
Sep 16 19:45:47.330234 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330102    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-dshm\") pod \"3d82113c-7636-4e12-8a64-e2843dea2f3f\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") "
Sep 16 19:45:47.330234 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330129    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-tmp-dir\") pod \"3d82113c-7636-4e12-8a64-e2843dea2f3f\" (UID: \"3d82113c-7636-4e12-8a64-e2843dea2f3f\") "
Sep 16 19:45:47.330464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330224    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d82113c-7636-4e12-8a64-e2843dea2f3f" (UID: "3d82113c-7636-4e12-8a64-e2843dea2f3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:47.330464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330254    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-home" (OuterVolumeSpecName: "home") pod "3d82113c-7636-4e12-8a64-e2843dea2f3f" (UID: "3d82113c-7636-4e12-8a64-e2843dea2f3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:47.330464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330356    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:45:47.330464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330395    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:45:47.330620 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.330465    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d82113c-7636-4e12-8a64-e2843dea2f3f" (UID: "3d82113c-7636-4e12-8a64-e2843dea2f3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:47.332454 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.332425    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d82113c-7636-4e12-8a64-e2843dea2f3f-kube-api-access-6rvln" (OuterVolumeSpecName: "kube-api-access-6rvln") pod "3d82113c-7636-4e12-8a64-e2843dea2f3f" (UID: "3d82113c-7636-4e12-8a64-e2843dea2f3f"). InnerVolumeSpecName "kube-api-access-6rvln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:47.332853 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.332832    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d82113c-7636-4e12-8a64-e2843dea2f3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d82113c-7636-4e12-8a64-e2843dea2f3f" (UID: "3d82113c-7636-4e12-8a64-e2843dea2f3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:47.332938 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.332863    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-dshm" (OuterVolumeSpecName: "dshm") pod "3d82113c-7636-4e12-8a64-e2843dea2f3f" (UID: "3d82113c-7636-4e12-8a64-e2843dea2f3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:47.431003 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.430983    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d82113c-7636-4e12-8a64-e2843dea2f3f-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:45:47.431091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.431005    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rvln\" (UniqueName: \"kubernetes.io/projected/3d82113c-7636-4e12-8a64-e2843dea2f3f-kube-api-access-6rvln\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:45:47.431091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.431016    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:45:47.431091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:47.431024    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d82113c-7636-4e12-8a64-e2843dea2f3f-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:45:48.062961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:48.062923    2583 generic.go:356] "Generic (PLEG): container finished" podID="3d82113c-7636-4e12-8a64-e2843dea2f3f" containerID="f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156" exitCode=0
Sep 16 19:45:48.063398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:48.062978    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"
Sep 16 19:45:48.063398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:48.062997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v" event={"ID":"3d82113c-7636-4e12-8a64-e2843dea2f3f","Type":"ContainerDied","Data":"f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156"}
Sep 16 19:45:48.063398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:48.063042    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v" event={"ID":"3d82113c-7636-4e12-8a64-e2843dea2f3f","Type":"ContainerDied","Data":"3e6e1753ba07c2fa540c27ddd0edfc8fafe3e58aa393936ebca1c25064e76fdb"}
Sep 16 19:45:48.063398 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:48.063065    2583 scope.go:117] "RemoveContainer" containerID="f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156"
Sep 16 19:45:48.074985 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:48.074955    2583 scope.go:117] "RemoveContainer" containerID="f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156"
Sep 16 19:45:48.075299 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:45:48.075273    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156\": container with ID starting with f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156 not found: ID does not exist" containerID="f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156"
Sep 16 19:45:48.075443 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:48.075306    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156"} err="failed to get container status \"f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156\": rpc error: code = NotFound desc = could not find container \"f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156\": container with ID starting with f34bb4d854e33c80e5b85db29dc295c038a25c230f30e345649c1fa55d261156 not found: ID does not exist"
Sep 16 19:45:48.086322 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:48.086296    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"]
Sep 16 19:45:48.094279 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:48.094254    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-bdn7v"]
Sep 16 19:45:49.411625 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:49.411593    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d82113c-7636-4e12-8a64-e2843dea2f3f" path="/var/lib/kubelet/pods/3d82113c-7636-4e12-8a64-e2843dea2f3f/volumes"
Sep 16 19:45:54.959087 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.959056    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"]
Sep 16 19:45:54.959474 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.959385    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d82113c-7636-4e12-8a64-e2843dea2f3f" containerName="main"
Sep 16 19:45:54.959474 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.959396    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d82113c-7636-4e12-8a64-e2843dea2f3f" containerName="main"
Sep 16 19:45:54.959565 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.959512    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d82113c-7636-4e12-8a64-e2843dea2f3f" containerName="main"
Sep 16 19:45:54.963704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.963688    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:54.966555 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.966529    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:45:54.966681 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.966625    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-2d9s2\""
Sep 16 19:45:54.967786 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.967765    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 19:45:54.967936 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.967792    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:45:54.975247 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.975224    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"]
Sep 16 19:45:54.985507 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.985485    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-dshm\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:54.985593 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.985534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tls-certs\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:54.985638 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.985604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-home\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:54.985638 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.985625    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:54.985705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.985651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-model-cache\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:54.985705 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:54.985678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlql\" (UniqueName: \"kubernetes.io/projected/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-kube-api-access-ndlql\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.086301 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.086276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-home\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.086465 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.086308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.086465 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.086330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-model-cache\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.086465 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.086354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlql\" (UniqueName: \"kubernetes.io/projected/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-kube-api-access-ndlql\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.086465 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.086391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-dshm\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.086686 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.086486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tls-certs\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.086747 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.086730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-home\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.086802 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.086746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-model-cache\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.086858 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.086798    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.088708 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.088691    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-dshm\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.089034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.089018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tls-certs\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.099700 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.099681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlql\" (UniqueName: \"kubernetes.io/projected/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-kube-api-access-ndlql\") pod \"late-v1-kserve-56cbc8456c-tttkk\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.274448 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.274362    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:45:55.556445 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:55.556319    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"]
Sep 16 19:45:55.559755 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:45:55.559727    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b48af2_8f9f_4fb4_99c2_e236f0eed71e.slice/crio-9657e1ca70c063e63a693e4138ff8d5a31de0986d1744c1a1f7f584ca37b8c86 WatchSource:0}: Error finding container 9657e1ca70c063e63a693e4138ff8d5a31de0986d1744c1a1f7f584ca37b8c86: Status 404 returned error can't find the container with id 9657e1ca70c063e63a693e4138ff8d5a31de0986d1744c1a1f7f584ca37b8c86
Sep 16 19:45:56.101704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:56.101668    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk" event={"ID":"37b48af2-8f9f-4fb4-99c2-e236f0eed71e","Type":"ContainerStarted","Data":"604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15"}
Sep 16 19:45:56.102110 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:56.101724    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk" event={"ID":"37b48af2-8f9f-4fb4-99c2-e236f0eed71e","Type":"ContainerStarted","Data":"9657e1ca70c063e63a693e4138ff8d5a31de0986d1744c1a1f7f584ca37b8c86"}
Sep 16 19:45:56.125206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:56.125156    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk" podStartSLOduration=2.125138497 podStartE2EDuration="2.125138497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:56.124585218 +0000 UTC m=+2433.250389597" watchObservedRunningTime="2026-09-16 19:45:56.125138497 +0000 UTC m=+2433.250942886"
Sep 16 19:45:56.275058 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:45:56.275019    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:46:05.275434 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:05.275381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:46:05.287964 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:05.287938    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:46:06.160791 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:06.160761    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:46:16.054288 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:16.054255    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:46:17.963698 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:17.963663    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"]
Sep 16 19:46:26.054254 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:26.054199    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" podUID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:32.983681 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:32.983633    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" podUID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" containerName="main" containerID="cri-o://4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa" gracePeriod=45
Sep 16 19:46:36.054143 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:36.054100    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" podUID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:46.053826 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:46.053778    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" podUID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:46.054206 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:46.053897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:46:56.053932 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:46:56.053889    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" podUID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:47:03.224818 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.224792    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:47:03.277811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.277744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncm7d\" (UniqueName: \"kubernetes.io/projected/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-kube-api-access-ncm7d\") pod \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\" (UID: \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\") "
Sep 16 19:47:03.277950 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.277836    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-tls-certs\") pod \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\" (UID: \"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1\") "
Sep 16 19:47:03.280054 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.280017    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" (UID: "74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:47:03.280054 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.280045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-kube-api-access-ncm7d" (OuterVolumeSpecName: "kube-api-access-ncm7d") pod "74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" (UID: "74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1"). InnerVolumeSpecName "kube-api-access-ncm7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:03.378875 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.378845    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncm7d\" (UniqueName: \"kubernetes.io/projected/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-kube-api-access-ncm7d\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:47:03.378875 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.378871    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:47:03.417466 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.417432    2583 generic.go:356] "Generic (PLEG): container finished" podID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" containerID="4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa" exitCode=0
Sep 16 19:47:03.417562 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.417470    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"
Sep 16 19:47:03.417562 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.417517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" event={"ID":"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1","Type":"ContainerDied","Data":"4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa"}
Sep 16 19:47:03.417658 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.417563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z" event={"ID":"74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1","Type":"ContainerDied","Data":"7fd1af1e2b61e9e8c63331af2b6e4b1ed2b710a8a49e4ac855277c37454930d0"}
Sep 16 19:47:03.417658 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.417589    2583 scope.go:117] "RemoveContainer" containerID="4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa"
Sep 16 19:47:03.426638 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.426620    2583 scope.go:117] "RemoveContainer" containerID="4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa"
Sep 16 19:47:03.426879 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:47:03.426853    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa\": container with ID starting with 4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa not found: ID does not exist" containerID="4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa"
Sep 16 19:47:03.426943 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.426880    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa"} err="failed to get container status \"4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa\": rpc error: code = NotFound desc = could not find container \"4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa\": container with ID starting with 4ace3c86f107ce3c5114aa82e071280c169affdae76b6bf0fa62753161cd2daa not found: ID does not exist"
Sep 16 19:47:03.448118 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.448097    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"]
Sep 16 19:47:03.458850 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:03.458825    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesz28z"]
Sep 16 19:47:05.411317 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:05.411279    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" path="/var/lib/kubelet/pods/74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1/volumes"
Sep 16 19:47:24.973679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.973644    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"]
Sep 16 19:47:24.974178 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.973991    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" containerName="main"
Sep 16 19:47:24.974178 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.974003    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" containerName="main"
Sep 16 19:47:24.974178 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.974063    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f9e75c-19e2-4b4e-bdae-fb43e1dad0f1" containerName="main"
Sep 16 19:47:24.978308 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.978290    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:24.986190 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.986167    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:47:24.986344 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.986175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:47:24.986344 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.986217    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-d9pzz\""
Sep 16 19:47:24.986344 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.986233    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-45pl8\""
Sep 16 19:47:24.986344 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.986239    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:47:24.992637 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:24.992615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"]
Sep 16 19:47:25.083616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:25.083583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a54ec-721b-4541-9dc2-bff591012c0f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd\" (UID: \"e47a54ec-721b-4541-9dc2-bff591012c0f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:25.083788 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:25.083635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkkr7\" (UniqueName: \"kubernetes.io/projected/e47a54ec-721b-4541-9dc2-bff591012c0f-kube-api-access-lkkr7\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd\" (UID: \"e47a54ec-721b-4541-9dc2-bff591012c0f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:25.185017 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:25.184982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a54ec-721b-4541-9dc2-bff591012c0f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd\" (UID: \"e47a54ec-721b-4541-9dc2-bff591012c0f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:25.185228 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:25.185027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkkr7\" (UniqueName: \"kubernetes.io/projected/e47a54ec-721b-4541-9dc2-bff591012c0f-kube-api-access-lkkr7\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd\" (UID: \"e47a54ec-721b-4541-9dc2-bff591012c0f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:25.187719 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:25.187697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a54ec-721b-4541-9dc2-bff591012c0f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd\" (UID: \"e47a54ec-721b-4541-9dc2-bff591012c0f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:25.194074 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:25.194045    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkkr7\" (UniqueName: \"kubernetes.io/projected/e47a54ec-721b-4541-9dc2-bff591012c0f-kube-api-access-lkkr7\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd\" (UID: \"e47a54ec-721b-4541-9dc2-bff591012c0f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:25.288460 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:25.288360    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:25.564542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:25.564518    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"]
Sep 16 19:47:25.566252 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:47:25.566222    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47a54ec_721b_4541_9dc2_bff591012c0f.slice/crio-7472bc8a6b1d7f92f2d41118b566d1b97e12f5966259c0052066c0fd6a9f11bd WatchSource:0}: Error finding container 7472bc8a6b1d7f92f2d41118b566d1b97e12f5966259c0052066c0fd6a9f11bd: Status 404 returned error can't find the container with id 7472bc8a6b1d7f92f2d41118b566d1b97e12f5966259c0052066c0fd6a9f11bd
Sep 16 19:47:26.527530 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:26.527490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" event={"ID":"e47a54ec-721b-4541-9dc2-bff591012c0f","Type":"ContainerStarted","Data":"4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5"}
Sep 16 19:47:26.528049 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:26.527549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" event={"ID":"e47a54ec-721b-4541-9dc2-bff591012c0f","Type":"ContainerStarted","Data":"7472bc8a6b1d7f92f2d41118b566d1b97e12f5966259c0052066c0fd6a9f11bd"}
Sep 16 19:47:26.528049 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:26.527652    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:26.551699 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:26.551660    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" podStartSLOduration=2.551646519 podStartE2EDuration="2.551646519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:26.54851205 +0000 UTC m=+2523.674316438" watchObservedRunningTime="2026-09-16 19:47:26.551646519 +0000 UTC m=+2523.677450908"
Sep 16 19:47:32.883004 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:32.882959    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"]
Sep 16 19:47:47.902726 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:47.902686    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk" podUID="37b48af2-8f9f-4fb4-99c2-e236f0eed71e" containerName="main" containerID="cri-o://604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15" gracePeriod=45
Sep 16 19:47:48.152553 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.152526    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:47:48.288370 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288292    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-home\") pod \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") "
Sep 16 19:47:48.288370 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288334    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tls-certs\") pod \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") "
Sep 16 19:47:48.288370 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288360    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-dshm\") pod \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") "
Sep 16 19:47:48.288671 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288496    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tmp-dir\") pod \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") "
Sep 16 19:47:48.288671 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288534    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndlql\" (UniqueName: \"kubernetes.io/projected/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-kube-api-access-ndlql\") pod \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") "
Sep 16 19:47:48.288671 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288560    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-model-cache\") pod \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\" (UID: \"37b48af2-8f9f-4fb4-99c2-e236f0eed71e\") "
Sep 16 19:47:48.288671 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288638    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-home" (OuterVolumeSpecName: "home") pod "37b48af2-8f9f-4fb4-99c2-e236f0eed71e" (UID: "37b48af2-8f9f-4fb4-99c2-e236f0eed71e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:48.288872 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288741    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37b48af2-8f9f-4fb4-99c2-e236f0eed71e" (UID: "37b48af2-8f9f-4fb4-99c2-e236f0eed71e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:48.288940 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288867    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-model-cache" (OuterVolumeSpecName: "model-cache") pod "37b48af2-8f9f-4fb4-99c2-e236f0eed71e" (UID: "37b48af2-8f9f-4fb4-99c2-e236f0eed71e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:48.288940 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288925    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:47:48.289017 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.288944    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:47:48.290830 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.290807    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-dshm" (OuterVolumeSpecName: "dshm") pod "37b48af2-8f9f-4fb4-99c2-e236f0eed71e" (UID: "37b48af2-8f9f-4fb4-99c2-e236f0eed71e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:48.290920 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.290854    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37b48af2-8f9f-4fb4-99c2-e236f0eed71e" (UID: "37b48af2-8f9f-4fb4-99c2-e236f0eed71e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:47:48.290965 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.290940    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-kube-api-access-ndlql" (OuterVolumeSpecName: "kube-api-access-ndlql") pod "37b48af2-8f9f-4fb4-99c2-e236f0eed71e" (UID: "37b48af2-8f9f-4fb4-99c2-e236f0eed71e"). InnerVolumeSpecName "kube-api-access-ndlql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:48.390139 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.390106    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:47:48.390139 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.390145    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:47:48.390323 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.390154    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndlql\" (UniqueName: \"kubernetes.io/projected/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-kube-api-access-ndlql\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:47:48.390323 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.390163    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b48af2-8f9f-4fb4-99c2-e236f0eed71e-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:47:48.634486 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.634456    2583 generic.go:356] "Generic (PLEG): container finished" podID="37b48af2-8f9f-4fb4-99c2-e236f0eed71e" containerID="604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15" exitCode=0
Sep 16 19:47:48.634673 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.634498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk" event={"ID":"37b48af2-8f9f-4fb4-99c2-e236f0eed71e","Type":"ContainerDied","Data":"604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15"}
Sep 16 19:47:48.634673 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.634500    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"
Sep 16 19:47:48.634673 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.634537    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk" event={"ID":"37b48af2-8f9f-4fb4-99c2-e236f0eed71e","Type":"ContainerDied","Data":"9657e1ca70c063e63a693e4138ff8d5a31de0986d1744c1a1f7f584ca37b8c86"}
Sep 16 19:47:48.634673 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.634559    2583 scope.go:117] "RemoveContainer" containerID="604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15"
Sep 16 19:47:48.643819 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.643799    2583 scope.go:117] "RemoveContainer" containerID="604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15"
Sep 16 19:47:48.644087 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:47:48.644067    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15\": container with ID starting with 604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15 not found: ID does not exist" containerID="604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15"
Sep 16 19:47:48.644150 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.644098    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15"} err="failed to get container status \"604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15\": rpc error: code = NotFound desc = could not find container \"604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15\": container with ID starting with 604e57e645b623fd34756507776ecacafe1c789ef8f78426be5996dfdf2c4e15 not found: ID does not exist"
Sep 16 19:47:48.671155 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.671125    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"]
Sep 16 19:47:48.673820 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:48.673796    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-tttkk"]
Sep 16 19:47:49.360100 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.360067    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"]
Sep 16 19:47:49.360475 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.360435    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b48af2-8f9f-4fb4-99c2-e236f0eed71e" containerName="main"
Sep 16 19:47:49.360475 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.360448    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b48af2-8f9f-4fb4-99c2-e236f0eed71e" containerName="main"
Sep 16 19:47:49.360553 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.360514    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b48af2-8f9f-4fb4-99c2-e236f0eed71e" containerName="main"
Sep 16 19:47:49.364936 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.364910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.368942 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.368920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-b2vnw\""
Sep 16 19:47:49.369056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.368920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 19:47:49.369176 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.369157    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 19:47:49.369251 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.369197    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 19:47:49.376053 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.376032    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"]
Sep 16 19:47:49.412168 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.412146    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b48af2-8f9f-4fb4-99c2-e236f0eed71e" path="/var/lib/kubelet/pods/37b48af2-8f9f-4fb4-99c2-e236f0eed71e/volumes"
Sep 16 19:47:49.499399 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.499374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wxj\" (UniqueName: \"kubernetes.io/projected/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-kube-api-access-d2wxj\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.499511 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.499436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.499511 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.499474    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-home\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.499511 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.499505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-dshm\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.499664 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.499644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.499724 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.499708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.600316 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.600287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.600462 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.600322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-home\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.600462 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.600391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-dshm\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.600578 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.600470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.600578 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.600499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.600578 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.600523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wxj\" (UniqueName: \"kubernetes.io/projected/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-kube-api-access-d2wxj\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.600805 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.600783    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-home\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.600805 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.600794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.600897 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.600875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.602867 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.602846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-dshm\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.603229 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.603210    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.610147 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.610076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wxj\" (UniqueName: \"kubernetes.io/projected/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-kube-api-access-d2wxj\") pod \"delw-v1-kserve-6cf8cbb847-ct626\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.676659 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.676635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:49.965307 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:49.965277    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"]
Sep 16 19:47:49.967188 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:47:49.967138    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ec3ea0_7b31_4a1c_ac4f_dff21e651682.slice/crio-8a90d1bf38d81f92e5fa24f2259bff63e1e7963aeaaa33ce93f31d62118b7dc0 WatchSource:0}: Error finding container 8a90d1bf38d81f92e5fa24f2259bff63e1e7963aeaaa33ce93f31d62118b7dc0: Status 404 returned error can't find the container with id 8a90d1bf38d81f92e5fa24f2259bff63e1e7963aeaaa33ce93f31d62118b7dc0
Sep 16 19:47:50.647295 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:50.647259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626" event={"ID":"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682","Type":"ContainerStarted","Data":"38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa"}
Sep 16 19:47:50.647704 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:50.647334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626" event={"ID":"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682","Type":"ContainerStarted","Data":"8a90d1bf38d81f92e5fa24f2259bff63e1e7963aeaaa33ce93f31d62118b7dc0"}
Sep 16 19:47:50.671716 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:50.671670    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626" podStartSLOduration=1.671656117 podStartE2EDuration="1.671656117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:50.670960158 +0000 UTC m=+2547.796764548" watchObservedRunningTime="2026-09-16 19:47:50.671656117 +0000 UTC m=+2547.797460505"
Sep 16 19:47:50.677152 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:50.677131    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:55.760293 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.760264    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"]
Sep 16 19:47:55.763846 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.763829    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.767464 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.767439    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 19:47:55.776431 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.776391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"]
Sep 16 19:47:55.854558 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.854515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlpq\" (UniqueName: \"kubernetes.io/projected/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-kube-api-access-6xlpq\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.854689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.854571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.854689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.854636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.854689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.854664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.854831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.854734    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-home\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.854831 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.854762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.955773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.955747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-home\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.955893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.955784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.955893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.955806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlpq\" (UniqueName: \"kubernetes.io/projected/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-kube-api-access-6xlpq\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.955893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.955829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.955893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.955850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.955893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.955881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.956299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.956208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-home\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.956299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.956244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.956299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.956262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.958327 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.958306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.958605 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.958586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:55.965289 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:55.965262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlpq\" (UniqueName: \"kubernetes.io/projected/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-kube-api-access-6xlpq\") pod \"delw-v2-kserve-5b84cc7ddf-nrqst\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:56.074520 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:56.074457    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:56.352277 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:56.352251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"]
Sep 16 19:47:56.355054 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:47:56.355023    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb9e9c5_3ea4_4c7b_8d62_70e4f156e88d.slice/crio-971def6f2506306e959c3366b3e098ba96f9b9ece30b8841454304ca14b80b02 WatchSource:0}: Error finding container 971def6f2506306e959c3366b3e098ba96f9b9ece30b8841454304ca14b80b02: Status 404 returned error can't find the container with id 971def6f2506306e959c3366b3e098ba96f9b9ece30b8841454304ca14b80b02
Sep 16 19:47:56.677749 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:56.677700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst" event={"ID":"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d","Type":"ContainerStarted","Data":"2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7"}
Sep 16 19:47:56.678022 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:56.677779    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst" event={"ID":"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d","Type":"ContainerStarted","Data":"971def6f2506306e959c3366b3e098ba96f9b9ece30b8841454304ca14b80b02"}
Sep 16 19:47:56.701744 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:56.701690    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst" podStartSLOduration=1.701674058 podStartE2EDuration="1.701674058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:56.700250627 +0000 UTC m=+2553.826055016" watchObservedRunningTime="2026-09-16 19:47:56.701674058 +0000 UTC m=+2553.827478445"
Sep 16 19:47:57.074954 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:57.074876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:47:57.534869 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:57.534839    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:47:59.676822 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:59.676784    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:59.689485 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:59.689462    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:47:59.701513 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:47:59.701493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:48:06.075368 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:06.075330    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:48:06.088076 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:06.088053    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:48:06.735521 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:06.735494    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:48:21.879173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:21.879139    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"]
Sep 16 19:48:21.882820 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:21.882797    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:21.886192 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:21.886171    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:48:21.897842 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:21.897784    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"]
Sep 16 19:48:22.068731 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.068699    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcwsb\" (UniqueName: \"kubernetes.io/projected/87a98e80-0faf-466e-94b1-54ab3669ff4e-kube-api-access-jcwsb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.068731 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.068735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.068973 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.068756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.068973 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.068822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.068973 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.068900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a98e80-0faf-466e-94b1-54ab3669ff4e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.068973 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.068938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.169556 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.169477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.169556 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.169524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a98e80-0faf-466e-94b1-54ab3669ff4e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.169770 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.169561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.169770 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.169601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcwsb\" (UniqueName: \"kubernetes.io/projected/87a98e80-0faf-466e-94b1-54ab3669ff4e-kube-api-access-jcwsb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.169770 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.169627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.169770 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.169655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.169989 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.169936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.169989 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.169975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.170071 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.170007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.171909 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.171886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.172215 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.172193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a98e80-0faf-466e-94b1-54ab3669ff4e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.181812 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.181791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcwsb\" (UniqueName: \"kubernetes.io/projected/87a98e80-0faf-466e-94b1-54ab3669ff4e-kube-api-access-jcwsb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.193611 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.193591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:22.475919 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.475834    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"]
Sep 16 19:48:22.479271 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:48:22.479246    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a98e80_0faf_466e_94b1_54ab3669ff4e.slice/crio-711ac4f7661bd37ffe447ef0dfaab1dd4c18cbb16a7963cebdcc631d1e186063 WatchSource:0}: Error finding container 711ac4f7661bd37ffe447ef0dfaab1dd4c18cbb16a7963cebdcc631d1e186063: Status 404 returned error can't find the container with id 711ac4f7661bd37ffe447ef0dfaab1dd4c18cbb16a7963cebdcc631d1e186063
Sep 16 19:48:22.802390 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.802312    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq" event={"ID":"87a98e80-0faf-466e-94b1-54ab3669ff4e","Type":"ContainerStarted","Data":"5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296"}
Sep 16 19:48:22.802390 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.802377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq" event={"ID":"87a98e80-0faf-466e-94b1-54ab3669ff4e","Type":"ContainerStarted","Data":"711ac4f7661bd37ffe447ef0dfaab1dd4c18cbb16a7963cebdcc631d1e186063"}
Sep 16 19:48:22.829691 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:22.829640    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq" podStartSLOduration=1.829625845 podStartE2EDuration="1.829625845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:22.827761612 +0000 UTC m=+2579.953565999" watchObservedRunningTime="2026-09-16 19:48:22.829625845 +0000 UTC m=+2579.955430266"
Sep 16 19:48:23.194522 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:23.194492    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:26.312810 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:26.312771    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"]
Sep 16 19:48:32.194370 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:32.194333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:32.207215 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:32.207192    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:32.862310 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:32.862279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:48:41.332857 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.332800    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst" podUID="fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" containerName="main" containerID="cri-o://2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7" gracePeriod=45
Sep 16 19:48:41.592679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.592614    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:48:41.636657 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.636633    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tls-certs\") pod \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") "
Sep 16 19:48:41.636815 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.636732    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-model-cache\") pod \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") "
Sep 16 19:48:41.636815 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.636767    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-dshm\") pod \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") "
Sep 16 19:48:41.636815 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.636798    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xlpq\" (UniqueName: \"kubernetes.io/projected/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-kube-api-access-6xlpq\") pod \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") "
Sep 16 19:48:41.636986 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.636861    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tmp-dir\") pod \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") "
Sep 16 19:48:41.636986 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.636891    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-home\") pod \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\" (UID: \"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d\") "
Sep 16 19:48:41.637092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.636981    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" (UID: "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:41.637198 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.637144    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" (UID: "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:41.637271 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.637202    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-home" (OuterVolumeSpecName: "home") pod "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" (UID: "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:41.637309 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.637262    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:41.637309 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.637290    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:41.639481 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.639360    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-kube-api-access-6xlpq" (OuterVolumeSpecName: "kube-api-access-6xlpq") pod "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" (UID: "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d"). InnerVolumeSpecName "kube-api-access-6xlpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:41.639597 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.639522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-dshm" (OuterVolumeSpecName: "dshm") pod "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" (UID: "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:41.639597 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.639556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" (UID: "fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:48:41.738191 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.738157    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:41.738191 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.738184    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:41.738191 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.738194    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:41.738435 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.738202    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlpq\" (UniqueName: \"kubernetes.io/projected/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d-kube-api-access-6xlpq\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:41.896591 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.896563    2583 generic.go:356] "Generic (PLEG): container finished" podID="fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" containerID="2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7" exitCode=0
Sep 16 19:48:41.896745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.896608    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"
Sep 16 19:48:41.896745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.896647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst" event={"ID":"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d","Type":"ContainerDied","Data":"2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7"}
Sep 16 19:48:41.896745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.896709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst" event={"ID":"fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d","Type":"ContainerDied","Data":"971def6f2506306e959c3366b3e098ba96f9b9ece30b8841454304ca14b80b02"}
Sep 16 19:48:41.896745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.896739    2583 scope.go:117] "RemoveContainer" containerID="2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7"
Sep 16 19:48:41.905737 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.905718    2583 scope.go:117] "RemoveContainer" containerID="2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7"
Sep 16 19:48:41.906007 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:48:41.905989    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7\": container with ID starting with 2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7 not found: ID does not exist" containerID="2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7"
Sep 16 19:48:41.906070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.906016    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7"} err="failed to get container status \"2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7\": rpc error: code = NotFound desc = could not find container \"2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7\": container with ID starting with 2e72d5f47ef779ff1b76a8b71e627110ceb3b457fcb2c660133eb05f15c168d7 not found: ID does not exist"
Sep 16 19:48:41.921489 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.921469    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"]
Sep 16 19:48:41.925748 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:41.925725    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nrqst"]
Sep 16 19:48:43.045138 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:43.045106    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"]
Sep 16 19:48:43.412677 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:43.412643    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" path="/var/lib/kubelet/pods/fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d/volumes"
Sep 16 19:48:58.064359 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.064255    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626" podUID="a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" containerName="main" containerID="cri-o://38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa" gracePeriod=45
Sep 16 19:48:58.303956 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.303932    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:48:58.474902 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.474874    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tmp-dir\") pod \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") "
Sep 16 19:48:58.475056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.474917    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tls-certs\") pod \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") "
Sep 16 19:48:58.475056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.474948    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-home\") pod \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") "
Sep 16 19:48:58.475056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.474990    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-dshm\") pod \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") "
Sep 16 19:48:58.475056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.475049    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-model-cache\") pod \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") "
Sep 16 19:48:58.475220 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.475111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2wxj\" (UniqueName: \"kubernetes.io/projected/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-kube-api-access-d2wxj\") pod \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\" (UID: \"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682\") "
Sep 16 19:48:58.475220 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.475176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" (UID: "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:58.475335 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.475211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-home" (OuterVolumeSpecName: "home") pod "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" (UID: "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:58.475335 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.475313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" (UID: "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:58.475502 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.475481    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:58.475502 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.475500    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:58.475861 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.475510    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:58.477213 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.477189    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" (UID: "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:48:58.477314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.477253    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-dshm" (OuterVolumeSpecName: "dshm") pod "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" (UID: "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:58.477314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.477269    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-kube-api-access-d2wxj" (OuterVolumeSpecName: "kube-api-access-d2wxj") pod "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" (UID: "a0ec3ea0-7b31-4a1c-ac4f-dff21e651682"). InnerVolumeSpecName "kube-api-access-d2wxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:58.578302 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.578268    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2wxj\" (UniqueName: \"kubernetes.io/projected/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-kube-api-access-d2wxj\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:58.578476 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.578312    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:58.578476 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.578393    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:48:58.976534 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.976498    2583 generic.go:356] "Generic (PLEG): container finished" podID="a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" containerID="38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa" exitCode=0
Sep 16 19:48:58.976703 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.976542    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626" event={"ID":"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682","Type":"ContainerDied","Data":"38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa"}
Sep 16 19:48:58.976703 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.976566    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"
Sep 16 19:48:58.976703 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.976608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626" event={"ID":"a0ec3ea0-7b31-4a1c-ac4f-dff21e651682","Type":"ContainerDied","Data":"8a90d1bf38d81f92e5fa24f2259bff63e1e7963aeaaa33ce93f31d62118b7dc0"}
Sep 16 19:48:58.976703 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.976630    2583 scope.go:117] "RemoveContainer" containerID="38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa"
Sep 16 19:48:58.985882 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.985864    2583 scope.go:117] "RemoveContainer" containerID="38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa"
Sep 16 19:48:58.986141 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:48:58.986116    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa\": container with ID starting with 38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa not found: ID does not exist" containerID="38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa"
Sep 16 19:48:58.986185 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:58.986150    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa"} err="failed to get container status \"38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa\": rpc error: code = NotFound desc = could not find container \"38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa\": container with ID starting with 38b81bacdd11512715c58e02e461d7a4f3987875fe48a6215feb83964052d8fa not found: ID does not exist"
Sep 16 19:48:59.000826 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:59.000803    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"]
Sep 16 19:48:59.005696 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:59.005677    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ct626"]
Sep 16 19:48:59.411115 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:48:59.411088    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" path="/var/lib/kubelet/pods/a0ec3ea0-7b31-4a1c-ac4f-dff21e651682/volumes"
Sep 16 19:49:33.692679 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:33.692640    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"]
Sep 16 19:49:33.702679 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:49:33.702643    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:49:33.702679 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:49:33.702679 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:49:33.702679 ip-10-0-139-148 kubenswrapper[2583]:  > containerID="5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296" cmd=["/bin/sleep","15"]
Sep 16 19:49:33.702972 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:33.702691    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq" message="" err=<
Sep 16 19:49:33.702972 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:49:33.702972 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:49:33.702972 ip-10-0-139-148 kubenswrapper[2583]:  >
Sep 16 19:49:33.702972 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:49:33.702704    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:49:33.702972 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:49:33.702972 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:49:33.702972 ip-10-0-139-148 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq" podUID="87a98e80-0faf-466e-94b1-54ab3669ff4e" containerName="main" containerID="cri-o://5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296"
Sep 16 19:49:33.702972 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:33.702741    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq" podUID="87a98e80-0faf-466e-94b1-54ab3669ff4e" containerName="main" containerID="cri-o://5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296" gracePeriod=60
Sep 16 19:49:33.852319 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:33.852285    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq" podUID="87a98e80-0faf-466e-94b1-54ab3669ff4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 16 19:49:34.043173 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.043137    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:49:34.146147 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.146115    2583 generic.go:356] "Generic (PLEG): container finished" podID="87a98e80-0faf-466e-94b1-54ab3669ff4e" containerID="5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296" exitCode=0
Sep 16 19:49:34.146345 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.146164    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"
Sep 16 19:49:34.146345 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.146161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq" event={"ID":"87a98e80-0faf-466e-94b1-54ab3669ff4e","Type":"ContainerDied","Data":"5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296"}
Sep 16 19:49:34.146345 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.146230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq" event={"ID":"87a98e80-0faf-466e-94b1-54ab3669ff4e","Type":"ContainerDied","Data":"711ac4f7661bd37ffe447ef0dfaab1dd4c18cbb16a7963cebdcc631d1e186063"}
Sep 16 19:49:34.146345 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.146271    2583 scope.go:117] "RemoveContainer" containerID="5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296"
Sep 16 19:49:34.155038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.155018    2583 scope.go:117] "RemoveContainer" containerID="5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296"
Sep 16 19:49:34.155315 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:49:34.155294    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296\": container with ID starting with 5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296 not found: ID does not exist" containerID="5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296"
Sep 16 19:49:34.155363 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.155326    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296"} err="failed to get container status \"5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296\": rpc error: code = NotFound desc = could not find container \"5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296\": container with ID starting with 5c12f2929bc8e801ac50e5b6515e41d0a3500b440b28a822e025793a814d9296 not found: ID does not exist"
Sep 16 19:49:34.188677 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.188654    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-home\") pod \"87a98e80-0faf-466e-94b1-54ab3669ff4e\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") "
Sep 16 19:49:34.188773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.188694    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-model-cache\") pod \"87a98e80-0faf-466e-94b1-54ab3669ff4e\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") "
Sep 16 19:49:34.188773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.188721    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcwsb\" (UniqueName: \"kubernetes.io/projected/87a98e80-0faf-466e-94b1-54ab3669ff4e-kube-api-access-jcwsb\") pod \"87a98e80-0faf-466e-94b1-54ab3669ff4e\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") "
Sep 16 19:49:34.188773 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.188749    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a98e80-0faf-466e-94b1-54ab3669ff4e-tls-certs\") pod \"87a98e80-0faf-466e-94b1-54ab3669ff4e\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") "
Sep 16 19:49:34.188892 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.188858    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-dshm\") pod \"87a98e80-0faf-466e-94b1-54ab3669ff4e\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") "
Sep 16 19:49:34.188935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.188903    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-tmp-dir\") pod \"87a98e80-0faf-466e-94b1-54ab3669ff4e\" (UID: \"87a98e80-0faf-466e-94b1-54ab3669ff4e\") "
Sep 16 19:49:34.188977 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.188953    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "87a98e80-0faf-466e-94b1-54ab3669ff4e" (UID: "87a98e80-0faf-466e-94b1-54ab3669ff4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:34.188977 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.188962    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-home" (OuterVolumeSpecName: "home") pod "87a98e80-0faf-466e-94b1-54ab3669ff4e" (UID: "87a98e80-0faf-466e-94b1-54ab3669ff4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:34.189209 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.189188    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87a98e80-0faf-466e-94b1-54ab3669ff4e" (UID: "87a98e80-0faf-466e-94b1-54ab3669ff4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:34.189209 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.189200    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:49:34.189338 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.189220    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:49:34.191114 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.191095    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a98e80-0faf-466e-94b1-54ab3669ff4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87a98e80-0faf-466e-94b1-54ab3669ff4e" (UID: "87a98e80-0faf-466e-94b1-54ab3669ff4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:34.191181 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.191147    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-dshm" (OuterVolumeSpecName: "dshm") pod "87a98e80-0faf-466e-94b1-54ab3669ff4e" (UID: "87a98e80-0faf-466e-94b1-54ab3669ff4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:34.191225 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.191190    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a98e80-0faf-466e-94b1-54ab3669ff4e-kube-api-access-jcwsb" (OuterVolumeSpecName: "kube-api-access-jcwsb") pod "87a98e80-0faf-466e-94b1-54ab3669ff4e" (UID: "87a98e80-0faf-466e-94b1-54ab3669ff4e"). InnerVolumeSpecName "kube-api-access-jcwsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:34.289951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.289871    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcwsb\" (UniqueName: \"kubernetes.io/projected/87a98e80-0faf-466e-94b1-54ab3669ff4e-kube-api-access-jcwsb\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:49:34.289951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.289901    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a98e80-0faf-466e-94b1-54ab3669ff4e-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:49:34.289951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.289915    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:49:34.289951 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.289925    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a98e80-0faf-466e-94b1-54ab3669ff4e-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:49:34.469235 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.469202    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"]
Sep 16 19:49:34.474183 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:34.474155    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995wxtmq"]
Sep 16 19:49:35.411474 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:49:35.411443    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a98e80-0faf-466e-94b1-54ab3669ff4e" path="/var/lib/kubelet/pods/87a98e80-0faf-466e-94b1-54ab3669ff4e/volumes"
Sep 16 19:50:22.629868 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:22.629736    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"]
Sep 16 19:50:26.928184 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:26.928157    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:50:26.933665 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:26.933644    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:50:26.934148 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:26.934133    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:50:26.939425 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:26.939385    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:50:27.534827 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:27.534791    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:37.535540 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:37.535493    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:37.656296 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:37.656252    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerName="main" containerID="cri-o://4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5" gracePeriod=45
Sep 16 19:50:47.535014 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:47.534971    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:47.535446 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:47.535113    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:50:57.535150 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:50:57.535107    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:51:07.535430 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:07.535372    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:51:07.911745 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:07.911720    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:51:08.024463 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.024435    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a54ec-721b-4541-9dc2-bff591012c0f-tls-certs\") pod \"e47a54ec-721b-4541-9dc2-bff591012c0f\" (UID: \"e47a54ec-721b-4541-9dc2-bff591012c0f\") "
Sep 16 19:51:08.024592 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.024549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkkr7\" (UniqueName: \"kubernetes.io/projected/e47a54ec-721b-4541-9dc2-bff591012c0f-kube-api-access-lkkr7\") pod \"e47a54ec-721b-4541-9dc2-bff591012c0f\" (UID: \"e47a54ec-721b-4541-9dc2-bff591012c0f\") "
Sep 16 19:51:08.026665 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.026637    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47a54ec-721b-4541-9dc2-bff591012c0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e47a54ec-721b-4541-9dc2-bff591012c0f" (UID: "e47a54ec-721b-4541-9dc2-bff591012c0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:51:08.026790 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.026766    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47a54ec-721b-4541-9dc2-bff591012c0f-kube-api-access-lkkr7" (OuterVolumeSpecName: "kube-api-access-lkkr7") pod "e47a54ec-721b-4541-9dc2-bff591012c0f" (UID: "e47a54ec-721b-4541-9dc2-bff591012c0f"). InnerVolumeSpecName "kube-api-access-lkkr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:51:08.125293 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.125269    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkkr7\" (UniqueName: \"kubernetes.io/projected/e47a54ec-721b-4541-9dc2-bff591012c0f-kube-api-access-lkkr7\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:51:08.125293 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.125292    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47a54ec-721b-4541-9dc2-bff591012c0f-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:51:08.579893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.579805    2583 generic.go:356] "Generic (PLEG): container finished" podID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerID="4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5" exitCode=0
Sep 16 19:51:08.579893 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.579861    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"
Sep 16 19:51:08.580354 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.579895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" event={"ID":"e47a54ec-721b-4541-9dc2-bff591012c0f","Type":"ContainerDied","Data":"4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5"}
Sep 16 19:51:08.580354 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.579959    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd" event={"ID":"e47a54ec-721b-4541-9dc2-bff591012c0f","Type":"ContainerDied","Data":"7472bc8a6b1d7f92f2d41118b566d1b97e12f5966259c0052066c0fd6a9f11bd"}
Sep 16 19:51:08.580354 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.579982    2583 scope.go:117] "RemoveContainer" containerID="4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5"
Sep 16 19:51:08.589357 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.589330    2583 scope.go:117] "RemoveContainer" containerID="4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5"
Sep 16 19:51:08.589644 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:51:08.589624    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5\": container with ID starting with 4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5 not found: ID does not exist" containerID="4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5"
Sep 16 19:51:08.589709 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.589652    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5"} err="failed to get container status \"4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5\": rpc error: code = NotFound desc = could not find container \"4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5\": container with ID starting with 4a142bdf5249e15c0db0b19e45ece6daa937d29bf685401273f2d78d323a36b5 not found: ID does not exist"
Sep 16 19:51:08.602142 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.602117    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"]
Sep 16 19:51:08.606476 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:08.606452    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schezrnvd"]
Sep 16 19:51:09.411774 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:09.411740    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" path="/var/lib/kubelet/pods/e47a54ec-721b-4541-9dc2-bff591012c0f/volumes"
Sep 16 19:51:11.751154 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.750986    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"]
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751335    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751347    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751355    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a98e80-0faf-466e-94b1-54ab3669ff4e" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751360    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a98e80-0faf-466e-94b1-54ab3669ff4e" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751380    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751385    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751395    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751403    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751477    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e47a54ec-721b-4541-9dc2-bff591012c0f" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751489    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ec3ea0-7b31-4a1c-ac4f-dff21e651682" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751497    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdb9e9c5-3ea4-4c7b-8d62-70e4f156e88d" containerName="main"
Sep 16 19:51:11.751697 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.751502    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a98e80-0faf-466e-94b1-54ab3669ff4e" containerName="main"
Sep 16 19:51:11.755708 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.755685    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.758421 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.758386    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 16 19:51:11.759534 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.759514    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 16 19:51:11.759624 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.759603    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 16 19:51:11.759698 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.759614    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-bgtl9\""
Sep 16 19:51:11.770422 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.770392    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"]
Sep 16 19:51:11.850964 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.850932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9x6\" (UniqueName: \"kubernetes.io/projected/abea579b-ae2b-40b8-badc-003d66b2eb91-kube-api-access-rs9x6\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.851092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.850979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-home\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.851092 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.851000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.851171 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.851096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-model-cache\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.851171 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.851138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-dshm\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.851240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.851172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abea579b-ae2b-40b8-badc-003d66b2eb91-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.952521 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.952491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9x6\" (UniqueName: \"kubernetes.io/projected/abea579b-ae2b-40b8-badc-003d66b2eb91-kube-api-access-rs9x6\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.952656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.952526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-home\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.952656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.952542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.952656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.952577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-model-cache\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.952656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.952603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-dshm\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.952656 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.952619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abea579b-ae2b-40b8-badc-003d66b2eb91-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.952940 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.952923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-home\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.953003 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.952985    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.953049 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.953021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-model-cache\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.955019 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.955003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-dshm\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.955337 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.955320    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abea579b-ae2b-40b8-badc-003d66b2eb91-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:11.961158 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:11.961135    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9x6\" (UniqueName: \"kubernetes.io/projected/abea579b-ae2b-40b8-badc-003d66b2eb91-kube-api-access-rs9x6\") pod \"owner-v2-kserve-5cf8959f95-bdfgq\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:12.065515 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:12.065452    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:12.341291 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:12.341266    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"]
Sep 16 19:51:12.343876 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:51:12.343847    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabea579b_ae2b_40b8_badc_003d66b2eb91.slice/crio-a8ee51393fadc8d892c9dbc29b51b8265fd3c4ff6ad535638ec3e12c34b20f59 WatchSource:0}: Error finding container a8ee51393fadc8d892c9dbc29b51b8265fd3c4ff6ad535638ec3e12c34b20f59: Status 404 returned error can't find the container with id a8ee51393fadc8d892c9dbc29b51b8265fd3c4ff6ad535638ec3e12c34b20f59
Sep 16 19:51:12.602853 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:12.602750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq" event={"ID":"abea579b-ae2b-40b8-badc-003d66b2eb91","Type":"ContainerStarted","Data":"9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8"}
Sep 16 19:51:12.602997 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:12.602858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq" event={"ID":"abea579b-ae2b-40b8-badc-003d66b2eb91","Type":"ContainerStarted","Data":"a8ee51393fadc8d892c9dbc29b51b8265fd3c4ff6ad535638ec3e12c34b20f59"}
Sep 16 19:51:12.623576 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:12.623514    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq" podStartSLOduration=1.623498528 podStartE2EDuration="1.623498528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:12.621874943 +0000 UTC m=+2749.747679332" watchObservedRunningTime="2026-09-16 19:51:12.623498528 +0000 UTC m=+2749.749302916"
Sep 16 19:51:13.066430 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:13.066381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:22.066029 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:22.065996    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:22.078296 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:22.078271    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:22.662334 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:22.662305    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:51:39.731981 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.731940    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"]
Sep 16 19:51:39.735351 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.735331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.737948 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.737927    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:51:39.739177 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.739157    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:51:39.739260 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.739160    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:51:39.739260 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.739244    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-dl6cj\""
Sep 16 19:51:39.745129 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.745097    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"]
Sep 16 19:51:39.780121 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.780097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.780211 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.780135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.780261 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.780216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.780261 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.780242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.780332 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.780281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a22882-5fe8-4dec-ae84-051b0c69c915-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.780332 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.780306    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54d96\" (UniqueName: \"kubernetes.io/projected/30a22882-5fe8-4dec-ae84-051b0c69c915-kube-api-access-54d96\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.881260 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.881227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a22882-5fe8-4dec-ae84-051b0c69c915-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.881260 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.881265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54d96\" (UniqueName: \"kubernetes.io/projected/30a22882-5fe8-4dec-ae84-051b0c69c915-kube-api-access-54d96\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.881452 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.881295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.881452 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.881320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.881452 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.881357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.881452 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.881374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.881767 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.881740    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.881836 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.881815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.881889 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.881836    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.883803 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.883782    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.883981 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.883965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a22882-5fe8-4dec-ae84-051b0c69c915-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:39.890298 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:39.890275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54d96\" (UniqueName: \"kubernetes.io/projected/30a22882-5fe8-4dec-ae84-051b0c69c915-kube-api-access-54d96\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:40.045624 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:40.045556    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:40.328440 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:40.328397    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"]
Sep 16 19:51:40.331001 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:51:40.330972    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a22882_5fe8_4dec_ae84_051b0c69c915.slice/crio-a3579ff372471e238f2448f47c0e7fffe76558f9b6af14d6872d09b2b461c517 WatchSource:0}: Error finding container a3579ff372471e238f2448f47c0e7fffe76558f9b6af14d6872d09b2b461c517: Status 404 returned error can't find the container with id a3579ff372471e238f2448f47c0e7fffe76558f9b6af14d6872d09b2b461c517
Sep 16 19:51:40.740342 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:40.740310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6" event={"ID":"30a22882-5fe8-4dec-ae84-051b0c69c915","Type":"ContainerStarted","Data":"b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d"}
Sep 16 19:51:40.740728 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:40.740359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6" event={"ID":"30a22882-5fe8-4dec-ae84-051b0c69c915","Type":"ContainerStarted","Data":"a3579ff372471e238f2448f47c0e7fffe76558f9b6af14d6872d09b2b461c517"}
Sep 16 19:51:40.762259 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:40.762206    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6" podStartSLOduration=1.762190977 podStartE2EDuration="1.762190977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:40.759683007 +0000 UTC m=+2777.885487398" watchObservedRunningTime="2026-09-16 19:51:40.762190977 +0000 UTC m=+2777.887995364"
Sep 16 19:51:41.046341 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:41.046261    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:50.046027 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:50.045940    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:50.058711 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:50.058689    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:51:50.802100 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:51:50.802066    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:52:26.616056 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.616025    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"]
Sep 16 19:52:26.619929 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.619908    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.622925 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.622852    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 19:52:26.623281 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.623263    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-5tr29\""
Sep 16 19:52:26.623377 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.623327    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:52:26.623377 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.623325    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 19:52:26.633735 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.633715    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"]
Sep 16 19:52:26.654976 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.654943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ccdf1f-a929-458b-b74a-c6bc57b80621-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.655087 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.654983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.655087 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.655013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.655087 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.655049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.655265 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.655104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sgqx\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-kube-api-access-5sgqx\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.655265 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.655136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.655265 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.655167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.655470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.655281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.655470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.655318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.756434 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ccdf1f-a929-458b-b74a-c6bc57b80621-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.756614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.756614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.756614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.756614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sgqx\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-kube-api-access-5sgqx\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.756614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.756614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.756614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.756614 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.757022 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.756923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.757088 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.757066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.757125 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.757080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.757188 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.757173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.757242 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.757181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ccdf1f-a929-458b-b74a-c6bc57b80621-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.758906 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.758884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.759217 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.759200    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.767426 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.767389    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.767587 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.767570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sgqx\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-kube-api-access-5sgqx\") pod \"router-gateway-1-openshift-default-55f8b478f9-bcm8l\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:26.930358 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:26.930339    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:27.214669 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:27.214642    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"]
Sep 16 19:52:27.216640 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:52:27.216609    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ccdf1f_a929_458b_b74a_c6bc57b80621.slice/crio-00c27ad65cd725d6136fd41427da7275beb2be9f6d6089fce632d014a8c3a0fa WatchSource:0}: Error finding container 00c27ad65cd725d6136fd41427da7275beb2be9f6d6089fce632d014a8c3a0fa: Status 404 returned error can't find the container with id 00c27ad65cd725d6136fd41427da7275beb2be9f6d6089fce632d014a8c3a0fa
Sep 16 19:52:27.218836 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:27.218799    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:52:27.218929 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:27.218889    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:52:27.218979 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:27.218931    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:52:27.966771 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:27.966737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" event={"ID":"82ccdf1f-a929-458b-b74a-c6bc57b80621","Type":"ContainerStarted","Data":"99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298"}
Sep 16 19:52:27.967136 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:27.966794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" event={"ID":"82ccdf1f-a929-458b-b74a-c6bc57b80621","Type":"ContainerStarted","Data":"00c27ad65cd725d6136fd41427da7275beb2be9f6d6089fce632d014a8c3a0fa"}
Sep 16 19:52:27.992338 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:27.992288    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" podStartSLOduration=1.992273535 podStartE2EDuration="1.992273535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:27.989661927 +0000 UTC m=+2825.115466316" watchObservedRunningTime="2026-09-16 19:52:27.992273535 +0000 UTC m=+2825.118077922"
Sep 16 19:52:28.930635 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:28.930602    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:28.935037 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:28.935013    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:28.971668 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:28.971643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:28.972444 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:28.972398    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:52:31.357282 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:31.357248    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"]
Sep 16 19:52:46.377094 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.377034    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq" podUID="abea579b-ae2b-40b8-badc-003d66b2eb91" containerName="main" containerID="cri-o://9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8" gracePeriod=45
Sep 16 19:52:46.619015 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.618991    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:52:46.726033 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.725956    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-home\") pod \"abea579b-ae2b-40b8-badc-003d66b2eb91\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") "
Sep 16 19:52:46.726167 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.726096    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abea579b-ae2b-40b8-badc-003d66b2eb91-tls-certs\") pod \"abea579b-ae2b-40b8-badc-003d66b2eb91\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") "
Sep 16 19:52:46.726167 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.726141    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs9x6\" (UniqueName: \"kubernetes.io/projected/abea579b-ae2b-40b8-badc-003d66b2eb91-kube-api-access-rs9x6\") pod \"abea579b-ae2b-40b8-badc-003d66b2eb91\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") "
Sep 16 19:52:46.726243 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.726175    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-tmp-dir\") pod \"abea579b-ae2b-40b8-badc-003d66b2eb91\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") "
Sep 16 19:52:46.726243 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.726199    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-model-cache\") pod \"abea579b-ae2b-40b8-badc-003d66b2eb91\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") "
Sep 16 19:52:46.726243 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.726233    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-dshm\") pod \"abea579b-ae2b-40b8-badc-003d66b2eb91\" (UID: \"abea579b-ae2b-40b8-badc-003d66b2eb91\") "
Sep 16 19:52:46.726369 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.726245    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-home" (OuterVolumeSpecName: "home") pod "abea579b-ae2b-40b8-badc-003d66b2eb91" (UID: "abea579b-ae2b-40b8-badc-003d66b2eb91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:46.726512 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.726483    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abea579b-ae2b-40b8-badc-003d66b2eb91" (UID: "abea579b-ae2b-40b8-badc-003d66b2eb91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:46.726616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.726506    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-model-cache" (OuterVolumeSpecName: "model-cache") pod "abea579b-ae2b-40b8-badc-003d66b2eb91" (UID: "abea579b-ae2b-40b8-badc-003d66b2eb91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:46.726616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.726504    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:52:46.728528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.728503    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abea579b-ae2b-40b8-badc-003d66b2eb91-kube-api-access-rs9x6" (OuterVolumeSpecName: "kube-api-access-rs9x6") pod "abea579b-ae2b-40b8-badc-003d66b2eb91" (UID: "abea579b-ae2b-40b8-badc-003d66b2eb91"). InnerVolumeSpecName "kube-api-access-rs9x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:52:46.728528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.728517    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abea579b-ae2b-40b8-badc-003d66b2eb91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abea579b-ae2b-40b8-badc-003d66b2eb91" (UID: "abea579b-ae2b-40b8-badc-003d66b2eb91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:52:46.728659 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.728539    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-dshm" (OuterVolumeSpecName: "dshm") pod "abea579b-ae2b-40b8-badc-003d66b2eb91" (UID: "abea579b-ae2b-40b8-badc-003d66b2eb91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:46.827293 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.827273    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abea579b-ae2b-40b8-badc-003d66b2eb91-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:52:46.827293 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.827294    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9x6\" (UniqueName: \"kubernetes.io/projected/abea579b-ae2b-40b8-badc-003d66b2eb91-kube-api-access-rs9x6\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:52:46.827449 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.827303    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:52:46.827449 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.827315    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:52:46.827449 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:46.827322    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abea579b-ae2b-40b8-badc-003d66b2eb91-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:52:47.055820 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.055743    2583 generic.go:356] "Generic (PLEG): container finished" podID="abea579b-ae2b-40b8-badc-003d66b2eb91" containerID="9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8" exitCode=0
Sep 16 19:52:47.055820 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.055786    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"
Sep 16 19:52:47.055961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.055783    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq" event={"ID":"abea579b-ae2b-40b8-badc-003d66b2eb91","Type":"ContainerDied","Data":"9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8"}
Sep 16 19:52:47.055961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.055925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq" event={"ID":"abea579b-ae2b-40b8-badc-003d66b2eb91","Type":"ContainerDied","Data":"a8ee51393fadc8d892c9dbc29b51b8265fd3c4ff6ad535638ec3e12c34b20f59"}
Sep 16 19:52:47.055961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.055952    2583 scope.go:117] "RemoveContainer" containerID="9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8"
Sep 16 19:52:47.065026 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.065008    2583 scope.go:117] "RemoveContainer" containerID="9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8"
Sep 16 19:52:47.065312 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:52:47.065295    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8\": container with ID starting with 9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8 not found: ID does not exist" containerID="9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8"
Sep 16 19:52:47.065362 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.065320    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8"} err="failed to get container status \"9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8\": rpc error: code = NotFound desc = could not find container \"9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8\": container with ID starting with 9e6a5d08a0fc336200e0da3977aab2f5ffa35c27aa3846f8eb190b4f6743fde8 not found: ID does not exist"
Sep 16 19:52:47.081784 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.081759    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"]
Sep 16 19:52:47.087308 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.087283    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdfgq"]
Sep 16 19:52:47.411465 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:52:47.411435    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abea579b-ae2b-40b8-badc-003d66b2eb91" path="/var/lib/kubelet/pods/abea579b-ae2b-40b8-badc-003d66b2eb91/volumes"
Sep 16 19:53:08.973606 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:08.973566    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"]
Sep 16 19:53:08.974125 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:08.974110    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abea579b-ae2b-40b8-badc-003d66b2eb91" containerName="main"
Sep 16 19:53:08.974171 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:08.974129    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abea579b-ae2b-40b8-badc-003d66b2eb91" containerName="main"
Sep 16 19:53:08.974222 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:08.974212    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abea579b-ae2b-40b8-badc-003d66b2eb91" containerName="main"
Sep 16 19:53:08.977571 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:08.977553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:08.981348 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:08.981317    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-kvqgf\""
Sep 16 19:53:08.981498 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:08.981317    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:53:08.985454 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:08.985349    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"]
Sep 16 19:53:09.103706 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:09.103661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0037de88-eddb-43ea-9628-4fe9f6306dcb-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88\" (UID: \"0037de88-eddb-43ea-9628-4fe9f6306dcb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:09.103886 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:09.103741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hdr\" (UniqueName: \"kubernetes.io/projected/0037de88-eddb-43ea-9628-4fe9f6306dcb-kube-api-access-85hdr\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88\" (UID: \"0037de88-eddb-43ea-9628-4fe9f6306dcb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:09.205217 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:09.205186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0037de88-eddb-43ea-9628-4fe9f6306dcb-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88\" (UID: \"0037de88-eddb-43ea-9628-4fe9f6306dcb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:09.205448 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:09.205279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hdr\" (UniqueName: \"kubernetes.io/projected/0037de88-eddb-43ea-9628-4fe9f6306dcb-kube-api-access-85hdr\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88\" (UID: \"0037de88-eddb-43ea-9628-4fe9f6306dcb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:09.208552 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:09.208527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0037de88-eddb-43ea-9628-4fe9f6306dcb-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88\" (UID: \"0037de88-eddb-43ea-9628-4fe9f6306dcb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:09.213314 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:09.213290    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hdr\" (UniqueName: \"kubernetes.io/projected/0037de88-eddb-43ea-9628-4fe9f6306dcb-kube-api-access-85hdr\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88\" (UID: \"0037de88-eddb-43ea-9628-4fe9f6306dcb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:09.288950 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:09.288893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:09.564932 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:09.564906    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"]
Sep 16 19:53:09.567609 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:53:09.567584    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0037de88_eddb_43ea_9628_4fe9f6306dcb.slice/crio-76ce159e705d906b99c966cb5f41cd53268d1bad262c6b45dfbde429803ef8c7 WatchSource:0}: Error finding container 76ce159e705d906b99c966cb5f41cd53268d1bad262c6b45dfbde429803ef8c7: Status 404 returned error can't find the container with id 76ce159e705d906b99c966cb5f41cd53268d1bad262c6b45dfbde429803ef8c7
Sep 16 19:53:10.173981 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:10.173939    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" event={"ID":"0037de88-eddb-43ea-9628-4fe9f6306dcb","Type":"ContainerStarted","Data":"0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e"}
Sep 16 19:53:10.174392 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:10.173989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" event={"ID":"0037de88-eddb-43ea-9628-4fe9f6306dcb","Type":"ContainerStarted","Data":"76ce159e705d906b99c966cb5f41cd53268d1bad262c6b45dfbde429803ef8c7"}
Sep 16 19:53:10.174392 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:10.174061    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:10.195134 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:10.195084    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" podStartSLOduration=2.195070876 podStartE2EDuration="2.195070876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:53:10.193210341 +0000 UTC m=+2867.319014730" watchObservedRunningTime="2026-09-16 19:53:10.195070876 +0000 UTC m=+2867.320875264"
Sep 16 19:53:28.267682 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:28.267643    2583 generic.go:356] "Generic (PLEG): container finished" podID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerID="99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298" exitCode=0
Sep 16 19:53:28.268127 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:28.267717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" event={"ID":"82ccdf1f-a929-458b-b74a-c6bc57b80621","Type":"ContainerDied","Data":"99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298"}
Sep 16 19:53:28.268194 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:28.268155    2583 scope.go:117] "RemoveContainer" containerID="99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298"
Sep 16 19:53:28.269146 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:28.269117    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:53:28.269240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:28.269185    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:53:28.269240 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:28.269218    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 19:53:28.972321 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:28.972294    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:53:29.273931 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:29.273844    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" event={"ID":"82ccdf1f-a929-458b-b74a-c6bc57b80621","Type":"ContainerStarted","Data":"f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0"}
Sep 16 19:53:29.931091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:29.931058    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:53:29.932073 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:29.932042    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 19:53:30.931242 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:30.931201    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 19:53:31.931191 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:31.931153    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 19:53:32.931181 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:32.931136    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 19:53:33.931714 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:33.931676    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 19:53:34.934949 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:34.934921    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:53:34.935341 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:34.935191    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:53:34.935966 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:34.935949    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:53:41.180990 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:41.180960    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:53:46.012707 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:46.012669    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"]
Sep 16 19:53:51.181091 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:51.181052    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:52.652176 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:52.652139    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"]
Sep 16 19:53:52.652644 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:52.652349    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy" containerID="cri-o://f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0" gracePeriod=30
Sep 16 19:53:57.919045 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:57.918984    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:53:58.044848 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.044815    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-socket\") pod \"82ccdf1f-a929-458b-b74a-c6bc57b80621\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") "
Sep 16 19:53:58.045009 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.044858    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-token\") pod \"82ccdf1f-a929-458b-b74a-c6bc57b80621\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") "
Sep 16 19:53:58.045009 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.044876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-envoy\") pod \"82ccdf1f-a929-458b-b74a-c6bc57b80621\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") "
Sep 16 19:53:58.045009 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.044896    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ccdf1f-a929-458b-b74a-c6bc57b80621-istiod-ca-cert\") pod \"82ccdf1f-a929-458b-b74a-c6bc57b80621\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") "
Sep 16 19:53:58.045009 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.044916    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sgqx\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-kube-api-access-5sgqx\") pod \"82ccdf1f-a929-458b-b74a-c6bc57b80621\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") "
Sep 16 19:53:58.045239 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045112    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82ccdf1f-a929-458b-b74a-c6bc57b80621" (UID: "82ccdf1f-a929-458b-b74a-c6bc57b80621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:58.045239 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-podinfo\") pod \"82ccdf1f-a929-458b-b74a-c6bc57b80621\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") "
Sep 16 19:53:58.045239 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045209    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-data\") pod \"82ccdf1f-a929-458b-b74a-c6bc57b80621\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") "
Sep 16 19:53:58.045445 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045237    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-certs\") pod \"82ccdf1f-a929-458b-b74a-c6bc57b80621\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") "
Sep 16 19:53:58.045445 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045312    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ccdf1f-a929-458b-b74a-c6bc57b80621-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82ccdf1f-a929-458b-b74a-c6bc57b80621" (UID: "82ccdf1f-a929-458b-b74a-c6bc57b80621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:53:58.045445 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045329    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-credential-socket\") pod \"82ccdf1f-a929-458b-b74a-c6bc57b80621\" (UID: \"82ccdf1f-a929-458b-b74a-c6bc57b80621\") "
Sep 16 19:53:58.045634 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045447    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-data" (OuterVolumeSpecName: "istio-data") pod "82ccdf1f-a929-458b-b74a-c6bc57b80621" (UID: "82ccdf1f-a929-458b-b74a-c6bc57b80621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:58.045803 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045736    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:53:58.045803 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045750    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ccdf1f-a929-458b-b74a-c6bc57b80621-istiod-ca-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:53:58.045803 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045758    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-data\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:53:58.045803 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045780    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82ccdf1f-a929-458b-b74a-c6bc57b80621" (UID: "82ccdf1f-a929-458b-b74a-c6bc57b80621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:58.046060 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.045901    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82ccdf1f-a929-458b-b74a-c6bc57b80621" (UID: "82ccdf1f-a929-458b-b74a-c6bc57b80621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:58.047615 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.047591    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-token" (OuterVolumeSpecName: "istio-token") pod "82ccdf1f-a929-458b-b74a-c6bc57b80621" (UID: "82ccdf1f-a929-458b-b74a-c6bc57b80621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:53:58.047734 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.047670    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-kube-api-access-5sgqx" (OuterVolumeSpecName: "kube-api-access-5sgqx") pod "82ccdf1f-a929-458b-b74a-c6bc57b80621" (UID: "82ccdf1f-a929-458b-b74a-c6bc57b80621"). InnerVolumeSpecName "kube-api-access-5sgqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:53:58.047734 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.047709    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82ccdf1f-a929-458b-b74a-c6bc57b80621" (UID: "82ccdf1f-a929-458b-b74a-c6bc57b80621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:58.047899 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.047883    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82ccdf1f-a929-458b-b74a-c6bc57b80621" (UID: "82ccdf1f-a929-458b-b74a-c6bc57b80621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:53:58.146317 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.146296    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-token\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:53:58.146317 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.146314    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-envoy\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:53:58.146470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.146323    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sgqx\" (UniqueName: \"kubernetes.io/projected/82ccdf1f-a929-458b-b74a-c6bc57b80621-kube-api-access-5sgqx\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:53:58.146470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.146334    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ccdf1f-a929-458b-b74a-c6bc57b80621-istio-podinfo\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:53:58.146470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.146342    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-workload-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:53:58.146470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.146350    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ccdf1f-a929-458b-b74a-c6bc57b80621-credential-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:53:58.417061 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.417030    2583 generic.go:356] "Generic (PLEG): container finished" podID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerID="f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0" exitCode=0
Sep 16 19:53:58.417214 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.417076    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"
Sep 16 19:53:58.417214 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.417114    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" event={"ID":"82ccdf1f-a929-458b-b74a-c6bc57b80621","Type":"ContainerDied","Data":"f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0"}
Sep 16 19:53:58.417214 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.417173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l" event={"ID":"82ccdf1f-a929-458b-b74a-c6bc57b80621","Type":"ContainerDied","Data":"00c27ad65cd725d6136fd41427da7275beb2be9f6d6089fce632d014a8c3a0fa"}
Sep 16 19:53:58.417214 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.417201    2583 scope.go:117] "RemoveContainer" containerID="f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0"
Sep 16 19:53:58.427026 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.427009    2583 scope.go:117] "RemoveContainer" containerID="99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298"
Sep 16 19:53:58.434935 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.434917    2583 scope.go:117] "RemoveContainer" containerID="f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0"
Sep 16 19:53:58.435188 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:53:58.435170    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0\": container with ID starting with f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0 not found: ID does not exist" containerID="f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0"
Sep 16 19:53:58.435232 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.435197    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0"} err="failed to get container status \"f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0\": rpc error: code = NotFound desc = could not find container \"f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0\": container with ID starting with f683c10a3d4bd945977cc1dbaef47f21d7687515c3561c9912df8c0ff96991a0 not found: ID does not exist"
Sep 16 19:53:58.435232 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.435213    2583 scope.go:117] "RemoveContainer" containerID="99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298"
Sep 16 19:53:58.435496 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:53:58.435475    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298\": container with ID starting with 99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298 not found: ID does not exist" containerID="99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298"
Sep 16 19:53:58.435563 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.435500    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298"} err="failed to get container status \"99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298\": rpc error: code = NotFound desc = could not find container \"99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298\": container with ID starting with 99309f6d4df016206b589e3fbd073eff3f2a7cff407982423e9771cbd2a7c298 not found: ID does not exist"
Sep 16 19:53:58.439963 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.439939    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"]
Sep 16 19:53:58.446339 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:58.446320    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-bcm8l"]
Sep 16 19:53:59.411553 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:53:59.411522    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" path="/var/lib/kubelet/pods/82ccdf1f-a929-458b-b74a-c6bc57b80621/volumes"
Sep 16 19:54:01.033374 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:01.033340    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerName="main" containerID="cri-o://0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e" gracePeriod=45
Sep 16 19:54:01.180458 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:01.180406    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:03.758137 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.758106    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"]
Sep 16 19:54:03.758542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.758470    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy"
Sep 16 19:54:03.758542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.758483    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy"
Sep 16 19:54:03.758542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.758496    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy"
Sep 16 19:54:03.758542 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.758502    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy"
Sep 16 19:54:03.758689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.758574    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy"
Sep 16 19:54:03.758689 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.758583    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ccdf1f-a929-458b-b74a-c6bc57b80621" containerName="istio-proxy"
Sep 16 19:54:03.760616 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.760599    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:54:03.763980 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.763957    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-frq67\""
Sep 16 19:54:03.764920 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.764897    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:54:03.765031 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.764947    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:54:03.765031 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.764962    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:54:03.765031 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.764989    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-hvwzb\""
Sep 16 19:54:03.794244 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.794201    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"]
Sep 16 19:54:03.893892 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.893867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2wl\" (UniqueName: \"kubernetes.io/projected/3250a78a-96a9-41cb-bff9-61dd1427810d-kube-api-access-zf2wl\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2\" (UID: \"3250a78a-96a9-41cb-bff9-61dd1427810d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:54:03.894004 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.893910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250a78a-96a9-41cb-bff9-61dd1427810d-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2\" (UID: \"3250a78a-96a9-41cb-bff9-61dd1427810d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:54:03.994822 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.994790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2wl\" (UniqueName: \"kubernetes.io/projected/3250a78a-96a9-41cb-bff9-61dd1427810d-kube-api-access-zf2wl\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2\" (UID: \"3250a78a-96a9-41cb-bff9-61dd1427810d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:54:03.994947 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.994829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250a78a-96a9-41cb-bff9-61dd1427810d-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2\" (UID: \"3250a78a-96a9-41cb-bff9-61dd1427810d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:54:03.997243 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:03.997218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250a78a-96a9-41cb-bff9-61dd1427810d-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2\" (UID: \"3250a78a-96a9-41cb-bff9-61dd1427810d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:54:04.019771 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:04.019713    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2wl\" (UniqueName: \"kubernetes.io/projected/3250a78a-96a9-41cb-bff9-61dd1427810d-kube-api-access-zf2wl\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2\" (UID: \"3250a78a-96a9-41cb-bff9-61dd1427810d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:54:04.070681 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:04.070662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:54:04.346908 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:04.346884    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"]
Sep 16 19:54:04.349254 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:54:04.349224    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3250a78a_96a9_41cb_bff9_61dd1427810d.slice/crio-f0dd725e8ef76adeea13d2fdf7d2d54b2f6c3e950920f05046ceb0365c195980 WatchSource:0}: Error finding container f0dd725e8ef76adeea13d2fdf7d2d54b2f6c3e950920f05046ceb0365c195980: Status 404 returned error can't find the container with id f0dd725e8ef76adeea13d2fdf7d2d54b2f6c3e950920f05046ceb0365c195980
Sep 16 19:54:04.448650 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:04.448614    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" event={"ID":"3250a78a-96a9-41cb-bff9-61dd1427810d","Type":"ContainerStarted","Data":"1154e3db349db2a0f5cc580b0f7c6ff31fa5fa8e65a3e91ba2602d8a5f35f022"}
Sep 16 19:54:04.448822 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:04.448674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" event={"ID":"3250a78a-96a9-41cb-bff9-61dd1427810d","Type":"ContainerStarted","Data":"f0dd725e8ef76adeea13d2fdf7d2d54b2f6c3e950920f05046ceb0365c195980"}
Sep 16 19:54:04.448822 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:04.448765    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:54:04.466132 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:04.466089    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" podStartSLOduration=1.466074779 podStartE2EDuration="1.466074779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:04.465090187 +0000 UTC m=+2921.590894577" watchObservedRunningTime="2026-09-16 19:54:04.466074779 +0000 UTC m=+2921.591879166"
Sep 16 19:54:11.180805 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:11.180766    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:11.181303 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:11.180865    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:54:21.181150 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:21.181113    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:31.180649 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:54:31.180621    2583 logging.go:55] [core] [Channel #659 SubChannel #660]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.66:9003", ServerName: "10.134.0.66:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.66:9003: connect: connection refused"
Sep 16 19:54:31.277632 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.277608    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:54:31.312911 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.312837    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85hdr\" (UniqueName: \"kubernetes.io/projected/0037de88-eddb-43ea-9628-4fe9f6306dcb-kube-api-access-85hdr\") pod \"0037de88-eddb-43ea-9628-4fe9f6306dcb\" (UID: \"0037de88-eddb-43ea-9628-4fe9f6306dcb\") "
Sep 16 19:54:31.312911 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.312900    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0037de88-eddb-43ea-9628-4fe9f6306dcb-tls-certs\") pod \"0037de88-eddb-43ea-9628-4fe9f6306dcb\" (UID: \"0037de88-eddb-43ea-9628-4fe9f6306dcb\") "
Sep 16 19:54:31.315153 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.315123    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0037de88-eddb-43ea-9628-4fe9f6306dcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0037de88-eddb-43ea-9628-4fe9f6306dcb" (UID: "0037de88-eddb-43ea-9628-4fe9f6306dcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:31.315250 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.315196    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0037de88-eddb-43ea-9628-4fe9f6306dcb-kube-api-access-85hdr" (OuterVolumeSpecName: "kube-api-access-85hdr") pod "0037de88-eddb-43ea-9628-4fe9f6306dcb" (UID: "0037de88-eddb-43ea-9628-4fe9f6306dcb"). InnerVolumeSpecName "kube-api-access-85hdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:31.413694 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.413668    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0037de88-eddb-43ea-9628-4fe9f6306dcb-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.413694 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.413689    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hdr\" (UniqueName: \"kubernetes.io/projected/0037de88-eddb-43ea-9628-4fe9f6306dcb-kube-api-access-85hdr\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.579274 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.579208    2583 generic.go:356] "Generic (PLEG): container finished" podID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerID="0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e" exitCode=0
Sep 16 19:54:31.579274 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.579246    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"
Sep 16 19:54:31.579429 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.579289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" event={"ID":"0037de88-eddb-43ea-9628-4fe9f6306dcb","Type":"ContainerDied","Data":"0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e"}
Sep 16 19:54:31.579429 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.579339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" event={"ID":"0037de88-eddb-43ea-9628-4fe9f6306dcb","Type":"ContainerDied","Data":"76ce159e705d906b99c966cb5f41cd53268d1bad262c6b45dfbde429803ef8c7"}
Sep 16 19:54:31.579429 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.579365    2583 scope.go:117] "RemoveContainer" containerID="0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e"
Sep 16 19:54:31.588131 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.588116    2583 scope.go:117] "RemoveContainer" containerID="0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e"
Sep 16 19:54:31.588391 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:54:31.588376    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e\": container with ID starting with 0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e not found: ID does not exist" containerID="0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e"
Sep 16 19:54:31.588470 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.588396    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e"} err="failed to get container status \"0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e\": rpc error: code = NotFound desc = could not find container \"0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e\": container with ID starting with 0bc1947110aa798746a7d4122e9dfbd43c97ea8acaad990a90a707210631f08e not found: ID does not exist"
Sep 16 19:54:31.597475 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.597456    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"]
Sep 16 19:54:31.603562 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:31.603543    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88"]
Sep 16 19:54:32.180930 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:32.180893    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69l88" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.66:9003\" within 1s: context deadline exceeded"
Sep 16 19:54:32.181315 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:54:32.181007    2583 logging.go:55] [core] [Channel #659 SubChannel #660]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.66:9003", ServerName: "10.134.0.66:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.66:9003: operation was canceled"
Sep 16 19:54:33.411993 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:33.411953    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" path="/var/lib/kubelet/pods/0037de88-eddb-43ea-9628-4fe9f6306dcb/volumes"
Sep 16 19:54:34.119913 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.119883    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"]
Sep 16 19:54:34.130125 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:54:34.130090    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:54:34.130125 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:54:34.130125 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:54:34.130125 ip-10-0-139-148 kubenswrapper[2583]:  > containerID="b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d" cmd=["/bin/sleep","15"]
Sep 16 19:54:34.130454 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.130143    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6" message="" err=<
Sep 16 19:54:34.130454 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:54:34.130454 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:54:34.130454 ip-10-0-139-148 kubenswrapper[2583]:  >
Sep 16 19:54:34.130454 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:54:34.130156    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:54:34.130454 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:54:34.130454 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 19:54:34.130454 ip-10-0-139-148 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6" podUID="30a22882-5fe8-4dec-ae84-051b0c69c915" containerName="main" containerID="cri-o://b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d"
Sep 16 19:54:34.130454 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.130203    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6" podUID="30a22882-5fe8-4dec-ae84-051b0c69c915" containerName="main" containerID="cri-o://b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d" gracePeriod=60
Sep 16 19:54:34.398278 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.398253    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:54:34.440034 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.439997    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-dshm\") pod \"30a22882-5fe8-4dec-ae84-051b0c69c915\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") "
Sep 16 19:54:34.440493 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.440076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a22882-5fe8-4dec-ae84-051b0c69c915-tls-certs\") pod \"30a22882-5fe8-4dec-ae84-051b0c69c915\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") "
Sep 16 19:54:34.440493 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.440135    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-tmp-dir\") pod \"30a22882-5fe8-4dec-ae84-051b0c69c915\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") "
Sep 16 19:54:34.440493 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.440189    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-model-cache\") pod \"30a22882-5fe8-4dec-ae84-051b0c69c915\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") "
Sep 16 19:54:34.440493 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.440219    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54d96\" (UniqueName: \"kubernetes.io/projected/30a22882-5fe8-4dec-ae84-051b0c69c915-kube-api-access-54d96\") pod \"30a22882-5fe8-4dec-ae84-051b0c69c915\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") "
Sep 16 19:54:34.440493 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.440268    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-home\") pod \"30a22882-5fe8-4dec-ae84-051b0c69c915\" (UID: \"30a22882-5fe8-4dec-ae84-051b0c69c915\") "
Sep 16 19:54:34.440779 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.440710    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-home" (OuterVolumeSpecName: "home") pod "30a22882-5fe8-4dec-ae84-051b0c69c915" (UID: "30a22882-5fe8-4dec-ae84-051b0c69c915"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:34.441157 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.441131    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30a22882-5fe8-4dec-ae84-051b0c69c915" (UID: "30a22882-5fe8-4dec-ae84-051b0c69c915"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:34.441967 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.441936    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-model-cache" (OuterVolumeSpecName: "model-cache") pod "30a22882-5fe8-4dec-ae84-051b0c69c915" (UID: "30a22882-5fe8-4dec-ae84-051b0c69c915"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:34.444800 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.444763    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a22882-5fe8-4dec-ae84-051b0c69c915-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30a22882-5fe8-4dec-ae84-051b0c69c915" (UID: "30a22882-5fe8-4dec-ae84-051b0c69c915"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:34.444908 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.444878    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a22882-5fe8-4dec-ae84-051b0c69c915-kube-api-access-54d96" (OuterVolumeSpecName: "kube-api-access-54d96") pod "30a22882-5fe8-4dec-ae84-051b0c69c915" (UID: "30a22882-5fe8-4dec-ae84-051b0c69c915"). InnerVolumeSpecName "kube-api-access-54d96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:34.445277 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.445255    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-dshm" (OuterVolumeSpecName: "dshm") pod "30a22882-5fe8-4dec-ae84-051b0c69c915" (UID: "30a22882-5fe8-4dec-ae84-051b0c69c915"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:34.541557 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.541525    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:54:34.541711 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.541576    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54d96\" (UniqueName: \"kubernetes.io/projected/30a22882-5fe8-4dec-ae84-051b0c69c915-kube-api-access-54d96\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:54:34.541711 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.541586    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:54:34.541711 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.541595    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:54:34.541711 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.541603    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30a22882-5fe8-4dec-ae84-051b0c69c915-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:54:34.541711 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.541612    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a22882-5fe8-4dec-ae84-051b0c69c915-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:54:34.595228 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.595086    2583 generic.go:356] "Generic (PLEG): container finished" podID="30a22882-5fe8-4dec-ae84-051b0c69c915" containerID="b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d" exitCode=0
Sep 16 19:54:34.595228 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.595148    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"
Sep 16 19:54:34.595228 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.595149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6" event={"ID":"30a22882-5fe8-4dec-ae84-051b0c69c915","Type":"ContainerDied","Data":"b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d"}
Sep 16 19:54:34.595228 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.595222    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6" event={"ID":"30a22882-5fe8-4dec-ae84-051b0c69c915","Type":"ContainerDied","Data":"a3579ff372471e238f2448f47c0e7fffe76558f9b6af14d6872d09b2b461c517"}
Sep 16 19:54:34.595623 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.595260    2583 scope.go:117] "RemoveContainer" containerID="b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d"
Sep 16 19:54:34.605791 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.605770    2583 scope.go:117] "RemoveContainer" containerID="b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d"
Sep 16 19:54:34.606114 ip-10-0-139-148 kubenswrapper[2583]: E0916 19:54:34.606087    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d\": container with ID starting with b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d not found: ID does not exist" containerID="b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d"
Sep 16 19:54:34.606195 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.606128    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d"} err="failed to get container status \"b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d\": rpc error: code = NotFound desc = could not find container \"b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d\": container with ID starting with b127225f202397744710e59e082f29d28270c8dfc8ae0419fbabe4afb23e228d not found: ID does not exist"
Sep 16 19:54:34.620391 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.620361    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"]
Sep 16 19:54:34.626494 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:34.626468    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lzqr6"]
Sep 16 19:54:35.413267 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:35.413231    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a22882-5fe8-4dec-ae84-051b0c69c915" path="/var/lib/kubelet/pods/30a22882-5fe8-4dec-ae84-051b0c69c915/volumes"
Sep 16 19:54:35.455587 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:54:35.455557    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:55:26.968303 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:26.968274    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:55:26.973561 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:26.973540    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 19:55:26.973709 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:26.973562    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:55:26.979361 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:26.979346    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 19:55:29.080632 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.080602    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"]
Sep 16 19:55:29.081112 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.081092    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a22882-5fe8-4dec-ae84-051b0c69c915" containerName="main"
Sep 16 19:55:29.081112 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.081110    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a22882-5fe8-4dec-ae84-051b0c69c915" containerName="main"
Sep 16 19:55:29.081247 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.081125    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerName="main"
Sep 16 19:55:29.081247 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.081134    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerName="main"
Sep 16 19:55:29.081357 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.081252    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0037de88-eddb-43ea-9628-4fe9f6306dcb" containerName="main"
Sep 16 19:55:29.081357 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.081267    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a22882-5fe8-4dec-ae84-051b0c69c915" containerName="main"
Sep 16 19:55:29.084137 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.084115    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.087018 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.086979    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 19:55:29.087018 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.086991    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:55:29.087260 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.087129    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:55:29.087260 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.087176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:55:29.087381 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.087261    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:55:29.087381 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.087306    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-b6d42\""
Sep 16 19:55:29.101552 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.101525    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"]
Sep 16 19:55:29.158919 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.158895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.159036 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.158938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.159036 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.158968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.159036 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.159016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.159143 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.159057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.159143 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.159077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea3211-5bc5-450e-ae80-6f0d48f839ae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.159143 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.159097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.159143 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.159120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6wht\" (UniqueName: \"kubernetes.io/projected/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kube-api-access-j6wht\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260107 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260243 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260243 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260243 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260243 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260222    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea3211-5bc5-450e-ae80-6f0d48f839ae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260507 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260507 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wht\" (UniqueName: \"kubernetes.io/projected/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kube-api-access-j6wht\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260507 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260672 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260672 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260814 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.260814 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260805    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.261009 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.260989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea3211-5bc5-450e-ae80-6f0d48f839ae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.262811 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.262783    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.263102 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.263084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.271223 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.271205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wht\" (UniqueName: \"kubernetes.io/projected/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kube-api-access-j6wht\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.393829 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.393804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:29.677049 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.677023    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"]
Sep 16 19:55:29.679463 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:55:29.679432    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ea3211_5bc5_450e_ae80_6f0d48f839ae.slice/crio-20010e424cf73a85b51e890727fab061579eb6a2e60494de6db7e6906ee6fbe1 WatchSource:0}: Error finding container 20010e424cf73a85b51e890727fab061579eb6a2e60494de6db7e6906ee6fbe1: Status 404 returned error can't find the container with id 20010e424cf73a85b51e890727fab061579eb6a2e60494de6db7e6906ee6fbe1
Sep 16 19:55:29.858591 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.858554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" event={"ID":"31ea3211-5bc5-450e-ae80-6f0d48f839ae","Type":"ContainerStarted","Data":"acf0f08fb8ab03c9a7524ece5fdd88e49c2b1852cf50c853a512bddc11092738"}
Sep 16 19:55:29.858757 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:29.858616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" event={"ID":"31ea3211-5bc5-450e-ae80-6f0d48f839ae","Type":"ContainerStarted","Data":"20010e424cf73a85b51e890727fab061579eb6a2e60494de6db7e6906ee6fbe1"}
Sep 16 19:55:32.875486 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:32.875452    2583 generic.go:356] "Generic (PLEG): container finished" podID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerID="acf0f08fb8ab03c9a7524ece5fdd88e49c2b1852cf50c853a512bddc11092738" exitCode=0
Sep 16 19:55:32.875828 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:32.875536    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" event={"ID":"31ea3211-5bc5-450e-ae80-6f0d48f839ae","Type":"ContainerDied","Data":"acf0f08fb8ab03c9a7524ece5fdd88e49c2b1852cf50c853a512bddc11092738"}
Sep 16 19:55:33.883325 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:33.883293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" event={"ID":"31ea3211-5bc5-450e-ae80-6f0d48f839ae","Type":"ContainerStarted","Data":"f9c02b7d1b355be42789baad9eba8a4c8ae210deef18649235bf96a204edfddf"}
Sep 16 19:55:33.907683 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:33.907635    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podStartSLOduration=2.273890326 podStartE2EDuration="4.907619093s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.633728767s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:55:33.907205444 +0000 UTC m=+3011.033009833" watchObservedRunningTime="2026-09-16 19:55:33.907619093 +0000 UTC m=+3011.033423480"
Sep 16 19:55:34.394743 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:34.394705    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:39.394847 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:39.394800    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:55:39.396104 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:39.396075    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:55:49.395109 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:49.395067    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:55:59.395087 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:55:59.395045    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:56:09.120397 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:09.120355    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"]
Sep 16 19:56:09.394648 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:09.394533    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:56:15.455276 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:15.455174    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" podUID="3250a78a-96a9-41cb-bff9-61dd1427810d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:19.395196 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:19.395150    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:56:24.145275 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:24.145205    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" podUID="3250a78a-96a9-41cb-bff9-61dd1427810d" containerName="main" containerID="cri-o://1154e3db349db2a0f5cc580b0f7c6ff31fa5fa8e65a3e91ba2602d8a5f35f022" gracePeriod=45
Sep 16 19:56:25.455245 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:25.455203    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" podUID="3250a78a-96a9-41cb-bff9-61dd1427810d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:29.395263 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:29.395220    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:56:35.455193 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:35.455155    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" podUID="3250a78a-96a9-41cb-bff9-61dd1427810d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:35.455592 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:35.455252    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:56:39.395038 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:39.394988    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:56:45.454721 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:45.454676    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" podUID="3250a78a-96a9-41cb-bff9-61dd1427810d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:49.404932 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:49.404897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:56:49.414160 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:49.414125    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:56:54.283528 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:54.283492    2583 generic.go:356] "Generic (PLEG): container finished" podID="3250a78a-96a9-41cb-bff9-61dd1427810d" containerID="1154e3db349db2a0f5cc580b0f7c6ff31fa5fa8e65a3e91ba2602d8a5f35f022" exitCode=0
Sep 16 19:56:54.283871 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:54.283550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" event={"ID":"3250a78a-96a9-41cb-bff9-61dd1427810d","Type":"ContainerDied","Data":"1154e3db349db2a0f5cc580b0f7c6ff31fa5fa8e65a3e91ba2602d8a5f35f022"}
Sep 16 19:56:54.402159 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:54.402135    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:56:54.422292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:54.422214    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250a78a-96a9-41cb-bff9-61dd1427810d-tls-certs\") pod \"3250a78a-96a9-41cb-bff9-61dd1427810d\" (UID: \"3250a78a-96a9-41cb-bff9-61dd1427810d\") "
Sep 16 19:56:54.422292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:54.422280    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf2wl\" (UniqueName: \"kubernetes.io/projected/3250a78a-96a9-41cb-bff9-61dd1427810d-kube-api-access-zf2wl\") pod \"3250a78a-96a9-41cb-bff9-61dd1427810d\" (UID: \"3250a78a-96a9-41cb-bff9-61dd1427810d\") "
Sep 16 19:56:54.424932 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:54.424902    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3250a78a-96a9-41cb-bff9-61dd1427810d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3250a78a-96a9-41cb-bff9-61dd1427810d" (UID: "3250a78a-96a9-41cb-bff9-61dd1427810d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:54.425175 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:54.425149    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250a78a-96a9-41cb-bff9-61dd1427810d-kube-api-access-zf2wl" (OuterVolumeSpecName: "kube-api-access-zf2wl") pod "3250a78a-96a9-41cb-bff9-61dd1427810d" (UID: "3250a78a-96a9-41cb-bff9-61dd1427810d"). InnerVolumeSpecName "kube-api-access-zf2wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:54.523987 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:54.523952    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250a78a-96a9-41cb-bff9-61dd1427810d-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:56:54.523987 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:54.523981    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf2wl\" (UniqueName: \"kubernetes.io/projected/3250a78a-96a9-41cb-bff9-61dd1427810d-kube-api-access-zf2wl\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:56:55.289940 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:55.289908    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"
Sep 16 19:56:55.290461 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:55.289951    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2" event={"ID":"3250a78a-96a9-41cb-bff9-61dd1427810d","Type":"ContainerDied","Data":"f0dd725e8ef76adeea13d2fdf7d2d54b2f6c3e950920f05046ceb0365c195980"}
Sep 16 19:56:55.290461 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:55.290022    2583 scope.go:117] "RemoveContainer" containerID="1154e3db349db2a0f5cc580b0f7c6ff31fa5fa8e65a3e91ba2602d8a5f35f022"
Sep 16 19:56:55.312896 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:55.312873    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"]
Sep 16 19:56:55.318818 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:55.318793    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schev7tv2"]
Sep 16 19:56:55.412050 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:56:55.412023    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3250a78a-96a9-41cb-bff9-61dd1427810d" path="/var/lib/kubelet/pods/3250a78a-96a9-41cb-bff9-61dd1427810d/volumes"
Sep 16 19:57:40.924260 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.924212    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"]
Sep 16 19:57:40.924813 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.924793    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3250a78a-96a9-41cb-bff9-61dd1427810d" containerName="main"
Sep 16 19:57:40.924860 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.924818    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3250a78a-96a9-41cb-bff9-61dd1427810d" containerName="main"
Sep 16 19:57:40.924906 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.924895    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3250a78a-96a9-41cb-bff9-61dd1427810d" containerName="main"
Sep 16 19:57:40.928157 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.928137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:57:40.932684 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.932657    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:57:40.932823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.932657    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-hmxg7\""
Sep 16 19:57:40.932823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.932696    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:57:40.932823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.932734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:57:40.932823 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.932813    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-gslfn\""
Sep 16 19:57:40.939108 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:40.939087    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"]
Sep 16 19:57:41.026730 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:41.026698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgkx\" (UniqueName: \"kubernetes.io/projected/48ec502c-e16d-4b4d-9572-4148e26d567e-kube-api-access-bmgkx\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb\" (UID: \"48ec502c-e16d-4b4d-9572-4148e26d567e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:57:41.026873 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:41.026751    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec502c-e16d-4b4d-9572-4148e26d567e-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb\" (UID: \"48ec502c-e16d-4b4d-9572-4148e26d567e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:57:41.128148 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:41.128115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgkx\" (UniqueName: \"kubernetes.io/projected/48ec502c-e16d-4b4d-9572-4148e26d567e-kube-api-access-bmgkx\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb\" (UID: \"48ec502c-e16d-4b4d-9572-4148e26d567e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:57:41.128281 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:41.128166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec502c-e16d-4b4d-9572-4148e26d567e-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb\" (UID: \"48ec502c-e16d-4b4d-9572-4148e26d567e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:57:41.130859 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:41.130840    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec502c-e16d-4b4d-9572-4148e26d567e-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb\" (UID: \"48ec502c-e16d-4b4d-9572-4148e26d567e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:57:41.137097 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:41.137077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgkx\" (UniqueName: \"kubernetes.io/projected/48ec502c-e16d-4b4d-9572-4148e26d567e-kube-api-access-bmgkx\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb\" (UID: \"48ec502c-e16d-4b4d-9572-4148e26d567e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:57:41.239342 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:41.239273    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:57:41.514292 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:41.514269    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"]
Sep 16 19:57:41.516291 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:57:41.516266    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ec502c_e16d_4b4d_9572_4148e26d567e.slice/crio-96f095c11c331122142c1b7c528fb762762874d23b464f71304eb385e3f95999 WatchSource:0}: Error finding container 96f095c11c331122142c1b7c528fb762762874d23b464f71304eb385e3f95999: Status 404 returned error can't find the container with id 96f095c11c331122142c1b7c528fb762762874d23b464f71304eb385e3f95999
Sep 16 19:57:42.522161 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:42.522126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" event={"ID":"48ec502c-e16d-4b4d-9572-4148e26d567e","Type":"ContainerStarted","Data":"a89d99068214b941696c1503c85a10ad4f7a0e5d2ecfe98b7e9a84691c579cfb"}
Sep 16 19:57:42.522563 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:42.522181    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" event={"ID":"48ec502c-e16d-4b4d-9572-4148e26d567e","Type":"ContainerStarted","Data":"96f095c11c331122142c1b7c528fb762762874d23b464f71304eb385e3f95999"}
Sep 16 19:57:42.522563 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:42.522217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:57:42.539618 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:42.539574    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" podStartSLOduration=2.539558299 podStartE2EDuration="2.539558299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:42.538443057 +0000 UTC m=+3139.664247447" watchObservedRunningTime="2026-09-16 19:57:42.539558299 +0000 UTC m=+3139.665362724"
Sep 16 19:57:47.979456 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:57:47.979359    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"]
Sep 16 19:58:03.003050 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.002942    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main" containerID="cri-o://f9c02b7d1b355be42789baad9eba8a4c8ae210deef18649235bf96a204edfddf" gracePeriod=45
Sep 16 19:58:03.405832 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.405787    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 16 19:58:03.622559 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.622524    2583 generic.go:356] "Generic (PLEG): container finished" podID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerID="f9c02b7d1b355be42789baad9eba8a4c8ae210deef18649235bf96a204edfddf" exitCode=0
Sep 16 19:58:03.622737 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.622598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" event={"ID":"31ea3211-5bc5-450e-ae80-6f0d48f839ae","Type":"ContainerDied","Data":"f9c02b7d1b355be42789baad9eba8a4c8ae210deef18649235bf96a204edfddf"}
Sep 16 19:58:03.653361 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.653333    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:58:03.704754 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.704699    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6wht\" (UniqueName: \"kubernetes.io/projected/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kube-api-access-j6wht\") pod \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") "
Sep 16 19:58:03.704754 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.704731    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-dshm\") pod \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") "
Sep 16 19:58:03.704961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.704767    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea3211-5bc5-450e-ae80-6f0d48f839ae-cabundle-cert\") pod \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") "
Sep 16 19:58:03.704961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.704900    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-home\") pod \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") "
Sep 16 19:58:03.704961 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.704949    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tls-certs\") pod \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") "
Sep 16 19:58:03.705171 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.705019    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kserve-provision-location\") pod \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") "
Sep 16 19:58:03.705171 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.705052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tmp-dir\") pod \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") "
Sep 16 19:58:03.705171 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.705089    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-model-cache\") pod \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\" (UID: \"31ea3211-5bc5-450e-ae80-6f0d48f839ae\") "
Sep 16 19:58:03.705440 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.705308    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-home" (OuterVolumeSpecName: "home") pod "31ea3211-5bc5-450e-ae80-6f0d48f839ae" (UID: "31ea3211-5bc5-450e-ae80-6f0d48f839ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:58:03.705588 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.705367    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ea3211-5bc5-450e-ae80-6f0d48f839ae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31ea3211-5bc5-450e-ae80-6f0d48f839ae" (UID: "31ea3211-5bc5-450e-ae80-6f0d48f839ae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:58:03.705716 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.705648    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "31ea3211-5bc5-450e-ae80-6f0d48f839ae" (UID: "31ea3211-5bc5-450e-ae80-6f0d48f839ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:58:03.707450 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.707394    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kube-api-access-j6wht" (OuterVolumeSpecName: "kube-api-access-j6wht") pod "31ea3211-5bc5-450e-ae80-6f0d48f839ae" (UID: "31ea3211-5bc5-450e-ae80-6f0d48f839ae"). InnerVolumeSpecName "kube-api-access-j6wht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:58:03.707549 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.707527    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-dshm" (OuterVolumeSpecName: "dshm") pod "31ea3211-5bc5-450e-ae80-6f0d48f839ae" (UID: "31ea3211-5bc5-450e-ae80-6f0d48f839ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:58:03.707742 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.707713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31ea3211-5bc5-450e-ae80-6f0d48f839ae" (UID: "31ea3211-5bc5-450e-ae80-6f0d48f839ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:58:03.722009 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.721984    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31ea3211-5bc5-450e-ae80-6f0d48f839ae" (UID: "31ea3211-5bc5-450e-ae80-6f0d48f839ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:58:03.728031 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.728009    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31ea3211-5bc5-450e-ae80-6f0d48f839ae" (UID: "31ea3211-5bc5-450e-ae80-6f0d48f839ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:58:03.806300 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.806276    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6wht\" (UniqueName: \"kubernetes.io/projected/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kube-api-access-j6wht\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:58:03.806390 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.806303    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:58:03.806390 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.806317    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea3211-5bc5-450e-ae80-6f0d48f839ae-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:58:03.806390 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.806326    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:58:03.806390 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.806333    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:58:03.806390 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.806340    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:58:03.806390 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.806367    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:58:03.806390 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:03.806380    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea3211-5bc5-450e-ae80-6f0d48f839ae-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 19:58:04.629333 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:04.629297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf" event={"ID":"31ea3211-5bc5-450e-ae80-6f0d48f839ae","Type":"ContainerDied","Data":"20010e424cf73a85b51e890727fab061579eb6a2e60494de6db7e6906ee6fbe1"}
Sep 16 19:58:04.629861 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:04.629353    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"
Sep 16 19:58:04.629861 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:04.629364    2583 scope.go:117] "RemoveContainer" containerID="f9c02b7d1b355be42789baad9eba8a4c8ae210deef18649235bf96a204edfddf"
Sep 16 19:58:04.639797 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:04.639773    2583 scope.go:117] "RemoveContainer" containerID="acf0f08fb8ab03c9a7524ece5fdd88e49c2b1852cf50c853a512bddc11092738"
Sep 16 19:58:04.653604 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:04.653579    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"]
Sep 16 19:58:04.657299 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:04.657275    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-658d774dd7-jnqcf"]
Sep 16 19:58:05.411814 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:05.411780    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" path="/var/lib/kubelet/pods/31ea3211-5bc5-450e-ae80-6f0d48f839ae/volumes"
Sep 16 19:58:13.529100 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:13.529070    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 19:58:53.574718 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.574686    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"]
Sep 16 19:58:53.577084 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.575014    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="storage-initializer"
Sep 16 19:58:53.577084 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.575026    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="storage-initializer"
Sep 16 19:58:53.577084 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.575046    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main"
Sep 16 19:58:53.577084 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.575052    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main"
Sep 16 19:58:53.577084 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.575108    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ea3211-5bc5-450e-ae80-6f0d48f839ae" containerName="main"
Sep 16 19:58:53.577885 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.577868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.581065 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.581045    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 19:58:53.581197 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.581062    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:58:53.581197 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.581095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 19:58:53.581197 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.581108    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-zx49k\""
Sep 16 19:58:53.581197 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.581159    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 19:58:53.581450 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.581356    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 19:58:53.591532 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.591511    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"]
Sep 16 19:58:53.679236 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.679212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.679337 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.679248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.679337 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.679274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.679468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.679333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74cg7\" (UniqueName: \"kubernetes.io/projected/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kube-api-access-74cg7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.679468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.679359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.679468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.679385    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.679468 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.679452    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.679625 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.679486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10c23ebd-6f74-4c71-8c4d-b790857b0e38-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.780887 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.780856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.780897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.780920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74cg7\" (UniqueName: \"kubernetes.io/projected/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kube-api-access-74cg7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.780954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.780979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781070 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.781035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.781076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10c23ebd-6f74-4c71-8c4d-b790857b0e38-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781350 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.781124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781500 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.781380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781500 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.781390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.781500 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.781470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.782027 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.782003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.782237 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.782204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10c23ebd-6f74-4c71-8c4d-b790857b0e38-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.783576 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.783550    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.783969 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.783949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.795124 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.795105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74cg7\" (UniqueName: \"kubernetes.io/projected/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kube-api-access-74cg7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:53.888217 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:53.888195    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:54.179134 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:54.179105    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"]
Sep 16 19:58:54.180946 ip-10-0-139-148 kubenswrapper[2583]: W0916 19:58:54.180918    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c23ebd_6f74_4c71_8c4d_b790857b0e38.slice/crio-dc57f5eab333464f47ef5bb7dd5a6f66ba355f36693211df6d312f4f0e0d04a8 WatchSource:0}: Error finding container dc57f5eab333464f47ef5bb7dd5a6f66ba355f36693211df6d312f4f0e0d04a8: Status 404 returned error can't find the container with id dc57f5eab333464f47ef5bb7dd5a6f66ba355f36693211df6d312f4f0e0d04a8
Sep 16 19:58:54.861121 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:54.861073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" event={"ID":"10c23ebd-6f74-4c71-8c4d-b790857b0e38","Type":"ContainerStarted","Data":"8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89"}
Sep 16 19:58:54.861621 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:54.861149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" event={"ID":"10c23ebd-6f74-4c71-8c4d-b790857b0e38","Type":"ContainerStarted","Data":"dc57f5eab333464f47ef5bb7dd5a6f66ba355f36693211df6d312f4f0e0d04a8"}
Sep 16 19:58:57.875694 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:57.875654    2583 generic.go:356] "Generic (PLEG): container finished" podID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerID="8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89" exitCode=0
Sep 16 19:58:57.876172 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:57.875698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" event={"ID":"10c23ebd-6f74-4c71-8c4d-b790857b0e38","Type":"ContainerDied","Data":"8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89"}
Sep 16 19:58:58.882709 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:58.882674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" event={"ID":"10c23ebd-6f74-4c71-8c4d-b790857b0e38","Type":"ContainerStarted","Data":"cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853"}
Sep 16 19:58:58.888642 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:58.888613    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:58:58.909675 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:58:58.909620    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podStartSLOduration=3.16877997 podStartE2EDuration="5.909604045s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.740824075s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:58.907334462 +0000 UTC m=+3216.033138850" watchObservedRunningTime="2026-09-16 19:58:58.909604045 +0000 UTC m=+3216.035408433"
Sep 16 19:59:03.888317 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:59:03.888285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 19:59:03.889842 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:59:03.889813    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:59:13.888683 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:59:13.888646    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:59:23.889307 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:59:23.889268    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:59:33.889098 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:59:33.889050    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:59:43.889345 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:59:43.889305    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:59:53.889692 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:59:53.889643    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 19:59:56.116635 ip-10-0-139-148 kubenswrapper[2583]: I0916 19:59:56.116601    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"]
Sep 16 20:00:03.528772 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:03.528735    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" podUID="48ec502c-e16d-4b4d-9572-4148e26d567e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:03.889187 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:03.889152    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 20:00:11.137007 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:11.136954    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" podUID="48ec502c-e16d-4b4d-9572-4148e26d567e" containerName="main" containerID="cri-o://a89d99068214b941696c1503c85a10ad4f7a0e5d2ecfe98b7e9a84691c579cfb" gracePeriod=45
Sep 16 20:00:13.529473 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:13.529428    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" podUID="48ec502c-e16d-4b4d-9572-4148e26d567e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:13.911877 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:13.911843    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 20:00:13.920678 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:13.920650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 20:00:15.731513 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:15.731477    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"]
Sep 16 20:00:23.529466 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:23.529403    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" podUID="48ec502c-e16d-4b4d-9572-4148e26d567e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:23.529869 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:23.529553    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 20:00:27.000645 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:27.000617    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:00:27.006861 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:27.006838    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:00:27.007944 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:27.007924    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:00:27.013794 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:27.013775    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:00:30.754776 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:30.754719    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main" containerID="cri-o://cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853" gracePeriod=45
Sep 16 20:00:30.913076 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:30.913038    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 20:00:31.297663 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.297640    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 20:00:31.336078 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.336051    2583 generic.go:356] "Generic (PLEG): container finished" podID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerID="cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853" exitCode=0
Sep 16 20:00:31.336200 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.336118    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"
Sep 16 20:00:31.336200 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.336135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" event={"ID":"10c23ebd-6f74-4c71-8c4d-b790857b0e38","Type":"ContainerDied","Data":"cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853"}
Sep 16 20:00:31.336320 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.336198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn" event={"ID":"10c23ebd-6f74-4c71-8c4d-b790857b0e38","Type":"ContainerDied","Data":"dc57f5eab333464f47ef5bb7dd5a6f66ba355f36693211df6d312f4f0e0d04a8"}
Sep 16 20:00:31.336320 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.336230    2583 scope.go:117] "RemoveContainer" containerID="cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853"
Sep 16 20:00:31.346125 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.346108    2583 scope.go:117] "RemoveContainer" containerID="8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89"
Sep 16 20:00:31.355663 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.355646    2583 scope.go:117] "RemoveContainer" containerID="cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853"
Sep 16 20:00:31.355973 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:00:31.355950    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853\": container with ID starting with cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853 not found: ID does not exist" containerID="cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853"
Sep 16 20:00:31.356031 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.355985    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853"} err="failed to get container status \"cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853\": rpc error: code = NotFound desc = could not find container \"cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853\": container with ID starting with cd188513966b8990faac683a0c985768a07395379c138fd6fbaccacb595cf853 not found: ID does not exist"
Sep 16 20:00:31.356031 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.356011    2583 scope.go:117] "RemoveContainer" containerID="8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89"
Sep 16 20:00:31.356293 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:00:31.356274    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89\": container with ID starting with 8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89 not found: ID does not exist" containerID="8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89"
Sep 16 20:00:31.356362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.356302    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89"} err="failed to get container status \"8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89\": rpc error: code = NotFound desc = could not find container \"8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89\": container with ID starting with 8accd0af245a6498c1774b4ba2308a6b1f901198e086baeb3432d1cede890f89 not found: ID does not exist"
Sep 16 20:00:31.404763 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.404740    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tmp-dir\") pod \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") "
Sep 16 20:00:31.404878 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.404788    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10c23ebd-6f74-4c71-8c4d-b790857b0e38-cabundle-cert\") pod \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") "
Sep 16 20:00:31.404878 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.404835    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kserve-provision-location\") pod \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") "
Sep 16 20:00:31.404878 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.404863    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tls-certs\") pod \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") "
Sep 16 20:00:31.405040 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.404931    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74cg7\" (UniqueName: \"kubernetes.io/projected/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kube-api-access-74cg7\") pod \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") "
Sep 16 20:00:31.405040 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.404985    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-home\") pod \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") "
Sep 16 20:00:31.405150 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.405062    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-dshm\") pod \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") "
Sep 16 20:00:31.405150 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.405113    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-model-cache\") pod \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\" (UID: \"10c23ebd-6f74-4c71-8c4d-b790857b0e38\") "
Sep 16 20:00:31.405255 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.405212    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c23ebd-6f74-4c71-8c4d-b790857b0e38-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "10c23ebd-6f74-4c71-8c4d-b790857b0e38" (UID: "10c23ebd-6f74-4c71-8c4d-b790857b0e38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:00:31.405255 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.405244    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10c23ebd-6f74-4c71-8c4d-b790857b0e38" (UID: "10c23ebd-6f74-4c71-8c4d-b790857b0e38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:31.405380 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.405354    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-home" (OuterVolumeSpecName: "home") pod "10c23ebd-6f74-4c71-8c4d-b790857b0e38" (UID: "10c23ebd-6f74-4c71-8c4d-b790857b0e38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:31.405472 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.405460    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10c23ebd-6f74-4c71-8c4d-b790857b0e38-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.405529 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.405481    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.405529 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.405503    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-model-cache" (OuterVolumeSpecName: "model-cache") pod "10c23ebd-6f74-4c71-8c4d-b790857b0e38" (UID: "10c23ebd-6f74-4c71-8c4d-b790857b0e38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:31.407353 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.407326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10c23ebd-6f74-4c71-8c4d-b790857b0e38" (UID: "10c23ebd-6f74-4c71-8c4d-b790857b0e38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:31.407577 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.407561    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-dshm" (OuterVolumeSpecName: "dshm") pod "10c23ebd-6f74-4c71-8c4d-b790857b0e38" (UID: "10c23ebd-6f74-4c71-8c4d-b790857b0e38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:31.407729 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.407710    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kube-api-access-74cg7" (OuterVolumeSpecName: "kube-api-access-74cg7") pod "10c23ebd-6f74-4c71-8c4d-b790857b0e38" (UID: "10c23ebd-6f74-4c71-8c4d-b790857b0e38"). InnerVolumeSpecName "kube-api-access-74cg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:31.428356 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.428331    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10c23ebd-6f74-4c71-8c4d-b790857b0e38" (UID: "10c23ebd-6f74-4c71-8c4d-b790857b0e38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:31.506063 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.506036    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.506063 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.506060    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.506202 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.506073    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.506202 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.506089    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.506202 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.506102    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c23ebd-6f74-4c71-8c4d-b790857b0e38-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.506202 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.506114    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74cg7\" (UniqueName: \"kubernetes.io/projected/10c23ebd-6f74-4c71-8c4d-b790857b0e38-kube-api-access-74cg7\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:31.662391 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.662310    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"]
Sep 16 20:00:31.666623 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:31.666601    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67cf68d5cd-ntptn"]
Sep 16 20:00:33.411368 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:33.411328    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" path="/var/lib/kubelet/pods/10c23ebd-6f74-4c71-8c4d-b790857b0e38/volumes"
Sep 16 20:00:33.528917 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:33.528885    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" podUID="48ec502c-e16d-4b4d-9572-4148e26d567e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:41.386118 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.386088    2583 generic.go:356] "Generic (PLEG): container finished" podID="48ec502c-e16d-4b4d-9572-4148e26d567e" containerID="a89d99068214b941696c1503c85a10ad4f7a0e5d2ecfe98b7e9a84691c579cfb" exitCode=0
Sep 16 20:00:41.386460 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.386162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" event={"ID":"48ec502c-e16d-4b4d-9572-4148e26d567e","Type":"ContainerDied","Data":"a89d99068214b941696c1503c85a10ad4f7a0e5d2ecfe98b7e9a84691c579cfb"}
Sep 16 20:00:41.386460 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.386219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb" event={"ID":"48ec502c-e16d-4b4d-9572-4148e26d567e","Type":"ContainerDied","Data":"96f095c11c331122142c1b7c528fb762762874d23b464f71304eb385e3f95999"}
Sep 16 20:00:41.386460 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.386238    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f095c11c331122142c1b7c528fb762762874d23b464f71304eb385e3f95999"
Sep 16 20:00:41.394170 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.394155    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 20:00:41.580138 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.580073    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec502c-e16d-4b4d-9572-4148e26d567e-tls-certs\") pod \"48ec502c-e16d-4b4d-9572-4148e26d567e\" (UID: \"48ec502c-e16d-4b4d-9572-4148e26d567e\") "
Sep 16 20:00:41.580138 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.580111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmgkx\" (UniqueName: \"kubernetes.io/projected/48ec502c-e16d-4b4d-9572-4148e26d567e-kube-api-access-bmgkx\") pod \"48ec502c-e16d-4b4d-9572-4148e26d567e\" (UID: \"48ec502c-e16d-4b4d-9572-4148e26d567e\") "
Sep 16 20:00:41.582493 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.582458    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ec502c-e16d-4b4d-9572-4148e26d567e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48ec502c-e16d-4b4d-9572-4148e26d567e" (UID: "48ec502c-e16d-4b4d-9572-4148e26d567e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:41.582595 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.582489    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ec502c-e16d-4b4d-9572-4148e26d567e-kube-api-access-bmgkx" (OuterVolumeSpecName: "kube-api-access-bmgkx") pod "48ec502c-e16d-4b4d-9572-4148e26d567e" (UID: "48ec502c-e16d-4b4d-9572-4148e26d567e"). InnerVolumeSpecName "kube-api-access-bmgkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:41.680728 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.680700    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48ec502c-e16d-4b4d-9572-4148e26d567e-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:41.680728 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:41.680724    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmgkx\" (UniqueName: \"kubernetes.io/projected/48ec502c-e16d-4b4d-9572-4148e26d567e-kube-api-access-bmgkx\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:00:42.391742 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:42.391703    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"
Sep 16 20:00:42.417147 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:42.417117    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"]
Sep 16 20:00:42.422878 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:42.422853    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7bzmb"]
Sep 16 20:00:43.411701 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:43.411664    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ec502c-e16d-4b4d-9572-4148e26d567e" path="/var/lib/kubelet/pods/48ec502c-e16d-4b4d-9572-4148e26d567e/volumes"
Sep 16 20:00:54.394626 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.394535    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"]
Sep 16 20:00:54.395174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.394850    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="storage-initializer"
Sep 16 20:00:54.395174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.394861    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="storage-initializer"
Sep 16 20:00:54.395174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.394877    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main"
Sep 16 20:00:54.395174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.394882    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main"
Sep 16 20:00:54.395174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.394896    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ec502c-e16d-4b4d-9572-4148e26d567e" containerName="main"
Sep 16 20:00:54.395174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.394901    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ec502c-e16d-4b4d-9572-4148e26d567e" containerName="main"
Sep 16 20:00:54.395174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.394957    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ec502c-e16d-4b4d-9572-4148e26d567e" containerName="main"
Sep 16 20:00:54.395174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.394966    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c23ebd-6f74-4c71-8c4d-b790857b0e38" containerName="main"
Sep 16 20:00:54.398042 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.398022    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:00:54.402277 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.402250    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 20:00:54.402454 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.402295    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-j7nb9\""
Sep 16 20:00:54.402454 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.402295    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 20:00:54.402454 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.402390    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-p4fbd\""
Sep 16 20:00:54.402658 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.402609    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 20:00:54.410029 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.410009    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"]
Sep 16 20:00:54.473031 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.473005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv928\" (UniqueName: \"kubernetes.io/projected/aa23884c-d923-44e7-b066-4887786c2376-kube-api-access-pv928\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b776785922\" (UID: \"aa23884c-d923-44e7-b066-4887786c2376\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:00:54.473146 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.473066    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa23884c-d923-44e7-b066-4887786c2376-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b776785922\" (UID: \"aa23884c-d923-44e7-b066-4887786c2376\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:00:54.574133 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.574103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv928\" (UniqueName: \"kubernetes.io/projected/aa23884c-d923-44e7-b066-4887786c2376-kube-api-access-pv928\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b776785922\" (UID: \"aa23884c-d923-44e7-b066-4887786c2376\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:00:54.574313 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.574178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa23884c-d923-44e7-b066-4887786c2376-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b776785922\" (UID: \"aa23884c-d923-44e7-b066-4887786c2376\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:00:54.576814 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.576788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa23884c-d923-44e7-b066-4887786c2376-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b776785922\" (UID: \"aa23884c-d923-44e7-b066-4887786c2376\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:00:54.583236 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.583215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv928\" (UniqueName: \"kubernetes.io/projected/aa23884c-d923-44e7-b066-4887786c2376-kube-api-access-pv928\") pod \"custom-route-timeout-test-kserve-router-scheduler-564b776785922\" (UID: \"aa23884c-d923-44e7-b066-4887786c2376\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:00:54.709149 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.709066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:00:54.988511 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:54.988482    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"]
Sep 16 20:00:54.991595 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:00:54.991562    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa23884c_d923_44e7_b066_4887786c2376.slice/crio-d826c3d9dbc9ccc3e9ea31126d056aaeba9d00d45eb789d57b896ca4e89efa05 WatchSource:0}: Error finding container d826c3d9dbc9ccc3e9ea31126d056aaeba9d00d45eb789d57b896ca4e89efa05: Status 404 returned error can't find the container with id d826c3d9dbc9ccc3e9ea31126d056aaeba9d00d45eb789d57b896ca4e89efa05
Sep 16 20:00:55.448063 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:55.448029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" event={"ID":"aa23884c-d923-44e7-b066-4887786c2376","Type":"ContainerStarted","Data":"91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede"}
Sep 16 20:00:55.448460 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:55.448155    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:00:55.448460 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:55.448193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" event={"ID":"aa23884c-d923-44e7-b066-4887786c2376","Type":"ContainerStarted","Data":"d826c3d9dbc9ccc3e9ea31126d056aaeba9d00d45eb789d57b896ca4e89efa05"}
Sep 16 20:00:55.466569 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:00:55.466527    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" podStartSLOduration=1.466514173 podStartE2EDuration="1.466514173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:00:55.465356648 +0000 UTC m=+3332.591161037" watchObservedRunningTime="2026-09-16 20:00:55.466514173 +0000 UTC m=+3332.592318560"
Sep 16 20:01:15.958352 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:15.958314    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"]
Sep 16 20:01:15.961887 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:15.961867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:15.968798 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:15.968773    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 16 20:01:15.968929 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:15.968842    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-2fngv\""
Sep 16 20:01:15.969347 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:15.969324    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 20:01:15.969852 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:15.969611    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 20:01:15.970949 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:15.970934    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 20:01:15.986459 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:15.986436    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"]
Sep 16 20:01:16.040856 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.040829    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-home\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.040950 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.040863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-dshm\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.040950 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.040915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.040950 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.040941    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.041091 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.040961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.041091 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.040981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qsp\" (UniqueName: \"kubernetes.io/projected/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kube-api-access-q4qsp\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.041091 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.041028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-model-cache\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.141697 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.141673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-home\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.141814 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.141705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-dshm\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.141814 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.141743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.141814 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.141762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.141814 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.141784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.141814 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.141801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qsp\" (UniqueName: \"kubernetes.io/projected/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kube-api-access-q4qsp\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.142064 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.141818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-model-cache\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.142159 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.142130    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-home\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.142352 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.142216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.142466 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.142235    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.142466 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.142272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-model-cache\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.144271 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.144241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-dshm\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.144539 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.144519    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.156456 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.156434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qsp\" (UniqueName: \"kubernetes.io/projected/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kube-api-access-q4qsp\") pod \"e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.274297 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.274234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:16.553545 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:16.553517    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"]
Sep 16 20:01:16.555167 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:01:16.555142    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dacfad_6821_4375_92cc_5ea1d1394c9b.slice/crio-5dd7594762d34180ef4cf2e9c24267e0fe2b09c88d7f8abd90379f4c65343f13 WatchSource:0}: Error finding container 5dd7594762d34180ef4cf2e9c24267e0fe2b09c88d7f8abd90379f4c65343f13: Status 404 returned error can't find the container with id 5dd7594762d34180ef4cf2e9c24267e0fe2b09c88d7f8abd90379f4c65343f13
Sep 16 20:01:17.552563 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:17.552525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6" event={"ID":"e2dacfad-6821-4375-92cc-5ea1d1394c9b","Type":"ContainerStarted","Data":"aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae"}
Sep 16 20:01:17.553036 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:17.552590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6" event={"ID":"e2dacfad-6821-4375-92cc-5ea1d1394c9b","Type":"ContainerStarted","Data":"5dd7594762d34180ef4cf2e9c24267e0fe2b09c88d7f8abd90379f4c65343f13"}
Sep 16 20:01:18.654316 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:18.654267    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"]
Sep 16 20:01:18.667180 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:18.667151    2583 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 20:01:18.667371 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:18.667234    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs podName:e2dacfad-6821-4375-92cc-5ea1d1394c9b nodeName:}" failed. No retries permitted until 2026-09-16 20:01:19.167212311 +0000 UTC m=+3356.293016695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs") pod "e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 20:01:19.172537 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:19.172509    2583 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 20:01:19.172704 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:19.172578    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs podName:e2dacfad-6821-4375-92cc-5ea1d1394c9b nodeName:}" failed. No retries permitted until 2026-09-16 20:01:20.17256274 +0000 UTC m=+3357.298367106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs") pod "e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 20:01:19.564547 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:19.564398    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6" podUID="e2dacfad-6821-4375-92cc-5ea1d1394c9b" containerName="storage-initializer" containerID="cri-o://aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae" gracePeriod=60
Sep 16 20:01:20.183746 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:20.183708    2583 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 20:01:20.184133 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:20.183782    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs podName:e2dacfad-6821-4375-92cc-5ea1d1394c9b nodeName:}" failed. No retries permitted until 2026-09-16 20:01:22.18376432 +0000 UTC m=+3359.309568706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs") pod "e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 20:01:20.805891 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.805862    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:20.891556 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891534    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs\") pod \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") "
Sep 16 20:01:20.891702 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891574    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-model-cache\") pod \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") "
Sep 16 20:01:20.891702 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891606    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4qsp\" (UniqueName: \"kubernetes.io/projected/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kube-api-access-q4qsp\") pod \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") "
Sep 16 20:01:20.891702 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891635    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kserve-provision-location\") pod \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") "
Sep 16 20:01:20.891702 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891688    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-home\") pod \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") "
Sep 16 20:01:20.891920 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tmp-dir\") pod \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") "
Sep 16 20:01:20.891920 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891772    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-dshm\") pod \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\" (UID: \"e2dacfad-6821-4375-92cc-5ea1d1394c9b\") "
Sep 16 20:01:20.891920 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891825    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2dacfad-6821-4375-92cc-5ea1d1394c9b" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.892067 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891934    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-home" (OuterVolumeSpecName: "home") pod "e2dacfad-6821-4375-92cc-5ea1d1394c9b" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.892067 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.891998    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2dacfad-6821-4375-92cc-5ea1d1394c9b" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.892178 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.892096    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.892178 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.892108    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.892178 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.892117    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.893866 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.893841    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2dacfad-6821-4375-92cc-5ea1d1394c9b" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:20.894023 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.894007    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-dshm" (OuterVolumeSpecName: "dshm") pod "e2dacfad-6821-4375-92cc-5ea1d1394c9b" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.894171 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.894143    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kube-api-access-q4qsp" (OuterVolumeSpecName: "kube-api-access-q4qsp") pod "e2dacfad-6821-4375-92cc-5ea1d1394c9b" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b"). InnerVolumeSpecName "kube-api-access-q4qsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:20.964804 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.964772    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2dacfad-6821-4375-92cc-5ea1d1394c9b" (UID: "e2dacfad-6821-4375-92cc-5ea1d1394c9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.993000 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.992978    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dacfad-6821-4375-92cc-5ea1d1394c9b-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.993000 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.992997    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4qsp\" (UniqueName: \"kubernetes.io/projected/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kube-api-access-q4qsp\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.993111 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.993009    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.993111 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:20.993017    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dacfad-6821-4375-92cc-5ea1d1394c9b-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:01:21.575606 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:21.575571    2583 generic.go:356] "Generic (PLEG): container finished" podID="e2dacfad-6821-4375-92cc-5ea1d1394c9b" containerID="aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae" exitCode=0
Sep 16 20:01:21.575986 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:21.575622    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"
Sep 16 20:01:21.575986 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:21.575652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6" event={"ID":"e2dacfad-6821-4375-92cc-5ea1d1394c9b","Type":"ContainerDied","Data":"aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae"}
Sep 16 20:01:21.575986 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:21.575710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6" event={"ID":"e2dacfad-6821-4375-92cc-5ea1d1394c9b","Type":"ContainerDied","Data":"5dd7594762d34180ef4cf2e9c24267e0fe2b09c88d7f8abd90379f4c65343f13"}
Sep 16 20:01:21.575986 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:21.575734    2583 scope.go:117] "RemoveContainer" containerID="aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae"
Sep 16 20:01:21.585620 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:21.585600    2583 scope.go:117] "RemoveContainer" containerID="aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae"
Sep 16 20:01:21.585930 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:21.585909    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae\": container with ID starting with aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae not found: ID does not exist" containerID="aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae"
Sep 16 20:01:21.585994 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:21.585939    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae"} err="failed to get container status \"aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae\": rpc error: code = NotFound desc = could not find container \"aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae\": container with ID starting with aee8a278d5505a4f6f2fb2008e00544fe7996305f4699c629b5ba31529736cae not found: ID does not exist"
Sep 16 20:01:21.622267 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:21.622243    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"]
Sep 16 20:01:21.629151 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:21.629125    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5b4b77dfdf-4vhk6"]
Sep 16 20:01:23.412035 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:23.411996    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2dacfad-6821-4375-92cc-5ea1d1394c9b" path="/var/lib/kubelet/pods/e2dacfad-6821-4375-92cc-5ea1d1394c9b/volumes"
Sep 16 20:01:26.454763 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:26.454730    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:01:40.047451 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.047401    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"]
Sep 16 20:01:40.047855 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.047755    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2dacfad-6821-4375-92cc-5ea1d1394c9b" containerName="storage-initializer"
Sep 16 20:01:40.047855 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.047766    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2dacfad-6821-4375-92cc-5ea1d1394c9b" containerName="storage-initializer"
Sep 16 20:01:40.047855 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.047836    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2dacfad-6821-4375-92cc-5ea1d1394c9b" containerName="storage-initializer"
Sep 16 20:01:40.050889 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.050872    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:40.053795 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.053773    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 16 20:01:40.053901 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.053859    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-k2fkz\""
Sep 16 20:01:40.054920 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.054897    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 16 20:01:40.055011 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.054937    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 16 20:01:40.076251 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.076222    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"]
Sep 16 20:01:40.139646 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.139620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be78ef47-db19-46de-953d-95a7fffa422b-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh\" (UID: \"be78ef47-db19-46de-953d-95a7fffa422b\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:40.139766 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.139657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llnlw\" (UniqueName: \"kubernetes.io/projected/be78ef47-db19-46de-953d-95a7fffa422b-kube-api-access-llnlw\") pod \"e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh\" (UID: \"be78ef47-db19-46de-953d-95a7fffa422b\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:40.240488 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.240453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be78ef47-db19-46de-953d-95a7fffa422b-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh\" (UID: \"be78ef47-db19-46de-953d-95a7fffa422b\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:40.240626 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.240511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llnlw\" (UniqueName: \"kubernetes.io/projected/be78ef47-db19-46de-953d-95a7fffa422b-kube-api-access-llnlw\") pod \"e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh\" (UID: \"be78ef47-db19-46de-953d-95a7fffa422b\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:40.240798 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.240782    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be78ef47-db19-46de-953d-95a7fffa422b-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh\" (UID: \"be78ef47-db19-46de-953d-95a7fffa422b\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:40.250080 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.250050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llnlw\" (UniqueName: \"kubernetes.io/projected/be78ef47-db19-46de-953d-95a7fffa422b-kube-api-access-llnlw\") pod \"e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh\" (UID: \"be78ef47-db19-46de-953d-95a7fffa422b\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:40.348076 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.348005    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"]
Sep 16 20:01:40.351452 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.351430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:01:40.355027 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.355003    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-c9bj7\""
Sep 16 20:01:40.355106 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.355003    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 16 20:01:40.360109 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.360087    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"]
Sep 16 20:01:40.361206 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.361189    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:40.442141 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.442103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2\" (UID: \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:01:40.442293 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.442228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgks\" (UniqueName: \"kubernetes.io/projected/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-kube-api-access-jwgks\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2\" (UID: \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:01:40.543535 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.543506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgks\" (UniqueName: \"kubernetes.io/projected/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-kube-api-access-jwgks\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2\" (UID: \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:01:40.543712 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.543578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2\" (UID: \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:01:40.546133 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.546108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2\" (UID: \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:01:40.552014 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.551995    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgks\" (UniqueName: \"kubernetes.io/projected/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-kube-api-access-jwgks\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2\" (UID: \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:01:40.639550 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.639524    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"]
Sep 16 20:01:40.641288 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:01:40.641262    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe78ef47_db19_46de_953d_95a7fffa422b.slice/crio-7dadbacffce09e73ebb58b21f6fea74bb12020db4acd59bd0439d57d7c0e3005 WatchSource:0}: Error finding container 7dadbacffce09e73ebb58b21f6fea74bb12020db4acd59bd0439d57d7c0e3005: Status 404 returned error can't find the container with id 7dadbacffce09e73ebb58b21f6fea74bb12020db4acd59bd0439d57d7c0e3005
Sep 16 20:01:40.663354 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.663332    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:01:40.668645 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.668602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh" event={"ID":"be78ef47-db19-46de-953d-95a7fffa422b","Type":"ContainerStarted","Data":"7dadbacffce09e73ebb58b21f6fea74bb12020db4acd59bd0439d57d7c0e3005"}
Sep 16 20:01:40.949564 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:40.949480    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"]
Sep 16 20:01:40.952804 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:01:40.952774    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23461e5e_e39f_4d5a_bbd9_546eb0bd44fc.slice/crio-dcaddf863559b5c412672001eb95657be09bc01a13ad6334900958a96d3458ca WatchSource:0}: Error finding container dcaddf863559b5c412672001eb95657be09bc01a13ad6334900958a96d3458ca: Status 404 returned error can't find the container with id dcaddf863559b5c412672001eb95657be09bc01a13ad6334900958a96d3458ca
Sep 16 20:01:41.676424 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:41.676366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh" event={"ID":"be78ef47-db19-46de-953d-95a7fffa422b","Type":"ContainerStarted","Data":"2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9"}
Sep 16 20:01:41.678344 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:41.678309    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" event={"ID":"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc","Type":"ContainerStarted","Data":"0e92afc9d061406b718ac5b28630fdbc34fb3b97b211b7d1a43ab8d4c633c85f"}
Sep 16 20:01:41.678503 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:41.678358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" event={"ID":"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc","Type":"ContainerStarted","Data":"dcaddf863559b5c412672001eb95657be09bc01a13ad6334900958a96d3458ca"}
Sep 16 20:01:41.678503 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:41.678455    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:01:41.717598 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:41.717543    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" podStartSLOduration=1.717509741 podStartE2EDuration="1.717509741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:41.716683893 +0000 UTC m=+3378.842488282" watchObservedRunningTime="2026-09-16 20:01:41.717509741 +0000 UTC m=+3378.843314129"
Sep 16 20:01:42.169021 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:42.168988    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"]
Sep 16 20:01:42.175664 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:42.175641    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"]
Sep 16 20:01:42.683527 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:42.683494    2583 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" secret="" err="secret \"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-c9bj7\" not found"
Sep 16 20:01:42.866601 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:42.866569    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:42.866807 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:42.866639    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs podName:23461e5e-e39f-4d5a-bbd9-546eb0bd44fc nodeName:}" failed. No retries permitted until 2026-09-16 20:01:43.36662248 +0000 UTC m=+3380.492426847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" (UID: "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:43.372037 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:43.372003    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:43.372252 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:43.372071    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs podName:23461e5e-e39f-4d5a-bbd9-546eb0bd44fc nodeName:}" failed. No retries permitted until 2026-09-16 20:01:44.372056639 +0000 UTC m=+3381.497861005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" (UID: "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:43.688595 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:43.688283    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh" podUID="be78ef47-db19-46de-953d-95a7fffa422b" containerName="storage-initializer" containerID="cri-o://2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9" gracePeriod=30
Sep 16 20:01:44.382017 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:44.381977    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:44.382188 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:44.382052    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs podName:23461e5e-e39f-4d5a-bbd9-546eb0bd44fc nodeName:}" failed. No retries permitted until 2026-09-16 20:01:46.38203522 +0000 UTC m=+3383.507839586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" (UID: "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:45.445854 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.445826    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:45.491281 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.491260    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llnlw\" (UniqueName: \"kubernetes.io/projected/be78ef47-db19-46de-953d-95a7fffa422b-kube-api-access-llnlw\") pod \"be78ef47-db19-46de-953d-95a7fffa422b\" (UID: \"be78ef47-db19-46de-953d-95a7fffa422b\") "
Sep 16 20:01:45.491400 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.491353    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be78ef47-db19-46de-953d-95a7fffa422b-kserve-provision-location\") pod \"be78ef47-db19-46de-953d-95a7fffa422b\" (UID: \"be78ef47-db19-46de-953d-95a7fffa422b\") "
Sep 16 20:01:45.493553 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.493528    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be78ef47-db19-46de-953d-95a7fffa422b-kube-api-access-llnlw" (OuterVolumeSpecName: "kube-api-access-llnlw") pod "be78ef47-db19-46de-953d-95a7fffa422b" (UID: "be78ef47-db19-46de-953d-95a7fffa422b"). InnerVolumeSpecName "kube-api-access-llnlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:45.562388 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.562330    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be78ef47-db19-46de-953d-95a7fffa422b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be78ef47-db19-46de-953d-95a7fffa422b" (UID: "be78ef47-db19-46de-953d-95a7fffa422b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:45.592002 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.591983    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llnlw\" (UniqueName: \"kubernetes.io/projected/be78ef47-db19-46de-953d-95a7fffa422b-kube-api-access-llnlw\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:01:45.592060 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.592003    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be78ef47-db19-46de-953d-95a7fffa422b-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:01:45.699802 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.699775    2583 generic.go:356] "Generic (PLEG): container finished" podID="be78ef47-db19-46de-953d-95a7fffa422b" containerID="2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9" exitCode=0
Sep 16 20:01:45.699920 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.699833    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"
Sep 16 20:01:45.699920 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.699869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh" event={"ID":"be78ef47-db19-46de-953d-95a7fffa422b","Type":"ContainerDied","Data":"2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9"}
Sep 16 20:01:45.700034 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.699924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh" event={"ID":"be78ef47-db19-46de-953d-95a7fffa422b","Type":"ContainerDied","Data":"7dadbacffce09e73ebb58b21f6fea74bb12020db4acd59bd0439d57d7c0e3005"}
Sep 16 20:01:45.700034 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.699949    2583 scope.go:117] "RemoveContainer" containerID="2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9"
Sep 16 20:01:45.710964 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.710944    2583 scope.go:117] "RemoveContainer" containerID="2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9"
Sep 16 20:01:45.711244 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:45.711215    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9\": container with ID starting with 2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9 not found: ID does not exist" containerID="2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9"
Sep 16 20:01:45.711313 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.711245    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9"} err="failed to get container status \"2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9\": rpc error: code = NotFound desc = could not find container \"2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9\": container with ID starting with 2011bb6bf6313905e4f96b4dfa92bbaae347dfb9b231ff48df3041be7918eef9 not found: ID does not exist"
Sep 16 20:01:45.736057 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.736030    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"]
Sep 16 20:01:45.738164 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:45.738143    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-85b557bcdd-77lwh"]
Sep 16 20:01:46.398019 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:46.397985    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:46.398171 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:46.398053    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs podName:23461e5e-e39f-4d5a-bbd9-546eb0bd44fc nodeName:}" failed. No retries permitted until 2026-09-16 20:01:50.398038938 +0000 UTC m=+3387.523843304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" (UID: "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:47.411757 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:47.411724    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be78ef47-db19-46de-953d-95a7fffa422b" path="/var/lib/kubelet/pods/be78ef47-db19-46de-953d-95a7fffa422b/volumes"
Sep 16 20:01:49.430913 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.430878    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"]
Sep 16 20:01:49.431313 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.431249    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be78ef47-db19-46de-953d-95a7fffa422b" containerName="storage-initializer"
Sep 16 20:01:49.431313 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.431262    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="be78ef47-db19-46de-953d-95a7fffa422b" containerName="storage-initializer"
Sep 16 20:01:49.431397 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.431327    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="be78ef47-db19-46de-953d-95a7fffa422b" containerName="storage-initializer"
Sep 16 20:01:49.435765 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.435743    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:01:49.440802 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.440775    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 20:01:49.440929 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.440803    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 20:01:49.440929 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.440825    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 20:01:49.440929 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.440775    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-frwjw\""
Sep 16 20:01:49.440929 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.440883    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qqkdg\""
Sep 16 20:01:49.447747 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.447724    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"]
Sep 16 20:01:49.521695 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.521661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45mf\" (UniqueName: \"kubernetes.io/projected/2c95f5a1-e41b-449f-816c-7054a683219c-kube-api-access-g45mf\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-zpszs\" (UID: \"2c95f5a1-e41b-449f-816c-7054a683219c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:01:49.521809 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.521759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c95f5a1-e41b-449f-816c-7054a683219c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-zpszs\" (UID: \"2c95f5a1-e41b-449f-816c-7054a683219c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:01:49.622293 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.622262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c95f5a1-e41b-449f-816c-7054a683219c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-zpszs\" (UID: \"2c95f5a1-e41b-449f-816c-7054a683219c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:01:49.622472 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.622357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45mf\" (UniqueName: \"kubernetes.io/projected/2c95f5a1-e41b-449f-816c-7054a683219c-kube-api-access-g45mf\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-zpszs\" (UID: \"2c95f5a1-e41b-449f-816c-7054a683219c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:01:49.624976 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.624951    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c95f5a1-e41b-449f-816c-7054a683219c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-zpszs\" (UID: \"2c95f5a1-e41b-449f-816c-7054a683219c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:01:49.635829 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.635802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45mf\" (UniqueName: \"kubernetes.io/projected/2c95f5a1-e41b-449f-816c-7054a683219c-kube-api-access-g45mf\") pod \"stop-feature-test-kserve-router-scheduler-7468747994-zpszs\" (UID: \"2c95f5a1-e41b-449f-816c-7054a683219c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:01:49.746901 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:49.746840    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:01:50.045627 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:50.045601    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"]
Sep 16 20:01:50.049352 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:01:50.049319    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c95f5a1_e41b_449f_816c_7054a683219c.slice/crio-b5e2414ecb3bd65b6073fb16600a1855fd1d99f7ad100169aed346721186ea1f WatchSource:0}: Error finding container b5e2414ecb3bd65b6073fb16600a1855fd1d99f7ad100169aed346721186ea1f: Status 404 returned error can't find the container with id b5e2414ecb3bd65b6073fb16600a1855fd1d99f7ad100169aed346721186ea1f
Sep 16 20:01:50.431248 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:50.431218    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:50.431637 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:50.431284    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs podName:23461e5e-e39f-4d5a-bbd9-546eb0bd44fc nodeName:}" failed. No retries permitted until 2026-09-16 20:01:58.431267012 +0000 UTC m=+3395.557071398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" (UID: "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:50.725635 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:50.725551    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" event={"ID":"2c95f5a1-e41b-449f-816c-7054a683219c","Type":"ContainerStarted","Data":"55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122"}
Sep 16 20:01:50.725635 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:50.725605    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" event={"ID":"2c95f5a1-e41b-449f-816c-7054a683219c","Type":"ContainerStarted","Data":"b5e2414ecb3bd65b6073fb16600a1855fd1d99f7ad100169aed346721186ea1f"}
Sep 16 20:01:50.725808 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:50.725651    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:01:50.748551 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:50.748504    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" podStartSLOduration=1.74848955 podStartE2EDuration="1.74848955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:50.747584061 +0000 UTC m=+3387.873388454" watchObservedRunningTime="2026-09-16 20:01:50.74848955 +0000 UTC m=+3387.874293938"
Sep 16 20:01:58.498424 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:58.498371    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:58.498824 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:01:58.498458    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs podName:23461e5e-e39f-4d5a-bbd9-546eb0bd44fc nodeName:}" failed. No retries permitted until 2026-09-16 20:02:14.498444023 +0000 UTC m=+3411.624248390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" (UID: "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:01:58.707459 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:01:58.707395    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" podUID="23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" containerName="main" containerID="cri-o://0e92afc9d061406b718ac5b28630fdbc34fb3b97b211b7d1a43ab8d4c633c85f" gracePeriod=45
Sep 16 20:02:13.691598 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:13.691555    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" podUID="23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:14.531008 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:02:14.530969    2583 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:02:14.531193 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:02:14.531047    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs podName:23461e5e-e39f-4d5a-bbd9-546eb0bd44fc nodeName:}" failed. No retries permitted until 2026-09-16 20:02:46.531033377 +0000 UTC m=+3443.656837742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" (UID: "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 20:02:21.732486 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:21.732453    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:02:23.689839 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:23.689799    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" podUID="23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:02:28.914024 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:28.913992    2583 generic.go:356] "Generic (PLEG): container finished" podID="23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" containerID="0e92afc9d061406b718ac5b28630fdbc34fb3b97b211b7d1a43ab8d4c633c85f" exitCode=0
Sep 16 20:02:28.914396 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:28.914065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" event={"ID":"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc","Type":"ContainerDied","Data":"0e92afc9d061406b718ac5b28630fdbc34fb3b97b211b7d1a43ab8d4c633c85f"}
Sep 16 20:02:28.955682 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:28.955662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:02:29.049747 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.049688    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs\") pod \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\" (UID: \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\") "
Sep 16 20:02:29.049844 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.049787    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwgks\" (UniqueName: \"kubernetes.io/projected/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-kube-api-access-jwgks\") pod \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\" (UID: \"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc\") "
Sep 16 20:02:29.051902 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.051868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-kube-api-access-jwgks" (OuterVolumeSpecName: "kube-api-access-jwgks") pod "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" (UID: "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc"). InnerVolumeSpecName "kube-api-access-jwgks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:02:29.052034 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.051935    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" (UID: "23461e5e-e39f-4d5a-bbd9-546eb0bd44fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:02:29.150655 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.150621    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:02:29.150655 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.150653    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwgks\" (UniqueName: \"kubernetes.io/projected/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc-kube-api-access-jwgks\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:02:29.919707 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.919678    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"
Sep 16 20:02:29.919707 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.919683    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2" event={"ID":"23461e5e-e39f-4d5a-bbd9-546eb0bd44fc","Type":"ContainerDied","Data":"dcaddf863559b5c412672001eb95657be09bc01a13ad6334900958a96d3458ca"}
Sep 16 20:02:29.920162 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.919738    2583 scope.go:117] "RemoveContainer" containerID="0e92afc9d061406b718ac5b28630fdbc34fb3b97b211b7d1a43ab8d4c633c85f"
Sep 16 20:02:29.937275 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.937251    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"]
Sep 16 20:02:29.941082 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:29.941061    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7db6d67968sgdk2"]
Sep 16 20:02:31.412493 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:31.412458    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" path="/var/lib/kubelet/pods/23461e5e-e39f-4d5a-bbd9-546eb0bd44fc/volumes"
Sep 16 20:02:58.920589 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:02:58.920553    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"]
Sep 16 20:03:06.454870 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:06.454831    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" podUID="aa23884c-d923-44e7-b066-4887786c2376" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:13.940560 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:13.940486    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" podUID="aa23884c-d923-44e7-b066-4887786c2376" containerName="main" containerID="cri-o://91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede" gracePeriod=45
Sep 16 20:03:16.454322 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:16.454279    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" podUID="aa23884c-d923-44e7-b066-4887786c2376" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:26.454086 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:26.454046    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" podUID="aa23884c-d923-44e7-b066-4887786c2376" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:26.454550 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:26.454153    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:03:36.454137 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:36.454093    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" podUID="aa23884c-d923-44e7-b066-4887786c2376" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:44.193068 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.193038    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:03:44.237341 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.237278    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv928\" (UniqueName: \"kubernetes.io/projected/aa23884c-d923-44e7-b066-4887786c2376-kube-api-access-pv928\") pod \"aa23884c-d923-44e7-b066-4887786c2376\" (UID: \"aa23884c-d923-44e7-b066-4887786c2376\") "
Sep 16 20:03:44.237341 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.237336    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa23884c-d923-44e7-b066-4887786c2376-tls-certs\") pod \"aa23884c-d923-44e7-b066-4887786c2376\" (UID: \"aa23884c-d923-44e7-b066-4887786c2376\") "
Sep 16 20:03:44.239586 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.239558    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa23884c-d923-44e7-b066-4887786c2376-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa23884c-d923-44e7-b066-4887786c2376" (UID: "aa23884c-d923-44e7-b066-4887786c2376"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:44.239693 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.239579    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa23884c-d923-44e7-b066-4887786c2376-kube-api-access-pv928" (OuterVolumeSpecName: "kube-api-access-pv928") pod "aa23884c-d923-44e7-b066-4887786c2376" (UID: "aa23884c-d923-44e7-b066-4887786c2376"). InnerVolumeSpecName "kube-api-access-pv928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:44.279301 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.279271    2583 generic.go:356] "Generic (PLEG): container finished" podID="aa23884c-d923-44e7-b066-4887786c2376" containerID="91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede" exitCode=0
Sep 16 20:03:44.279434 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.279332    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"
Sep 16 20:03:44.279434 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.279358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" event={"ID":"aa23884c-d923-44e7-b066-4887786c2376","Type":"ContainerDied","Data":"91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede"}
Sep 16 20:03:44.279563 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.279443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922" event={"ID":"aa23884c-d923-44e7-b066-4887786c2376","Type":"ContainerDied","Data":"d826c3d9dbc9ccc3e9ea31126d056aaeba9d00d45eb789d57b896ca4e89efa05"}
Sep 16 20:03:44.279563 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.279477    2583 scope.go:117] "RemoveContainer" containerID="91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede"
Sep 16 20:03:44.289563 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.289529    2583 scope.go:117] "RemoveContainer" containerID="91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede"
Sep 16 20:03:44.289797 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:03:44.289774    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede\": container with ID starting with 91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede not found: ID does not exist" containerID="91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede"
Sep 16 20:03:44.289871 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.289810    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede"} err="failed to get container status \"91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede\": rpc error: code = NotFound desc = could not find container \"91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede\": container with ID starting with 91b1b823263d8ff6089fa6ad54611a9b588f62fb5db90c3a55da8a88202a5ede not found: ID does not exist"
Sep 16 20:03:44.301450 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.301421    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"]
Sep 16 20:03:44.306841 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.306819    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-564b776785922"]
Sep 16 20:03:44.338429 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.338373    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv928\" (UniqueName: \"kubernetes.io/projected/aa23884c-d923-44e7-b066-4887786c2376-kube-api-access-pv928\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:03:44.338429 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:44.338405    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa23884c-d923-44e7-b066-4887786c2376-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:03:45.411751 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:45.411712    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa23884c-d923-44e7-b066-4887786c2376" path="/var/lib/kubelet/pods/aa23884c-d923-44e7-b066-4887786c2376/volumes"
Sep 16 20:03:49.259696 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.259669    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"]
Sep 16 20:03:49.260018 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.260005    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa23884c-d923-44e7-b066-4887786c2376" containerName="main"
Sep 16 20:03:49.260018 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.260017    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa23884c-d923-44e7-b066-4887786c2376" containerName="main"
Sep 16 20:03:49.260097 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.260029    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" containerName="main"
Sep 16 20:03:49.260097 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.260034    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" containerName="main"
Sep 16 20:03:49.260097 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.260090    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa23884c-d923-44e7-b066-4887786c2376" containerName="main"
Sep 16 20:03:49.260198 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.260099    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="23461e5e-e39f-4d5a-bbd9-546eb0bd44fc" containerName="main"
Sep 16 20:03:49.264160 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.264142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.267496 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.267472    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 20:03:49.267627 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.267557    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-p96ln\""
Sep 16 20:03:49.267627 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.267478    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 16 20:03:49.267627 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.267472    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 20:03:49.275661 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.275637    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"]
Sep 16 20:03:49.278604 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.278575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jnk\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-kube-api-access-j5jnk\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.278723 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.278624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.278723 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.278689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.278828 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.278743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.278828 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.278776    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.278828 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.278813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.278950 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.278848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.278950 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.278880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.278950 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.278909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.379558 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.379527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.379701 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.379568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.379701 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.379589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.379777 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.379698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.379777 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.379739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.379777 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.379772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.379889 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.379797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.379889 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.379879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jnk\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-kube-api-access-j5jnk\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.379987 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.379937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.380192 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.380168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.380291 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.380215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.380291 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.380249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.380428 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.380306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.380479 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.380449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.382043 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.382022    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.382158 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.382139    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.390755 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.390736    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.391118 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.391097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jnk\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-kube-api-access-j5jnk\") pod \"router-gateway-1-openshift-default-6cc7987576-jb5k8\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.574252 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.574180    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:49.859929 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.859902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"]
Sep 16 20:03:49.862057 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:03:49.862032    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9f7ab4_5e88_414a_b314_cfd9157cf5fe.slice/crio-9ddba1ac4ac173f7f896e9ae1eefdbaa8d480bded762500479c3cf3d82efa730 WatchSource:0}: Error finding container 9ddba1ac4ac173f7f896e9ae1eefdbaa8d480bded762500479c3cf3d82efa730: Status 404 returned error can't find the container with id 9ddba1ac4ac173f7f896e9ae1eefdbaa8d480bded762500479c3cf3d82efa730
Sep 16 20:03:49.864053 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.864023    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:03:49.864153 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.864086    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:03:49.864153 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:49.864114    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:03:50.311232 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:50.311194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" event={"ID":"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe","Type":"ContainerStarted","Data":"13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49"}
Sep 16 20:03:50.311729 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:50.311259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" event={"ID":"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe","Type":"ContainerStarted","Data":"9ddba1ac4ac173f7f896e9ae1eefdbaa8d480bded762500479c3cf3d82efa730"}
Sep 16 20:03:50.337344 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:50.337286    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" podStartSLOduration=1.337263743 podStartE2EDuration="1.337263743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:03:50.334959545 +0000 UTC m=+3507.460763934" watchObservedRunningTime="2026-09-16 20:03:50.337263743 +0000 UTC m=+3507.463068134"
Sep 16 20:03:50.574479 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:50.574387    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:50.579248 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:50.579222    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:51.317102 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:51.317067    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:03:51.317923 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:03:51.317901    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:04:02.369422 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:02.369362    2583 generic.go:356] "Generic (PLEG): container finished" podID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerID="13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49" exitCode=0
Sep 16 20:04:02.369974 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:02.369447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" event={"ID":"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe","Type":"ContainerDied","Data":"13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49"}
Sep 16 20:04:02.369974 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:02.369910    2583 scope.go:117] "RemoveContainer" containerID="13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49"
Sep 16 20:04:02.370973 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:02.370943    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:04:02.371054 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:02.371018    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:04:02.371054 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:02.371049    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:04:03.376336 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:03.376303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" event={"ID":"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe","Type":"ContainerStarted","Data":"0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c"}
Sep 16 20:04:03.574839 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:03.574798    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:04:03.576020 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:03.575987    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 20:04:04.574910 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:04.574868    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 20:04:05.574738 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:05.574690    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 20:04:06.166717 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.166680    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"]
Sep 16 20:04:06.169635 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.169616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:06.173264 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.173230    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 20:04:06.173437 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.173288    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-qfb9t\""
Sep 16 20:04:06.173437 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.173379    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-cxp5k\""
Sep 16 20:04:06.179498 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.179476    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"]
Sep 16 20:04:06.238711 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.238682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvt2\" (UniqueName: \"kubernetes.io/projected/5658c743-e016-44c5-adb7-4346926cc8d1-kube-api-access-dhvt2\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg\" (UID: \"5658c743-e016-44c5-adb7-4346926cc8d1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:06.238876 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.238815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5658c743-e016-44c5-adb7-4346926cc8d1-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg\" (UID: \"5658c743-e016-44c5-adb7-4346926cc8d1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:06.318532 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.318497    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:04:06.339961 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.339934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5658c743-e016-44c5-adb7-4346926cc8d1-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg\" (UID: \"5658c743-e016-44c5-adb7-4346926cc8d1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:06.340104 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.339980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvt2\" (UniqueName: \"kubernetes.io/projected/5658c743-e016-44c5-adb7-4346926cc8d1-kube-api-access-dhvt2\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg\" (UID: \"5658c743-e016-44c5-adb7-4346926cc8d1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:06.342608 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.342586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5658c743-e016-44c5-adb7-4346926cc8d1-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg\" (UID: \"5658c743-e016-44c5-adb7-4346926cc8d1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:06.348839 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.348813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvt2\" (UniqueName: \"kubernetes.io/projected/5658c743-e016-44c5-adb7-4346926cc8d1-kube-api-access-dhvt2\") pod \"router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg\" (UID: \"5658c743-e016-44c5-adb7-4346926cc8d1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:06.480869 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.480791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:06.575057 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.575020    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 20:04:06.779916 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:06.779792    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"]
Sep 16 20:04:06.782638 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:04:06.782609    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5658c743_e016_44c5_adb7_4346926cc8d1.slice/crio-a602c1afa89059131fdf9ce09be3fe9a64a7d1f32aae0bb50461350806e2652c WatchSource:0}: Error finding container a602c1afa89059131fdf9ce09be3fe9a64a7d1f32aae0bb50461350806e2652c: Status 404 returned error can't find the container with id a602c1afa89059131fdf9ce09be3fe9a64a7d1f32aae0bb50461350806e2652c
Sep 16 20:04:07.396506 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:07.396463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" event={"ID":"5658c743-e016-44c5-adb7-4346926cc8d1","Type":"ContainerStarted","Data":"9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130"}
Sep 16 20:04:07.396978 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:07.396547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" event={"ID":"5658c743-e016-44c5-adb7-4346926cc8d1","Type":"ContainerStarted","Data":"a602c1afa89059131fdf9ce09be3fe9a64a7d1f32aae0bb50461350806e2652c"}
Sep 16 20:04:07.396978 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:07.396653    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:07.575164 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:07.575117    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 20:04:08.580380 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:08.580342    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:04:08.583232 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:08.583194    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:04:08.609166 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:08.609122    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" podStartSLOduration=2.609108326 podStartE2EDuration="2.609108326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:07.419114452 +0000 UTC m=+3524.544918840" watchObservedRunningTime="2026-09-16 20:04:08.609108326 +0000 UTC m=+3525.734912713"
Sep 16 20:04:09.008729 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:09.008701    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"]
Sep 16 20:04:11.731965 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:11.731923    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:21.732029 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:21.731989    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:23.855525 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:23.855481    2583 scope.go:117] "RemoveContainer" containerID="a89d99068214b941696c1503c85a10ad4f7a0e5d2ecfe98b7e9a84691c579cfb"
Sep 16 20:04:24.028754 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:24.028702    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" containerName="main" containerID="cri-o://55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122" gracePeriod=45
Sep 16 20:04:31.731988 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:31.731952    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:31.732493 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:31.732066    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:04:38.405173 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:38.405136    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:04:41.732133 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:41.732091    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:51.732275 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:51.732232    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:04:54.274861 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.274838    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:04:54.321891 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.321831    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g45mf\" (UniqueName: \"kubernetes.io/projected/2c95f5a1-e41b-449f-816c-7054a683219c-kube-api-access-g45mf\") pod \"2c95f5a1-e41b-449f-816c-7054a683219c\" (UID: \"2c95f5a1-e41b-449f-816c-7054a683219c\") "
Sep 16 20:04:54.321891 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.321864    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c95f5a1-e41b-449f-816c-7054a683219c-tls-certs\") pod \"2c95f5a1-e41b-449f-816c-7054a683219c\" (UID: \"2c95f5a1-e41b-449f-816c-7054a683219c\") "
Sep 16 20:04:54.324133 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.324105    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c95f5a1-e41b-449f-816c-7054a683219c-kube-api-access-g45mf" (OuterVolumeSpecName: "kube-api-access-g45mf") pod "2c95f5a1-e41b-449f-816c-7054a683219c" (UID: "2c95f5a1-e41b-449f-816c-7054a683219c"). InnerVolumeSpecName "kube-api-access-g45mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:04:54.324232 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.324135    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c95f5a1-e41b-449f-816c-7054a683219c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c95f5a1-e41b-449f-816c-7054a683219c" (UID: "2c95f5a1-e41b-449f-816c-7054a683219c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:04:54.423442 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.423396    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45mf\" (UniqueName: \"kubernetes.io/projected/2c95f5a1-e41b-449f-816c-7054a683219c-kube-api-access-g45mf\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:04:54.423442 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.423444    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c95f5a1-e41b-449f-816c-7054a683219c-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:04:54.614308 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.614216    2583 generic.go:356] "Generic (PLEG): container finished" podID="2c95f5a1-e41b-449f-816c-7054a683219c" containerID="55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122" exitCode=0
Sep 16 20:04:54.614308 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.614269    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"
Sep 16 20:04:54.614308 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.614295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" event={"ID":"2c95f5a1-e41b-449f-816c-7054a683219c","Type":"ContainerDied","Data":"55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122"}
Sep 16 20:04:54.614618 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.614358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs" event={"ID":"2c95f5a1-e41b-449f-816c-7054a683219c","Type":"ContainerDied","Data":"b5e2414ecb3bd65b6073fb16600a1855fd1d99f7ad100169aed346721186ea1f"}
Sep 16 20:04:54.614618 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.614386    2583 scope.go:117] "RemoveContainer" containerID="55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122"
Sep 16 20:04:54.623918 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.623902    2583 scope.go:117] "RemoveContainer" containerID="55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122"
Sep 16 20:04:54.624173 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:04:54.624155    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122\": container with ID starting with 55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122 not found: ID does not exist" containerID="55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122"
Sep 16 20:04:54.624221 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.624180    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122"} err="failed to get container status \"55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122\": rpc error: code = NotFound desc = could not find container \"55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122\": container with ID starting with 55a75cbc7d02e215fd6f84708ff60f01355480bb8ffac1439e5c396e9dae0122 not found: ID does not exist"
Sep 16 20:04:54.640014 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.639987    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"]
Sep 16 20:04:54.641547 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:54.641525    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7468747994-zpszs"]
Sep 16 20:04:55.411104 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:04:55.411073    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" path="/var/lib/kubelet/pods/2c95f5a1-e41b-449f-816c-7054a683219c/volumes"
Sep 16 20:05:27.034967 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:05:27.034929    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:05:27.041137 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:05:27.041116    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:05:27.044773 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:05:27.044753    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:05:27.050177 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:05:27.050156    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:08:35.321000 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:08:35.320912    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"]
Sep 16 20:08:38.404210 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:08:38.404156    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:08:48.404993 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:08:48.404944    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:08:50.345617 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:08:50.345552    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" containerName="main" containerID="cri-o://9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130" gracePeriod=45
Sep 16 20:08:58.404575 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:08:58.404535    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:08:58.404957 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:08:58.404650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:09:08.404806 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:08.404764    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:09:18.405057 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:18.405008    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:09:20.589744 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.589717    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:09:20.747930 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.747889    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhvt2\" (UniqueName: \"kubernetes.io/projected/5658c743-e016-44c5-adb7-4346926cc8d1-kube-api-access-dhvt2\") pod \"5658c743-e016-44c5-adb7-4346926cc8d1\" (UID: \"5658c743-e016-44c5-adb7-4346926cc8d1\") "
Sep 16 20:09:20.748096 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.747938    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5658c743-e016-44c5-adb7-4346926cc8d1-tls-certs\") pod \"5658c743-e016-44c5-adb7-4346926cc8d1\" (UID: \"5658c743-e016-44c5-adb7-4346926cc8d1\") "
Sep 16 20:09:20.750252 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.750228    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5658c743-e016-44c5-adb7-4346926cc8d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5658c743-e016-44c5-adb7-4346926cc8d1" (UID: "5658c743-e016-44c5-adb7-4346926cc8d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:09:20.750397 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.750377    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5658c743-e016-44c5-adb7-4346926cc8d1-kube-api-access-dhvt2" (OuterVolumeSpecName: "kube-api-access-dhvt2") pod "5658c743-e016-44c5-adb7-4346926cc8d1" (UID: "5658c743-e016-44c5-adb7-4346926cc8d1"). InnerVolumeSpecName "kube-api-access-dhvt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:09:20.848909 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.848872    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhvt2\" (UniqueName: \"kubernetes.io/projected/5658c743-e016-44c5-adb7-4346926cc8d1-kube-api-access-dhvt2\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:20.848909 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.848903    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5658c743-e016-44c5-adb7-4346926cc8d1-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:20.875718 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.875687    2583 generic.go:356] "Generic (PLEG): container finished" podID="5658c743-e016-44c5-adb7-4346926cc8d1" containerID="9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130" exitCode=0
Sep 16 20:09:20.875817 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.875731    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"
Sep 16 20:09:20.875817 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.875775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" event={"ID":"5658c743-e016-44c5-adb7-4346926cc8d1","Type":"ContainerDied","Data":"9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130"}
Sep 16 20:09:20.875898 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.875849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg" event={"ID":"5658c743-e016-44c5-adb7-4346926cc8d1","Type":"ContainerDied","Data":"a602c1afa89059131fdf9ce09be3fe9a64a7d1f32aae0bb50461350806e2652c"}
Sep 16 20:09:20.875898 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.875873    2583 scope.go:117] "RemoveContainer" containerID="9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130"
Sep 16 20:09:20.885160 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.885144    2583 scope.go:117] "RemoveContainer" containerID="9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130"
Sep 16 20:09:20.885395 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:09:20.885378    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130\": container with ID starting with 9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130 not found: ID does not exist" containerID="9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130"
Sep 16 20:09:20.885510 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.885401    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130"} err="failed to get container status \"9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130\": rpc error: code = NotFound desc = could not find container \"9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130\": container with ID starting with 9fde7fd06ad7bacfbd5bf01b567227963c466c5535df1c9d011afda29901f130 not found: ID does not exist"
Sep 16 20:09:20.897777 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.897758    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"]
Sep 16 20:09:20.902150 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:20.902125    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-56db58786c-t7gsg"]
Sep 16 20:09:21.412515 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:21.412481    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" path="/var/lib/kubelet/pods/5658c743-e016-44c5-adb7-4346926cc8d1/volumes"
Sep 16 20:09:30.336359 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:30.336318    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"]
Sep 16 20:09:30.336850 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:30.336591    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy" containerID="cri-o://0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c" gracePeriod=30
Sep 16 20:09:32.806835 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.806794    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"]
Sep 16 20:09:32.807251 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.807153    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" containerName="main"
Sep 16 20:09:32.807251 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.807165    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" containerName="main"
Sep 16 20:09:32.807251 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.807183    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" containerName="main"
Sep 16 20:09:32.807251 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.807189    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" containerName="main"
Sep 16 20:09:32.807251 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.807247    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5658c743-e016-44c5-adb7-4346926cc8d1" containerName="main"
Sep 16 20:09:32.807469 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.807257    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c95f5a1-e41b-449f-816c-7054a683219c" containerName="main"
Sep 16 20:09:32.810763 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.810743    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:32.814588 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.814562    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-hhfxw\""
Sep 16 20:09:32.814999 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.814972    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 20:09:32.815132 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.815086    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 20:09:32.815298 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.815265    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-cg7q2\""
Sep 16 20:09:32.815636 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.815612    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:09:32.815757 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.815721    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 20:09:32.819472 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.819447    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 20:09:32.842243 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.842211    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"]
Sep 16 20:09:32.948276 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.948252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4kx\" (UniqueName: \"kubernetes.io/projected/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kube-api-access-2n4kx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:32.948399 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.948309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:32.948486 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.948392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:32.948486 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.948440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:32.948486 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.948457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc22d1d8-c08f-4b66-8dea-8201f67ae991-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:32.948620 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.948494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:32.948620 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.948522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:32.948620 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:32.948543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.049857 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.049821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.049999 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.049883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.049999 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.049922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.049999 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.049947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc22d1d8-c08f-4b66-8dea-8201f67ae991-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.049999 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.049976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.050209 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.050013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.050209 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.050043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.050209 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.050076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4kx\" (UniqueName: \"kubernetes.io/projected/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kube-api-access-2n4kx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.050369 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.050304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.050369 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.050320    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.050514 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.050468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.050643 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.050618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.050868 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.050714    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc22d1d8-c08f-4b66-8dea-8201f67ae991-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.052384 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.052362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.052795 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.052774    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.064871 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.064815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4kx\" (UniqueName: \"kubernetes.io/projected/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kube-api-access-2n4kx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.122677 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.122655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:33.409384 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:09:33.409357    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc22d1d8_c08f_4b66_8dea_8201f67ae991.slice/crio-5ea5db948dcf50dee6fffc3a31885660bcd49f103eda35ce4abf75e696447d20 WatchSource:0}: Error finding container 5ea5db948dcf50dee6fffc3a31885660bcd49f103eda35ce4abf75e696447d20: Status 404 returned error can't find the container with id 5ea5db948dcf50dee6fffc3a31885660bcd49f103eda35ce4abf75e696447d20
Sep 16 20:09:33.411688 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.411672    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:09:33.414775 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.414752    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"]
Sep 16 20:09:33.939260 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:33.939219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" event={"ID":"dc22d1d8-c08f-4b66-8dea-8201f67ae991","Type":"ContainerStarted","Data":"5ea5db948dcf50dee6fffc3a31885660bcd49f103eda35ce4abf75e696447d20"}
Sep 16 20:09:35.586172 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.586141    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:09:35.676999 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.676961    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-token\") pod \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") "
Sep 16 20:09:35.677151 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677020    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-credential-socket\") pod \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") "
Sep 16 20:09:35.677151 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-data\") pod \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") "
Sep 16 20:09:35.677151 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-envoy\") pod \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") "
Sep 16 20:09:35.677320 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677191    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-podinfo\") pod \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") "
Sep 16 20:09:35.677320 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677260    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5jnk\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-kube-api-access-j5jnk\") pod \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") "
Sep 16 20:09:35.677320 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-certs\") pod \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") "
Sep 16 20:09:35.677564 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" (UID: "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:35.677564 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677337    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istiod-ca-cert\") pod \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") "
Sep 16 20:09:35.677564 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677320    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" (UID: "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:35.677564 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-socket\") pod \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\" (UID: \"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe\") "
Sep 16 20:09:35.677564 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677548    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" (UID: "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:35.677835 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677733    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:35.677835 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677752    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-credential-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:35.677835 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677766    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-data\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:35.677835 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" (UID: "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:09:35.677835 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.677811    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" (UID: "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:35.680068 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.680031    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" (UID: "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:09:35.680317 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.680286    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-kube-api-access-j5jnk" (OuterVolumeSpecName: "kube-api-access-j5jnk") pod "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" (UID: "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe"). InnerVolumeSpecName "kube-api-access-j5jnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:09:35.680465 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.680315    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" (UID: "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:09:35.680465 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.680444    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" (UID: "ae9f7ab4-5e88-414a-b314-cfd9157cf5fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 20:09:35.779271 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.779180    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-envoy\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:35.779271 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.779214    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-podinfo\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:35.779271 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.779223    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jnk\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-kube-api-access-j5jnk\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:35.779271 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.779233    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istiod-ca-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:35.779271 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.779242    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-workload-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:35.779271 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.779250    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe-istio-token\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:09:35.953474 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.953429    2583 generic.go:356] "Generic (PLEG): container finished" podID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerID="0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c" exitCode=0
Sep 16 20:09:35.953665 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.953509    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"
Sep 16 20:09:35.953665 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.953515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" event={"ID":"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe","Type":"ContainerDied","Data":"0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c"}
Sep 16 20:09:35.953810 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.953692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8" event={"ID":"ae9f7ab4-5e88-414a-b314-cfd9157cf5fe","Type":"ContainerDied","Data":"9ddba1ac4ac173f7f896e9ae1eefdbaa8d480bded762500479c3cf3d82efa730"}
Sep 16 20:09:35.953810 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.953734    2583 scope.go:117] "RemoveContainer" containerID="0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c"
Sep 16 20:09:35.956067 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.956037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" event={"ID":"dc22d1d8-c08f-4b66-8dea-8201f67ae991","Type":"ContainerStarted","Data":"9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f"}
Sep 16 20:09:35.956243 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.956222    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:35.964527 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.964508    2583 scope.go:117] "RemoveContainer" containerID="13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49"
Sep 16 20:09:35.974175 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.974154    2583 scope.go:117] "RemoveContainer" containerID="0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c"
Sep 16 20:09:35.974479 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:09:35.974454    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c\": container with ID starting with 0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c not found: ID does not exist" containerID="0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c"
Sep 16 20:09:35.974575 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.974485    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c"} err="failed to get container status \"0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c\": rpc error: code = NotFound desc = could not find container \"0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c\": container with ID starting with 0f3253a30f48bac9fc3dbfa0374e20c3d5962a5c31a39b6d47bc08042172d23c not found: ID does not exist"
Sep 16 20:09:35.974575 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.974509    2583 scope.go:117] "RemoveContainer" containerID="13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49"
Sep 16 20:09:35.974777 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:09:35.974757    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49\": container with ID starting with 13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49 not found: ID does not exist" containerID="13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49"
Sep 16 20:09:35.974821 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:35.974788    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49"} err="failed to get container status \"13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49\": rpc error: code = NotFound desc = could not find container \"13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49\": container with ID starting with 13107a50dae9d0873f8f783222b638794cdbc14bb5935977206e69a1470c6e49 not found: ID does not exist"
Sep 16 20:09:36.000493 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:36.000466    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"]
Sep 16 20:09:36.003273 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:36.003249    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-jb5k8"]
Sep 16 20:09:36.963481 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:36.963442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" event={"ID":"dc22d1d8-c08f-4b66-8dea-8201f67ae991","Type":"ContainerStarted","Data":"83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf"}
Sep 16 20:09:37.415558 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:37.415514    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" path="/var/lib/kubelet/pods/ae9f7ab4-5e88-414a-b314-cfd9157cf5fe/volumes"
Sep 16 20:09:38.974817 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:38.974783    2583 generic.go:356] "Generic (PLEG): container finished" podID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerID="83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf" exitCode=0
Sep 16 20:09:38.975228 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:38.974854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" event={"ID":"dc22d1d8-c08f-4b66-8dea-8201f67ae991","Type":"ContainerDied","Data":"83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf"}
Sep 16 20:09:39.983375 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:39.983344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" event={"ID":"dc22d1d8-c08f-4b66-8dea-8201f67ae991","Type":"ContainerStarted","Data":"5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70"}
Sep 16 20:09:40.012653 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:40.012605    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podStartSLOduration=4.065953476 podStartE2EDuration="8.012591878s" totalImagesPullingTime="1.774732674s" totalInitContainerRuntime="2.171905728s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:09:40.012087505 +0000 UTC m=+3857.137891930" watchObservedRunningTime="2026-09-16 20:09:40.012591878 +0000 UTC m=+3857.138396313"
Sep 16 20:09:43.122880 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:43.122843    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:43.123334 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:43.122914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:09:43.124348 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:43.124320    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:09:53.123744 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:53.123700    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:09:54.017701 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:09:54.017664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:10:03.124015 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:03.123970    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:10:13.123685 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:13.123622    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:10:23.123676 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:23.123627    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:10:27.068987 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:27.068962    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:10:27.074494 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:27.074468    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:10:27.078333 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:27.078312    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:10:27.083710 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:27.083693    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:10:33.123179 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:33.123139    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:10:43.123324 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:43.123271    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:10:53.124010 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:10:53.123965    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:11:03.123623 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:11:03.123581    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:11:13.123626 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:11:13.123575    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:11:23.123451 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:11:23.123332    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 16 20:11:33.132825 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:11:33.132790    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:11:33.144844 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:11:33.144818    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:11:55.336827 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:11:55.336793    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"]
Sep 16 20:12:10.360281 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:10.360235    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main" containerID="cri-o://5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70" gracePeriod=45
Sep 16 20:12:10.976976 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:10.976944    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="llm-d-routing-sidecar" containerID="cri-o://9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f" gracePeriod=45
Sep 16 20:12:11.216124 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.216099    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:12:11.289823 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.289794    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc22d1d8-c08f-4b66-8dea-8201f67ae991-cabundle-cert\") pod \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") "
Sep 16 20:12:11.289976 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.289829    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-dshm\") pod \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") "
Sep 16 20:12:11.289976 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.289862    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tmp-dir\") pod \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") "
Sep 16 20:12:11.289976 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.289905    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n4kx\" (UniqueName: \"kubernetes.io/projected/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kube-api-access-2n4kx\") pod \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") "
Sep 16 20:12:11.289976 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.289963    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-home\") pod \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") "
Sep 16 20:12:11.290216 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.290000    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kserve-provision-location\") pod \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") "
Sep 16 20:12:11.290216 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.290046    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tls-certs\") pod \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") "
Sep 16 20:12:11.290216 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.290075    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-model-cache\") pod \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\" (UID: \"dc22d1d8-c08f-4b66-8dea-8201f67ae991\") "
Sep 16 20:12:11.290376 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.290306    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc22d1d8-c08f-4b66-8dea-8201f67ae991-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc22d1d8-c08f-4b66-8dea-8201f67ae991" (UID: "dc22d1d8-c08f-4b66-8dea-8201f67ae991"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:12:11.290842 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.290759    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-home" (OuterVolumeSpecName: "home") pod "dc22d1d8-c08f-4b66-8dea-8201f67ae991" (UID: "dc22d1d8-c08f-4b66-8dea-8201f67ae991"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:11.290842 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.290793    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc22d1d8-c08f-4b66-8dea-8201f67ae991" (UID: "dc22d1d8-c08f-4b66-8dea-8201f67ae991"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:11.292312 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.292285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-dshm" (OuterVolumeSpecName: "dshm") pod "dc22d1d8-c08f-4b66-8dea-8201f67ae991" (UID: "dc22d1d8-c08f-4b66-8dea-8201f67ae991"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:11.292497 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.292468    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kube-api-access-2n4kx" (OuterVolumeSpecName: "kube-api-access-2n4kx") pod "dc22d1d8-c08f-4b66-8dea-8201f67ae991" (UID: "dc22d1d8-c08f-4b66-8dea-8201f67ae991"). InnerVolumeSpecName "kube-api-access-2n4kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:12:11.292914 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.292885    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc22d1d8-c08f-4b66-8dea-8201f67ae991" (UID: "dc22d1d8-c08f-4b66-8dea-8201f67ae991"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:12:11.302967 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.302943    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc22d1d8-c08f-4b66-8dea-8201f67ae991" (UID: "dc22d1d8-c08f-4b66-8dea-8201f67ae991"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:11.312617 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.312593    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc22d1d8-c08f-4b66-8dea-8201f67ae991" (UID: "dc22d1d8-c08f-4b66-8dea-8201f67ae991"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:11.391660 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.391639    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:12:11.391924 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.391661    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:12:11.391924 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.391671    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:12:11.391924 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.391681    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:12:11.391924 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.391688    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc22d1d8-c08f-4b66-8dea-8201f67ae991-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:12:11.391924 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.391695    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:12:11.391924 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.391702    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc22d1d8-c08f-4b66-8dea-8201f67ae991-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:12:11.391924 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.391709    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4kx\" (UniqueName: \"kubernetes.io/projected/dc22d1d8-c08f-4b66-8dea-8201f67ae991-kube-api-access-2n4kx\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:12:11.719180 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.719091    2583 generic.go:356] "Generic (PLEG): container finished" podID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerID="5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70" exitCode=0
Sep 16 20:12:11.719180 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.719123    2583 generic.go:356] "Generic (PLEG): container finished" podID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerID="9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f" exitCode=0
Sep 16 20:12:11.719180 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.719164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" event={"ID":"dc22d1d8-c08f-4b66-8dea-8201f67ae991","Type":"ContainerDied","Data":"5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70"}
Sep 16 20:12:11.719392 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.719216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" event={"ID":"dc22d1d8-c08f-4b66-8dea-8201f67ae991","Type":"ContainerDied","Data":"9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f"}
Sep 16 20:12:11.719392 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.719235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf" event={"ID":"dc22d1d8-c08f-4b66-8dea-8201f67ae991","Type":"ContainerDied","Data":"5ea5db948dcf50dee6fffc3a31885660bcd49f103eda35ce4abf75e696447d20"}
Sep 16 20:12:11.719392 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.719167    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"
Sep 16 20:12:11.719392 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.719260    2583 scope.go:117] "RemoveContainer" containerID="5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70"
Sep 16 20:12:11.730437 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.730304    2583 scope.go:117] "RemoveContainer" containerID="83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf"
Sep 16 20:12:11.738721 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.738701    2583 scope.go:117] "RemoveContainer" containerID="9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f"
Sep 16 20:12:11.741445 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.741400    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"]
Sep 16 20:12:11.745614 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.745585    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6f7fcc968dxcmsf"]
Sep 16 20:12:11.747299 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.747280    2583 scope.go:117] "RemoveContainer" containerID="5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70"
Sep 16 20:12:11.747624 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:12:11.747603    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70\": container with ID starting with 5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70 not found: ID does not exist" containerID="5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70"
Sep 16 20:12:11.747721 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.747631    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70"} err="failed to get container status \"5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70\": rpc error: code = NotFound desc = could not find container \"5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70\": container with ID starting with 5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70 not found: ID does not exist"
Sep 16 20:12:11.747721 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.747650    2583 scope.go:117] "RemoveContainer" containerID="83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf"
Sep 16 20:12:11.747895 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:12:11.747877    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf\": container with ID starting with 83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf not found: ID does not exist" containerID="83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf"
Sep 16 20:12:11.747945 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.747902    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf"} err="failed to get container status \"83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf\": rpc error: code = NotFound desc = could not find container \"83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf\": container with ID starting with 83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf not found: ID does not exist"
Sep 16 20:12:11.747945 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.747919    2583 scope.go:117] "RemoveContainer" containerID="9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f"
Sep 16 20:12:11.748176 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:12:11.748158    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f\": container with ID starting with 9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f not found: ID does not exist" containerID="9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f"
Sep 16 20:12:11.748225 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.748178    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f"} err="failed to get container status \"9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f\": rpc error: code = NotFound desc = could not find container \"9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f\": container with ID starting with 9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f not found: ID does not exist"
Sep 16 20:12:11.748225 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.748192    2583 scope.go:117] "RemoveContainer" containerID="5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70"
Sep 16 20:12:11.748692 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.748667    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70"} err="failed to get container status \"5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70\": rpc error: code = NotFound desc = could not find container \"5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70\": container with ID starting with 5254aefcc1c4c46ceeb049395ed3d33228b5ea6abe32655396c0aeb412c85b70 not found: ID does not exist"
Sep 16 20:12:11.748692 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.748692    2583 scope.go:117] "RemoveContainer" containerID="83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf"
Sep 16 20:12:11.748943 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.748926    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf"} err="failed to get container status \"83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf\": rpc error: code = NotFound desc = could not find container \"83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf\": container with ID starting with 83b338e0475b3be268488919e29e883cfba3ad7036ac6289ed51098ca3508fdf not found: ID does not exist"
Sep 16 20:12:11.748943 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.748943    2583 scope.go:117] "RemoveContainer" containerID="9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f"
Sep 16 20:12:11.749169 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:11.749152    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f"} err="failed to get container status \"9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f\": rpc error: code = NotFound desc = could not find container \"9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f\": container with ID starting with 9bd10eb6d5d73e483775e54fde03a220c751d384bf99de54375e1053b034cb4f not found: ID does not exist"
Sep 16 20:12:13.412614 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:12:13.412582    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" path="/var/lib/kubelet/pods/dc22d1d8-c08f-4b66-8dea-8201f67ae991/volumes"
Sep 16 20:13:05.201222 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201183    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"]
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201591    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="llm-d-routing-sidecar"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201607    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="llm-d-routing-sidecar"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201626    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="storage-initializer"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201632    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="storage-initializer"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201642    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201648    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201655    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201661    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201668    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201674    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201726    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="llm-d-routing-sidecar"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201734    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201739    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc22d1d8-c08f-4b66-8dea-8201f67ae991" containerName="main"
Sep 16 20:13:05.201740 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.201748    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9f7ab4-5e88-414a-b314-cfd9157cf5fe" containerName="istio-proxy"
Sep 16 20:13:05.204275 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.204256    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.206929 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.206909    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:13:05.207063 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.206989    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 20:13:05.209361 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.209326    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 20:13:05.209361 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.209341    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 20:13:05.209361 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.209353    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-dfhz5\""
Sep 16 20:13:05.209637 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.209361    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zhrkw\""
Sep 16 20:13:05.209637 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.209338    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 20:13:05.219849 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.219829    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"]
Sep 16 20:13:05.319643 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.319609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a458d7-49a5-4349-a7e1-788447ce3c42-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.319839 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.319648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nkj\" (UniqueName: \"kubernetes.io/projected/69a458d7-49a5-4349-a7e1-788447ce3c42-kube-api-access-b9nkj\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.319839 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.319684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.319839 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.319759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-home\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.319839 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.319809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.319839 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.319826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-model-cache\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.320012 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.319877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-dshm\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.320012 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.319894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a458d7-49a5-4349-a7e1-788447ce3c42-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.420646 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.420615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a458d7-49a5-4349-a7e1-788447ce3c42-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.420646 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.420650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nkj\" (UniqueName: \"kubernetes.io/projected/69a458d7-49a5-4349-a7e1-788447ce3c42-kube-api-access-b9nkj\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.420898 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.420685    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.420898 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.420704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-home\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.420898 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.420747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.420898 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.420789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-model-cache\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.420898 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.420846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-dshm\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.420898 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.420874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a458d7-49a5-4349-a7e1-788447ce3c42-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.421213 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.421058    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-home\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.421213 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.421122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.421213 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.421182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-model-cache\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.421329 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.421221    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.421433 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.421391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a458d7-49a5-4349-a7e1-788447ce3c42-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.423394 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.423373    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-dshm\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.423491 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.423468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a458d7-49a5-4349-a7e1-788447ce3c42-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.431392 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.431369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nkj\" (UniqueName: \"kubernetes.io/projected/69a458d7-49a5-4349-a7e1-788447ce3c42-kube-api-access-b9nkj\") pod \"custom-route-timeout-pd-test-kserve-67549f495f-hgp99\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.513629 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.513540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:05.517472 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.517400    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"]
Sep 16 20:13:05.519880 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.519859    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:05.522533 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.522511    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-g6qz5\""
Sep 16 20:13:05.530612 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.530588    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"]
Sep 16 20:13:05.622386 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.622352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83063b0-e0cc-469f-8e59-98a06196ac12-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g\" (UID: \"e83063b0-e0cc-469f-8e59-98a06196ac12\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:05.622561 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.622530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2czt\" (UniqueName: \"kubernetes.io/projected/e83063b0-e0cc-469f-8e59-98a06196ac12-kube-api-access-t2czt\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g\" (UID: \"e83063b0-e0cc-469f-8e59-98a06196ac12\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:05.723571 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.723540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83063b0-e0cc-469f-8e59-98a06196ac12-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g\" (UID: \"e83063b0-e0cc-469f-8e59-98a06196ac12\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:05.723707 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.723629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2czt\" (UniqueName: \"kubernetes.io/projected/e83063b0-e0cc-469f-8e59-98a06196ac12-kube-api-access-t2czt\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g\" (UID: \"e83063b0-e0cc-469f-8e59-98a06196ac12\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:05.726177 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.726156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83063b0-e0cc-469f-8e59-98a06196ac12-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g\" (UID: \"e83063b0-e0cc-469f-8e59-98a06196ac12\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:05.732235 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.732216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2czt\" (UniqueName: \"kubernetes.io/projected/e83063b0-e0cc-469f-8e59-98a06196ac12-kube-api-access-t2czt\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g\" (UID: \"e83063b0-e0cc-469f-8e59-98a06196ac12\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:05.797202 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.797178    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"]
Sep 16 20:13:05.798988 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:13:05.798964    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a458d7_49a5_4349_a7e1_788447ce3c42.slice/crio-5859ddf41ada10f2ea38a7453b4ec490b96e48946e232f19e54f73a0dfbbfedd WatchSource:0}: Error finding container 5859ddf41ada10f2ea38a7453b4ec490b96e48946e232f19e54f73a0dfbbfedd: Status 404 returned error can't find the container with id 5859ddf41ada10f2ea38a7453b4ec490b96e48946e232f19e54f73a0dfbbfedd
Sep 16 20:13:05.848296 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.848277    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:05.975468 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.975436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" event={"ID":"69a458d7-49a5-4349-a7e1-788447ce3c42","Type":"ContainerStarted","Data":"0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611"}
Sep 16 20:13:05.975599 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.975491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" event={"ID":"69a458d7-49a5-4349-a7e1-788447ce3c42","Type":"ContainerStarted","Data":"5859ddf41ada10f2ea38a7453b4ec490b96e48946e232f19e54f73a0dfbbfedd"}
Sep 16 20:13:05.975599 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:05.975579    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:06.124696 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:06.124670    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"]
Sep 16 20:13:06.126724 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:13:06.126687    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83063b0_e0cc_469f_8e59_98a06196ac12.slice/crio-1ba27b8e8c20baf2d689cb8bd0fed361a3d4c3a60948fbf910a8e29313afbeed WatchSource:0}: Error finding container 1ba27b8e8c20baf2d689cb8bd0fed361a3d4c3a60948fbf910a8e29313afbeed: Status 404 returned error can't find the container with id 1ba27b8e8c20baf2d689cb8bd0fed361a3d4c3a60948fbf910a8e29313afbeed
Sep 16 20:13:06.981873 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:06.981832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" event={"ID":"e83063b0-e0cc-469f-8e59-98a06196ac12","Type":"ContainerStarted","Data":"22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b"}
Sep 16 20:13:06.982335 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:06.981891    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:06.982335 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:06.981912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" event={"ID":"e83063b0-e0cc-469f-8e59-98a06196ac12","Type":"ContainerStarted","Data":"1ba27b8e8c20baf2d689cb8bd0fed361a3d4c3a60948fbf910a8e29313afbeed"}
Sep 16 20:13:06.983737 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:06.983704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" event={"ID":"69a458d7-49a5-4349-a7e1-788447ce3c42","Type":"ContainerStarted","Data":"3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96"}
Sep 16 20:13:06.998517 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:06.998464    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" podStartSLOduration=1.998445464 podStartE2EDuration="1.998445464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:13:06.997437128 +0000 UTC m=+4064.123241518" watchObservedRunningTime="2026-09-16 20:13:06.998445464 +0000 UTC m=+4064.124249853"
Sep 16 20:13:08.995647 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:08.995616    2583 generic.go:356] "Generic (PLEG): container finished" podID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerID="3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96" exitCode=0
Sep 16 20:13:08.996057 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:08.995685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" event={"ID":"69a458d7-49a5-4349-a7e1-788447ce3c42","Type":"ContainerDied","Data":"3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96"}
Sep 16 20:13:10.002830 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:10.002798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" event={"ID":"69a458d7-49a5-4349-a7e1-788447ce3c42","Type":"ContainerStarted","Data":"9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337"}
Sep 16 20:13:10.028643 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:10.028582    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podStartSLOduration=2.798090515 podStartE2EDuration="5.028560647s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.230470132s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:13:10.027805696 +0000 UTC m=+4067.153610110" watchObservedRunningTime="2026-09-16 20:13:10.028560647 +0000 UTC m=+4067.154365039"
Sep 16 20:13:15.513775 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:15.513739    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:15.514216 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:15.513810    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:15.515009 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:15.514975    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:13:15.527882 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:15.527853    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:13:25.514294 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:25.514250    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:13:35.513919 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:35.513869    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:13:37.991033 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:37.990999    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:13:45.514614 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:45.514568    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:13:55.514809 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:13:55.514764    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:14:05.514545 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:14:05.514492    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:14:15.514869 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:14:15.514762    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:14:25.514844 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:14:25.514793    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:14:35.514890 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:14:35.514829    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:14:45.514647 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:14:45.514587    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:14:55.514167 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:14:55.514109    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8001/health\": dial tcp 10.134.0.79:8001: connect: connection refused"
Sep 16 20:15:05.524038 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:05.524007    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:15:05.540603 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:05.540576    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:15:27.100844 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:27.100814    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:15:27.106442 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:27.106390    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:15:27.111030 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:27.111013    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:15:27.116642 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:27.116626    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:15:44.979322 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:44.979232    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"]
Sep 16 20:15:44.986395 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:44.986367    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"]
Sep 16 20:15:47.990657 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:47.990611    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:15:57.990311 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:57.990268    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:15:59.999774 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:15:59.999714    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerName="main" containerID="cri-o://22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b" gracePeriod=45
Sep 16 20:16:00.013523 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.013488    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main" containerID="cri-o://9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337" gracePeriod=45
Sep 16 20:16:00.541524 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.541492    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="llm-d-routing-sidecar" containerID="cri-o://0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611" gracePeriod=45
Sep 16 20:16:00.791770 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.791745    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:16:00.821381 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.821353    2583 generic.go:356] "Generic (PLEG): container finished" podID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerID="9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337" exitCode=0
Sep 16 20:16:00.821381 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.821375    2583 generic.go:356] "Generic (PLEG): container finished" podID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerID="0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611" exitCode=0
Sep 16 20:16:00.821565 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.821439    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"
Sep 16 20:16:00.821565 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.821437    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" event={"ID":"69a458d7-49a5-4349-a7e1-788447ce3c42","Type":"ContainerDied","Data":"9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337"}
Sep 16 20:16:00.821651 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.821565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" event={"ID":"69a458d7-49a5-4349-a7e1-788447ce3c42","Type":"ContainerDied","Data":"0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611"}
Sep 16 20:16:00.821651 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.821584    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99" event={"ID":"69a458d7-49a5-4349-a7e1-788447ce3c42","Type":"ContainerDied","Data":"5859ddf41ada10f2ea38a7453b4ec490b96e48946e232f19e54f73a0dfbbfedd"}
Sep 16 20:16:00.821651 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.821596    2583 scope.go:117] "RemoveContainer" containerID="9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337"
Sep 16 20:16:00.831544 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.831505    2583 scope.go:117] "RemoveContainer" containerID="3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96"
Sep 16 20:16:00.839545 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.839526    2583 scope.go:117] "RemoveContainer" containerID="0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611"
Sep 16 20:16:00.847158 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.847129    2583 scope.go:117] "RemoveContainer" containerID="9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337"
Sep 16 20:16:00.847444 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:16:00.847404    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337\": container with ID starting with 9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337 not found: ID does not exist" containerID="9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337"
Sep 16 20:16:00.847499 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.847452    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337"} err="failed to get container status \"9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337\": rpc error: code = NotFound desc = could not find container \"9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337\": container with ID starting with 9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337 not found: ID does not exist"
Sep 16 20:16:00.847499 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.847467    2583 scope.go:117] "RemoveContainer" containerID="3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96"
Sep 16 20:16:00.847710 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:16:00.847692    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96\": container with ID starting with 3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96 not found: ID does not exist" containerID="3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96"
Sep 16 20:16:00.847757 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.847717    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96"} err="failed to get container status \"3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96\": rpc error: code = NotFound desc = could not find container \"3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96\": container with ID starting with 3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96 not found: ID does not exist"
Sep 16 20:16:00.847757 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.847733    2583 scope.go:117] "RemoveContainer" containerID="0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611"
Sep 16 20:16:00.847995 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:16:00.847973    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611\": container with ID starting with 0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611 not found: ID does not exist" containerID="0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611"
Sep 16 20:16:00.848038 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.848004    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611"} err="failed to get container status \"0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611\": rpc error: code = NotFound desc = could not find container \"0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611\": container with ID starting with 0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611 not found: ID does not exist"
Sep 16 20:16:00.848038 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.848025    2583 scope.go:117] "RemoveContainer" containerID="9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337"
Sep 16 20:16:00.848281 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.848263    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337"} err="failed to get container status \"9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337\": rpc error: code = NotFound desc = could not find container \"9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337\": container with ID starting with 9f3da6ef9367ff55bb7031df4be24a3039708bed12e5e6f644f995400aeaa337 not found: ID does not exist"
Sep 16 20:16:00.848329 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.848282    2583 scope.go:117] "RemoveContainer" containerID="3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96"
Sep 16 20:16:00.848553 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.848529    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96"} err="failed to get container status \"3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96\": rpc error: code = NotFound desc = could not find container \"3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96\": container with ID starting with 3c47586f62fa77df51f02f00713eb8f3a9fd67909cfbc21e7127ddcd4b2c2a96 not found: ID does not exist"
Sep 16 20:16:00.848625 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.848556    2583 scope.go:117] "RemoveContainer" containerID="0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611"
Sep 16 20:16:00.848820 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.848801    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611"} err="failed to get container status \"0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611\": rpc error: code = NotFound desc = could not find container \"0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611\": container with ID starting with 0cc70f7df1200c657e9cb6987748f50cd9e163cce8277665e92ae8cda53fd611 not found: ID does not exist"
Sep 16 20:16:00.853100 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853082    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-model-cache\") pod \"69a458d7-49a5-4349-a7e1-788447ce3c42\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") "
Sep 16 20:16:00.853190 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853119    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-home\") pod \"69a458d7-49a5-4349-a7e1-788447ce3c42\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") "
Sep 16 20:16:00.853190 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853166    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9nkj\" (UniqueName: \"kubernetes.io/projected/69a458d7-49a5-4349-a7e1-788447ce3c42-kube-api-access-b9nkj\") pod \"69a458d7-49a5-4349-a7e1-788447ce3c42\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") "
Sep 16 20:16:00.853299 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853196    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-kserve-provision-location\") pod \"69a458d7-49a5-4349-a7e1-788447ce3c42\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") "
Sep 16 20:16:00.853299 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853228    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-tmp-dir\") pod \"69a458d7-49a5-4349-a7e1-788447ce3c42\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") "
Sep 16 20:16:00.853299 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853266    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a458d7-49a5-4349-a7e1-788447ce3c42-tls-certs\") pod \"69a458d7-49a5-4349-a7e1-788447ce3c42\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") "
Sep 16 20:16:00.853476 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853327    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-dshm\") pod \"69a458d7-49a5-4349-a7e1-788447ce3c42\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") "
Sep 16 20:16:00.853476 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853312    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-model-cache" (OuterVolumeSpecName: "model-cache") pod "69a458d7-49a5-4349-a7e1-788447ce3c42" (UID: "69a458d7-49a5-4349-a7e1-788447ce3c42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:16:00.853476 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853450    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a458d7-49a5-4349-a7e1-788447ce3c42-cabundle-cert\") pod \"69a458d7-49a5-4349-a7e1-788447ce3c42\" (UID: \"69a458d7-49a5-4349-a7e1-788447ce3c42\") "
Sep 16 20:16:00.853847 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853758    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:00.854094 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.853888    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-home" (OuterVolumeSpecName: "home") pod "69a458d7-49a5-4349-a7e1-788447ce3c42" (UID: "69a458d7-49a5-4349-a7e1-788447ce3c42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:16:00.854284 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.854262    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a458d7-49a5-4349-a7e1-788447ce3c42-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69a458d7-49a5-4349-a7e1-788447ce3c42" (UID: "69a458d7-49a5-4349-a7e1-788447ce3c42"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:16:00.855578 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.855556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-dshm" (OuterVolumeSpecName: "dshm") pod "69a458d7-49a5-4349-a7e1-788447ce3c42" (UID: "69a458d7-49a5-4349-a7e1-788447ce3c42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:16:00.855724 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.855704    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a458d7-49a5-4349-a7e1-788447ce3c42-kube-api-access-b9nkj" (OuterVolumeSpecName: "kube-api-access-b9nkj") pod "69a458d7-49a5-4349-a7e1-788447ce3c42" (UID: "69a458d7-49a5-4349-a7e1-788447ce3c42"). InnerVolumeSpecName "kube-api-access-b9nkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:16:00.857196 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.857174    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a458d7-49a5-4349-a7e1-788447ce3c42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69a458d7-49a5-4349-a7e1-788447ce3c42" (UID: "69a458d7-49a5-4349-a7e1-788447ce3c42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:16:00.871230 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.871196    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69a458d7-49a5-4349-a7e1-788447ce3c42" (UID: "69a458d7-49a5-4349-a7e1-788447ce3c42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:16:00.882194 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.882167    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69a458d7-49a5-4349-a7e1-788447ce3c42" (UID: "69a458d7-49a5-4349-a7e1-788447ce3c42"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:16:00.954490 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.954468    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69a458d7-49a5-4349-a7e1-788447ce3c42-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:00.954490 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.954490    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:00.954637 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.954499    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9nkj\" (UniqueName: \"kubernetes.io/projected/69a458d7-49a5-4349-a7e1-788447ce3c42-kube-api-access-b9nkj\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:00.954637 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.954509    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:00.954637 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.954518    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:00.954637 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.954529    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a458d7-49a5-4349-a7e1-788447ce3c42-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:00.954637 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:00.954537    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a458d7-49a5-4349-a7e1-788447ce3c42-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:01.147608 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:01.147581    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"]
Sep 16 20:16:01.153306 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:01.153286    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-67549f495f-hgp99"]
Sep 16 20:16:01.411381 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:01.411301    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" path="/var/lib/kubelet/pods/69a458d7-49a5-4349-a7e1-788447ce3c42/volumes"
Sep 16 20:16:07.990309 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:07.990264    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:16:07.990724 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:07.990392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:16:17.990769 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:17.990730    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:16:27.990619 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:27.990574    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:16:30.252357 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.252333    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:16:30.385772 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.385743    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83063b0-e0cc-469f-8e59-98a06196ac12-tls-certs\") pod \"e83063b0-e0cc-469f-8e59-98a06196ac12\" (UID: \"e83063b0-e0cc-469f-8e59-98a06196ac12\") "
Sep 16 20:16:30.385916 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.385853    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2czt\" (UniqueName: \"kubernetes.io/projected/e83063b0-e0cc-469f-8e59-98a06196ac12-kube-api-access-t2czt\") pod \"e83063b0-e0cc-469f-8e59-98a06196ac12\" (UID: \"e83063b0-e0cc-469f-8e59-98a06196ac12\") "
Sep 16 20:16:30.388060 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.388034    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83063b0-e0cc-469f-8e59-98a06196ac12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e83063b0-e0cc-469f-8e59-98a06196ac12" (UID: "e83063b0-e0cc-469f-8e59-98a06196ac12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:16:30.388163 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.388141    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83063b0-e0cc-469f-8e59-98a06196ac12-kube-api-access-t2czt" (OuterVolumeSpecName: "kube-api-access-t2czt") pod "e83063b0-e0cc-469f-8e59-98a06196ac12" (UID: "e83063b0-e0cc-469f-8e59-98a06196ac12"). InnerVolumeSpecName "kube-api-access-t2czt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:16:30.486768 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.486741    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2czt\" (UniqueName: \"kubernetes.io/projected/e83063b0-e0cc-469f-8e59-98a06196ac12-kube-api-access-t2czt\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:30.486768 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.486763    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83063b0-e0cc-469f-8e59-98a06196ac12-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:16:30.971147 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.971111    2583 generic.go:356] "Generic (PLEG): container finished" podID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerID="22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b" exitCode=0
Sep 16 20:16:30.971325 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.971154    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"
Sep 16 20:16:30.971325 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.971196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" event={"ID":"e83063b0-e0cc-469f-8e59-98a06196ac12","Type":"ContainerDied","Data":"22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b"}
Sep 16 20:16:30.971325 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.971250    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g" event={"ID":"e83063b0-e0cc-469f-8e59-98a06196ac12","Type":"ContainerDied","Data":"1ba27b8e8c20baf2d689cb8bd0fed361a3d4c3a60948fbf910a8e29313afbeed"}
Sep 16 20:16:30.971325 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.971274    2583 scope.go:117] "RemoveContainer" containerID="22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b"
Sep 16 20:16:30.980499 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.980481    2583 scope.go:117] "RemoveContainer" containerID="22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b"
Sep 16 20:16:30.980754 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:16:30.980738    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b\": container with ID starting with 22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b not found: ID does not exist" containerID="22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b"
Sep 16 20:16:30.980805 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.980761    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b"} err="failed to get container status \"22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b\": rpc error: code = NotFound desc = could not find container \"22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b\": container with ID starting with 22d649c5fef982f17246362812b1b93f927cef100e9c1c739b45cd0a4d8fb84b not found: ID does not exist"
Sep 16 20:16:30.993846 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.993822    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"]
Sep 16 20:16:30.998656 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:30.998636    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-67cffg857g"]
Sep 16 20:16:31.411024 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:31.410992    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" path="/var/lib/kubelet/pods/e83063b0-e0cc-469f-8e59-98a06196ac12/volumes"
Sep 16 20:16:35.026574 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.026543    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"]
Sep 16 20:16:35.026992 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.026971    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="llm-d-routing-sidecar"
Sep 16 20:16:35.027037 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.026997    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="llm-d-routing-sidecar"
Sep 16 20:16:35.027037 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.027021    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="storage-initializer"
Sep 16 20:16:35.027037 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.027031    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="storage-initializer"
Sep 16 20:16:35.027137 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.027055    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main"
Sep 16 20:16:35.027137 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.027064    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main"
Sep 16 20:16:35.027137 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.027081    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerName="main"
Sep 16 20:16:35.027137 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.027089    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerName="main"
Sep 16 20:16:35.027260 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.027181    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="llm-d-routing-sidecar"
Sep 16 20:16:35.027260 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.027197    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83063b0-e0cc-469f-8e59-98a06196ac12" containerName="main"
Sep 16 20:16:35.027260 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.027207    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a458d7-49a5-4349-a7e1-788447ce3c42" containerName="main"
Sep 16 20:16:35.031850 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.031829    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.037995 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.037966    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 20:16:35.038464 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.038443    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 16 20:16:35.038581 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.038524    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-hkrzs\""
Sep 16 20:16:35.039323 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.039289    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 20:16:35.059376 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.059351    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"]
Sep 16 20:16:35.220258 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.220228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.220400 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.220262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.220400 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.220283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.220400 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.220374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.220538 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.220430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.220538 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.220447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpp4\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-kube-api-access-tmpp4\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.220538 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.220496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.220639 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.220549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.220639 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.220577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174316d-c60f-4130-aec4-c9d6e316f0c2-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.321145 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.321145 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpp4\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-kube-api-access-tmpp4\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.321339 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.321339 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.321339 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174316d-c60f-4130-aec4-c9d6e316f0c2-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.321339 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.321339 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.321339 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.321339 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.322255 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.322255 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.322255 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.321888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.322255 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.322027    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174316d-c60f-4130-aec4-c9d6e316f0c2-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.322255 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.322121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.323549 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.323530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.323907 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.323892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.330895 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.330875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpp4\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-kube-api-access-tmpp4\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.331140 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.331112    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-5wzzp\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.343023 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.343005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:35.626257 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.626230    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"]
Sep 16 20:16:35.628355 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:16:35.628329    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb174316d_c60f_4130_aec4_c9d6e316f0c2.slice/crio-d30bf25f51a220082be4ad8b6b8bc52fadabbc9a73c62825a96aa813da26c799 WatchSource:0}: Error finding container d30bf25f51a220082be4ad8b6b8bc52fadabbc9a73c62825a96aa813da26c799: Status 404 returned error can't find the container with id d30bf25f51a220082be4ad8b6b8bc52fadabbc9a73c62825a96aa813da26c799
Sep 16 20:16:35.631074 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.631037    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:16:35.631150 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.631119    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:16:35.631191 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:35.631157    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:16:36.003262 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:36.003176    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" event={"ID":"b174316d-c60f-4130-aec4-c9d6e316f0c2","Type":"ContainerStarted","Data":"c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8"}
Sep 16 20:16:36.003262 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:36.003237    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" event={"ID":"b174316d-c60f-4130-aec4-c9d6e316f0c2","Type":"ContainerStarted","Data":"d30bf25f51a220082be4ad8b6b8bc52fadabbc9a73c62825a96aa813da26c799"}
Sep 16 20:16:36.034981 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:36.034937    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" podStartSLOduration=2.034921906 podStartE2EDuration="2.034921906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:16:36.032259549 +0000 UTC m=+4273.158063936" watchObservedRunningTime="2026-09-16 20:16:36.034921906 +0000 UTC m=+4273.160726294"
Sep 16 20:16:36.343872 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:36.343788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:36.348492 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:36.348468    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:37.009031 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:37.008995    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:37.009790 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:37.009767    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:49.076241 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:49.076202    2583 generic.go:356] "Generic (PLEG): container finished" podID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerID="c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8" exitCode=0
Sep 16 20:16:49.076241 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:49.076247    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" event={"ID":"b174316d-c60f-4130-aec4-c9d6e316f0c2","Type":"ContainerDied","Data":"c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8"}
Sep 16 20:16:49.076743 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:49.076625    2583 scope.go:117] "RemoveContainer" containerID="c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8"
Sep 16 20:16:49.077559 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:49.077527    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:16:49.077641 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:49.077600    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:16:49.077686 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:49.077665    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 20:16:50.082435 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:50.082382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" event={"ID":"b174316d-c60f-4130-aec4-c9d6e316f0c2","Type":"ContainerStarted","Data":"9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f"}
Sep 16 20:16:50.343236 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:50.343142    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:50.343982 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:50.343954    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 20:16:51.343873 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:51.343826    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 20:16:52.009498 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:52.009460    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:52.343979 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:52.343892    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 20:16:53.343527 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:53.343481    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 20:16:54.343951 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:54.343905    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 16 20:16:55.347463 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:55.347397    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:16:55.348454 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:16:55.348435    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:17:06.023798 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.023757    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"]
Sep 16 20:17:06.027196 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.027171    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.030506 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.030478    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:17:06.030649 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.030482    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 20:17:06.031112 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.031094    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 20:17:06.031333 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.031312    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-h5w44\""
Sep 16 20:17:06.031467 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.031386    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-rkmk2\""
Sep 16 20:17:06.046189 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.046162    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"]
Sep 16 20:17:06.068322 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.068290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-dshm\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.068481 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.068334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.068481 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.068372    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.068481 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.068437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c08b1ed-5f36-4944-80d2-52113041133b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.068481 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.068464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-home\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.068719 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.068530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-model-cache\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.068719 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.068597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c08b1ed-5f36-4944-80d2-52113041133b-tls-certs\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.068719 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.068629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2d4f\" (UniqueName: \"kubernetes.io/projected/0c08b1ed-5f36-4944-80d2-52113041133b-kube-api-access-n2d4f\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.169495 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-dshm\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.169495 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.169710 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.169710 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c08b1ed-5f36-4944-80d2-52113041133b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.169710 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169569    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-home\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.169710 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-model-cache\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.169710 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c08b1ed-5f36-4944-80d2-52113041133b-tls-certs\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.169710 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2d4f\" (UniqueName: \"kubernetes.io/projected/0c08b1ed-5f36-4944-80d2-52113041133b-kube-api-access-n2d4f\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.170035 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.170035 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.169965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.170143 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.170040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-model-cache\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.170248 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.170220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-home\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.170478 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.170459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c08b1ed-5f36-4944-80d2-52113041133b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.172136 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.172111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-dshm\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.172334 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.172316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c08b1ed-5f36-4944-80d2-52113041133b-tls-certs\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.187120 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.187099    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2d4f\" (UniqueName: \"kubernetes.io/projected/0c08b1ed-5f36-4944-80d2-52113041133b-kube-api-access-n2d4f\") pod \"router-with-refs-pd-test-kserve-57b69fc66f-s9bjs\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.339628 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.339553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:06.625302 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:06.625269    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"]
Sep 16 20:17:06.627682 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:17:06.627649    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c08b1ed_5f36_4944_80d2_52113041133b.slice/crio-81f804cf8daf33a0464a99abbc4d4ac2a814208792432d69b071eb2f86364928 WatchSource:0}: Error finding container 81f804cf8daf33a0464a99abbc4d4ac2a814208792432d69b071eb2f86364928: Status 404 returned error can't find the container with id 81f804cf8daf33a0464a99abbc4d4ac2a814208792432d69b071eb2f86364928
Sep 16 20:17:07.169488 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:07.169451    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" event={"ID":"0c08b1ed-5f36-4944-80d2-52113041133b","Type":"ContainerStarted","Data":"831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0"}
Sep 16 20:17:07.169960 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:07.169518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" event={"ID":"0c08b1ed-5f36-4944-80d2-52113041133b","Type":"ContainerStarted","Data":"81f804cf8daf33a0464a99abbc4d4ac2a814208792432d69b071eb2f86364928"}
Sep 16 20:17:07.169960 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:07.169554    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:08.175971 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:08.175931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" event={"ID":"0c08b1ed-5f36-4944-80d2-52113041133b","Type":"ContainerStarted","Data":"a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f"}
Sep 16 20:17:10.189098 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:10.189065    2583 generic.go:356] "Generic (PLEG): container finished" podID="0c08b1ed-5f36-4944-80d2-52113041133b" containerID="a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f" exitCode=0
Sep 16 20:17:10.189505 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:10.189144    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" event={"ID":"0c08b1ed-5f36-4944-80d2-52113041133b","Type":"ContainerDied","Data":"a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f"}
Sep 16 20:17:11.195812 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:11.195776    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" event={"ID":"0c08b1ed-5f36-4944-80d2-52113041133b","Type":"ContainerStarted","Data":"affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e"}
Sep 16 20:17:11.231658 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:11.231597    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podStartSLOduration=4.04351107 podStartE2EDuration="6.231580646s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.188069576s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:17:11.223612547 +0000 UTC m=+4308.349416932" watchObservedRunningTime="2026-09-16 20:17:11.231580646 +0000 UTC m=+4308.357385082"
Sep 16 20:17:16.340549 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:16.340459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:16.340549 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:16.340546    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:16.341823 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:16.341787    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:17:16.353775 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:16.353755    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:17:26.340613 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:26.340569    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:17:36.340562 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:36.340506    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:17:46.340261 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:46.340207    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:17:56.340604 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:17:56.340554    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:18:06.339917 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:06.339865    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:18:16.340863 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:16.340801    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:18:26.339941 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:26.339897    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:18:36.340267 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:36.340221    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:18:37.741301 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.741267    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"]
Sep 16 20:18:37.744191 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.744174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.762799 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.762770    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"]
Sep 16 20:18:37.799377 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.799338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-home\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.799377 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.799374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.799642 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.799390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22lj\" (UniqueName: \"kubernetes.io/projected/0434cb1d-5678-4491-940d-301834b2f7bf-kube-api-access-w22lj\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.799642 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.799480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.799642 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.799502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0434cb1d-5678-4491-940d-301834b2f7bf-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.799642 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.799532    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.799642 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.799547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0434cb1d-5678-4491-940d-301834b2f7bf-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.799642 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.799600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.900663 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.900625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.900663 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.900666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0434cb1d-5678-4491-940d-301834b2f7bf-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.900853 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.900693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.900853 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.900769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0434cb1d-5678-4491-940d-301834b2f7bf-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.900853 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.900841    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.900987 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.900890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-home\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.900987 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.900918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.900987 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.900940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22lj\" (UniqueName: \"kubernetes.io/projected/0434cb1d-5678-4491-940d-301834b2f7bf-kube-api-access-w22lj\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.901145 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.901093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.901348 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.901323    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.901705 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.901354    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-home\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.901705 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.901535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.901705 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.901596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0434cb1d-5678-4491-940d-301834b2f7bf-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.903237 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.903212    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.903429 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.903395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0434cb1d-5678-4491-940d-301834b2f7bf-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:37.917223 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:37.917197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22lj\" (UniqueName: \"kubernetes.io/projected/0434cb1d-5678-4491-940d-301834b2f7bf-kube-api-access-w22lj\") pod \"router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:38.054711 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:38.054629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:38.371363 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:38.371331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"]
Sep 16 20:18:38.372574 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:18:38.372543    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0434cb1d_5678_4491_940d_301834b2f7bf.slice/crio-eb3f4c8d35be7e49cfefc8955610e1293d59ad1642d698d5912ea859541b02e7 WatchSource:0}: Error finding container eb3f4c8d35be7e49cfefc8955610e1293d59ad1642d698d5912ea859541b02e7: Status 404 returned error can't find the container with id eb3f4c8d35be7e49cfefc8955610e1293d59ad1642d698d5912ea859541b02e7
Sep 16 20:18:38.630806 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:38.630710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" event={"ID":"0434cb1d-5678-4491-940d-301834b2f7bf","Type":"ContainerStarted","Data":"9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f"}
Sep 16 20:18:38.630806 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:38.630781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" event={"ID":"0434cb1d-5678-4491-940d-301834b2f7bf","Type":"ContainerStarted","Data":"eb3f4c8d35be7e49cfefc8955610e1293d59ad1642d698d5912ea859541b02e7"}
Sep 16 20:18:41.647549 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:41.647516    2583 generic.go:356] "Generic (PLEG): container finished" podID="0434cb1d-5678-4491-940d-301834b2f7bf" containerID="9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f" exitCode=0
Sep 16 20:18:41.648067 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:41.647602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" event={"ID":"0434cb1d-5678-4491-940d-301834b2f7bf","Type":"ContainerDied","Data":"9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f"}
Sep 16 20:18:42.655120 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:42.655083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" event={"ID":"0434cb1d-5678-4491-940d-301834b2f7bf","Type":"ContainerStarted","Data":"a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b"}
Sep 16 20:18:42.686656 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:42.686573    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podStartSLOduration=3.42217343 podStartE2EDuration="5.686553921s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.264380491s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:18:42.682168965 +0000 UTC m=+4399.807973353" watchObservedRunningTime="2026-09-16 20:18:42.686553921 +0000 UTC m=+4399.812358311"
Sep 16 20:18:46.340789 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:46.340678    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:18:48.055333 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:48.055288    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:48.055766 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:48.055357    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:18:48.057449 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:48.057398    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:18:54.566046 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.566007    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"]
Sep 16 20:18:54.570124 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.570099    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:18:54.573479 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.573452    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-wsfb2\""
Sep 16 20:18:54.574748 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.574721    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Sep 16 20:18:54.574847 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.574780    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-epp-sa-dockercfg-7hbhg\""
Sep 16 20:18:54.574847 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.574801    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 20:18:54.574969 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.574938    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Sep 16 20:18:54.589314 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.589281    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"]
Sep 16 20:18:54.672751 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.672695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz5z5\" (UniqueName: \"kubernetes.io/projected/118a2da6-c581-4038-9f4d-db53ca168bbe-kube-api-access-sz5z5\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq\" (UID: \"118a2da6-c581-4038-9f4d-db53ca168bbe\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:18:54.672941 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.672866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a2da6-c581-4038-9f4d-db53ca168bbe-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq\" (UID: \"118a2da6-c581-4038-9f4d-db53ca168bbe\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:18:54.773365 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.773333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz5z5\" (UniqueName: \"kubernetes.io/projected/118a2da6-c581-4038-9f4d-db53ca168bbe-kube-api-access-sz5z5\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq\" (UID: \"118a2da6-c581-4038-9f4d-db53ca168bbe\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:18:54.773581 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.773431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a2da6-c581-4038-9f4d-db53ca168bbe-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq\" (UID: \"118a2da6-c581-4038-9f4d-db53ca168bbe\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:18:54.776369 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.776334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a2da6-c581-4038-9f4d-db53ca168bbe-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq\" (UID: \"118a2da6-c581-4038-9f4d-db53ca168bbe\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:18:54.783687 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.783661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz5z5\" (UniqueName: \"kubernetes.io/projected/118a2da6-c581-4038-9f4d-db53ca168bbe-kube-api-access-sz5z5\") pod \"tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq\" (UID: \"118a2da6-c581-4038-9f4d-db53ca168bbe\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:18:54.883598 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:54.883564    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:18:55.186951 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:55.186917    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"]
Sep 16 20:18:55.189022 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:18:55.188988    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118a2da6_c581_4038_9f4d_db53ca168bbe.slice/crio-8c44c547b70283a278b1c3eeebe70a6bc1b22d07acb7b428314945ee7b53872d WatchSource:0}: Error finding container 8c44c547b70283a278b1c3eeebe70a6bc1b22d07acb7b428314945ee7b53872d: Status 404 returned error can't find the container with id 8c44c547b70283a278b1c3eeebe70a6bc1b22d07acb7b428314945ee7b53872d
Sep 16 20:18:55.731679 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:55.731628    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" event={"ID":"118a2da6-c581-4038-9f4d-db53ca168bbe","Type":"ContainerStarted","Data":"374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b"}
Sep 16 20:18:55.732221 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:55.731719    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" event={"ID":"118a2da6-c581-4038-9f4d-db53ca168bbe","Type":"ContainerStarted","Data":"8c44c547b70283a278b1c3eeebe70a6bc1b22d07acb7b428314945ee7b53872d"}
Sep 16 20:18:55.732553 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:55.732527    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:18:55.757799 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:55.757741    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" podStartSLOduration=1.75772296 podStartE2EDuration="1.75772296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:18:55.755891011 +0000 UTC m=+4412.881695398" watchObservedRunningTime="2026-09-16 20:18:55.75772296 +0000 UTC m=+4412.883527348"
Sep 16 20:18:56.340607 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:56.340537    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 20:18:58.055286 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:18:58.055246    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:19:06.356257 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:19:06.356223    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:19:06.368012 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:19:06.367983    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:19:08.055226 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:19:08.055184    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:19:18.055367 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:19:18.055330    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:19:27.743331 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:19:27.743301    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:19:28.055180 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:19:28.055080    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:19:38.055785 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:19:38.055738    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:19:48.055516 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:19:48.055473    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:19:58.055812 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:19:58.055709    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:20:08.055511 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:08.055460    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:20:18.055888 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:18.055777    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 20:20:19.666372 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:19.666329    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"]
Sep 16 20:20:19.689660 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:19.689627    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"]
Sep 16 20:20:27.134686 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:27.134654    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:20:27.142619 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:27.142593    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:20:27.149686 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:27.149666    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:20:27.155582 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:27.155564    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:20:34.690600 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:34.690545    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main" containerID="cri-o://a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b" gracePeriod=45
Sep 16 20:20:34.714604 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:34.714565    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main" containerID="cri-o://affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e" gracePeriod=45
Sep 16 20:20:35.444685 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.444623    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="llm-d-routing-sidecar" containerID="cri-o://831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0" gracePeriod=45
Sep 16 20:20:35.564392 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.564364    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:20:35.602448 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602400    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-kserve-provision-location\") pod \"0434cb1d-5678-4491-940d-301834b2f7bf\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") "
Sep 16 20:20:35.602603 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602494    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-home\") pod \"0434cb1d-5678-4491-940d-301834b2f7bf\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") "
Sep 16 20:20:35.602603 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0434cb1d-5678-4491-940d-301834b2f7bf-cabundle-cert\") pod \"0434cb1d-5678-4491-940d-301834b2f7bf\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") "
Sep 16 20:20:35.602603 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602556    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-model-cache\") pod \"0434cb1d-5678-4491-940d-301834b2f7bf\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") "
Sep 16 20:20:35.602603 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602595    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w22lj\" (UniqueName: \"kubernetes.io/projected/0434cb1d-5678-4491-940d-301834b2f7bf-kube-api-access-w22lj\") pod \"0434cb1d-5678-4491-940d-301834b2f7bf\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") "
Sep 16 20:20:35.602867 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602635    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0434cb1d-5678-4491-940d-301834b2f7bf-tls-certs\") pod \"0434cb1d-5678-4491-940d-301834b2f7bf\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") "
Sep 16 20:20:35.602867 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602717    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-tmp-dir\") pod \"0434cb1d-5678-4491-940d-301834b2f7bf\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") "
Sep 16 20:20:35.602867 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602801    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-dshm\") pod \"0434cb1d-5678-4491-940d-301834b2f7bf\" (UID: \"0434cb1d-5678-4491-940d-301834b2f7bf\") "
Sep 16 20:20:35.603040 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602878    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "0434cb1d-5678-4491-940d-301834b2f7bf" (UID: "0434cb1d-5678-4491-940d-301834b2f7bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.603040 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.602992    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0434cb1d-5678-4491-940d-301834b2f7bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0434cb1d-5678-4491-940d-301834b2f7bf" (UID: "0434cb1d-5678-4491-940d-301834b2f7bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:20:35.603277 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.603248    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-home" (OuterVolumeSpecName: "home") pod "0434cb1d-5678-4491-940d-301834b2f7bf" (UID: "0434cb1d-5678-4491-940d-301834b2f7bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.603372 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.603304    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0434cb1d-5678-4491-940d-301834b2f7bf-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.603372 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.603326    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.606014 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.605991    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0434cb1d-5678-4491-940d-301834b2f7bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0434cb1d-5678-4491-940d-301834b2f7bf" (UID: "0434cb1d-5678-4491-940d-301834b2f7bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:20:35.606092 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.605990    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0434cb1d-5678-4491-940d-301834b2f7bf-kube-api-access-w22lj" (OuterVolumeSpecName: "kube-api-access-w22lj") pod "0434cb1d-5678-4491-940d-301834b2f7bf" (UID: "0434cb1d-5678-4491-940d-301834b2f7bf"). InnerVolumeSpecName "kube-api-access-w22lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:20:35.606092 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.606050    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-dshm" (OuterVolumeSpecName: "dshm") pod "0434cb1d-5678-4491-940d-301834b2f7bf" (UID: "0434cb1d-5678-4491-940d-301834b2f7bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.618303 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.618277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0434cb1d-5678-4491-940d-301834b2f7bf" (UID: "0434cb1d-5678-4491-940d-301834b2f7bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.625697 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.625670    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0434cb1d-5678-4491-940d-301834b2f7bf" (UID: "0434cb1d-5678-4491-940d-301834b2f7bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.685798 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.685779    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:20:35.704045 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704023    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-dshm\") pod \"0c08b1ed-5f36-4944-80d2-52113041133b\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") "
Sep 16 20:20:35.704353 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-kserve-provision-location\") pod \"0c08b1ed-5f36-4944-80d2-52113041133b\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") "
Sep 16 20:20:35.704353 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704093    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c08b1ed-5f36-4944-80d2-52113041133b-tls-certs\") pod \"0c08b1ed-5f36-4944-80d2-52113041133b\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") "
Sep 16 20:20:35.704353 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704117    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c08b1ed-5f36-4944-80d2-52113041133b-cabundle-cert\") pod \"0c08b1ed-5f36-4944-80d2-52113041133b\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") "
Sep 16 20:20:35.704353 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704179    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-model-cache\") pod \"0c08b1ed-5f36-4944-80d2-52113041133b\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") "
Sep 16 20:20:35.704353 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704196    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-tmp-dir\") pod \"0c08b1ed-5f36-4944-80d2-52113041133b\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") "
Sep 16 20:20:35.704353 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704214    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2d4f\" (UniqueName: \"kubernetes.io/projected/0c08b1ed-5f36-4944-80d2-52113041133b-kube-api-access-n2d4f\") pod \"0c08b1ed-5f36-4944-80d2-52113041133b\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") "
Sep 16 20:20:35.704353 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-home\") pod \"0c08b1ed-5f36-4944-80d2-52113041133b\" (UID: \"0c08b1ed-5f36-4944-80d2-52113041133b\") "
Sep 16 20:20:35.704760 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704514    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0434cb1d-5678-4491-940d-301834b2f7bf-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.704760 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704520    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c08b1ed-5f36-4944-80d2-52113041133b" (UID: "0c08b1ed-5f36-4944-80d2-52113041133b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.704760 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704530    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.704760 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704581    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.704760 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704598    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.704760 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704613    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0434cb1d-5678-4491-940d-301834b2f7bf-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.704760 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704627    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22lj\" (UniqueName: \"kubernetes.io/projected/0434cb1d-5678-4491-940d-301834b2f7bf-kube-api-access-w22lj\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.704760 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.704632    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c08b1ed-5f36-4944-80d2-52113041133b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c08b1ed-5f36-4944-80d2-52113041133b" (UID: "0c08b1ed-5f36-4944-80d2-52113041133b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:20:35.705351 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.705328    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-home" (OuterVolumeSpecName: "home") pod "0c08b1ed-5f36-4944-80d2-52113041133b" (UID: "0c08b1ed-5f36-4944-80d2-52113041133b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.706441 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.706396    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-dshm" (OuterVolumeSpecName: "dshm") pod "0c08b1ed-5f36-4944-80d2-52113041133b" (UID: "0c08b1ed-5f36-4944-80d2-52113041133b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.706666 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.706649    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c08b1ed-5f36-4944-80d2-52113041133b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c08b1ed-5f36-4944-80d2-52113041133b" (UID: "0c08b1ed-5f36-4944-80d2-52113041133b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:20:35.706891 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.706875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c08b1ed-5f36-4944-80d2-52113041133b-kube-api-access-n2d4f" (OuterVolumeSpecName: "kube-api-access-n2d4f") pod "0c08b1ed-5f36-4944-80d2-52113041133b" (UID: "0c08b1ed-5f36-4944-80d2-52113041133b"). InnerVolumeSpecName "kube-api-access-n2d4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:20:35.718268 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.718243    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c08b1ed-5f36-4944-80d2-52113041133b" (UID: "0c08b1ed-5f36-4944-80d2-52113041133b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.730058 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.730033    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c08b1ed-5f36-4944-80d2-52113041133b" (UID: "0c08b1ed-5f36-4944-80d2-52113041133b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:35.805357 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.805328    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.805357 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.805354    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.805538 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.805368    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c08b1ed-5f36-4944-80d2-52113041133b-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.805538 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.805382    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c08b1ed-5f36-4944-80d2-52113041133b-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.805538 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.805393    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.805538 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.805405    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.805538 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.805446    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2d4f\" (UniqueName: \"kubernetes.io/projected/0c08b1ed-5f36-4944-80d2-52113041133b-kube-api-access-n2d4f\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:35.805538 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:35.805458    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c08b1ed-5f36-4944-80d2-52113041133b-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:20:36.255390 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.255355    2583 generic.go:356] "Generic (PLEG): container finished" podID="0c08b1ed-5f36-4944-80d2-52113041133b" containerID="affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e" exitCode=0
Sep 16 20:20:36.255390 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.255382    2583 generic.go:356] "Generic (PLEG): container finished" podID="0c08b1ed-5f36-4944-80d2-52113041133b" containerID="831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0" exitCode=0
Sep 16 20:20:36.255623 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.255438    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"
Sep 16 20:20:36.255623 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.255443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" event={"ID":"0c08b1ed-5f36-4944-80d2-52113041133b","Type":"ContainerDied","Data":"affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e"}
Sep 16 20:20:36.255623 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.255497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" event={"ID":"0c08b1ed-5f36-4944-80d2-52113041133b","Type":"ContainerDied","Data":"831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0"}
Sep 16 20:20:36.255623 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.255514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs" event={"ID":"0c08b1ed-5f36-4944-80d2-52113041133b","Type":"ContainerDied","Data":"81f804cf8daf33a0464a99abbc4d4ac2a814208792432d69b071eb2f86364928"}
Sep 16 20:20:36.255623 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.255535    2583 scope.go:117] "RemoveContainer" containerID="affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e"
Sep 16 20:20:36.257282 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.257260    2583 generic.go:356] "Generic (PLEG): container finished" podID="0434cb1d-5678-4491-940d-301834b2f7bf" containerID="a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b" exitCode=0
Sep 16 20:20:36.257430 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.257308    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"
Sep 16 20:20:36.257500 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.257397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" event={"ID":"0434cb1d-5678-4491-940d-301834b2f7bf","Type":"ContainerDied","Data":"a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b"}
Sep 16 20:20:36.257500 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.257470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6" event={"ID":"0434cb1d-5678-4491-940d-301834b2f7bf","Type":"ContainerDied","Data":"eb3f4c8d35be7e49cfefc8955610e1293d59ad1642d698d5912ea859541b02e7"}
Sep 16 20:20:36.266252 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.266224    2583 scope.go:117] "RemoveContainer" containerID="a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f"
Sep 16 20:20:36.274786 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.274771    2583 scope.go:117] "RemoveContainer" containerID="831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0"
Sep 16 20:20:36.285342 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.285311    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"]
Sep 16 20:20:36.285843 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.285812    2583 scope.go:117] "RemoveContainer" containerID="affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e"
Sep 16 20:20:36.286222 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:20:36.286190    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e\": container with ID starting with affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e not found: ID does not exist" containerID="affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e"
Sep 16 20:20:36.286323 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.286229    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e"} err="failed to get container status \"affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e\": rpc error: code = NotFound desc = could not find container \"affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e\": container with ID starting with affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e not found: ID does not exist"
Sep 16 20:20:36.286323 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.286248    2583 scope.go:117] "RemoveContainer" containerID="a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f"
Sep 16 20:20:36.286667 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:20:36.286648    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f\": container with ID starting with a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f not found: ID does not exist" containerID="a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f"
Sep 16 20:20:36.286736 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.286674    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f"} err="failed to get container status \"a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f\": rpc error: code = NotFound desc = could not find container \"a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f\": container with ID starting with a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f not found: ID does not exist"
Sep 16 20:20:36.286736 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.286689    2583 scope.go:117] "RemoveContainer" containerID="831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0"
Sep 16 20:20:36.286994 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:20:36.286967    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0\": container with ID starting with 831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0 not found: ID does not exist" containerID="831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0"
Sep 16 20:20:36.286994 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.286989    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0"} err="failed to get container status \"831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0\": rpc error: code = NotFound desc = could not find container \"831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0\": container with ID starting with 831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0 not found: ID does not exist"
Sep 16 20:20:36.287154 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.287004    2583 scope.go:117] "RemoveContainer" containerID="affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e"
Sep 16 20:20:36.287268 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.287252    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e"} err="failed to get container status \"affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e\": rpc error: code = NotFound desc = could not find container \"affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e\": container with ID starting with affffad92b0c77bc8dca11953994afd4d5b337897f0db93cb444d3080ab1585e not found: ID does not exist"
Sep 16 20:20:36.287322 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.287270    2583 scope.go:117] "RemoveContainer" containerID="a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f"
Sep 16 20:20:36.287590 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.287568    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f"} err="failed to get container status \"a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f\": rpc error: code = NotFound desc = could not find container \"a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f\": container with ID starting with a5f3d0ad27ed6b19905915de0644e8435f4ecde8dacdd48316d6ec7a4852856f not found: ID does not exist"
Sep 16 20:20:36.287667 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.287591    2583 scope.go:117] "RemoveContainer" containerID="831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0"
Sep 16 20:20:36.287831 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.287811    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0"} err="failed to get container status \"831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0\": rpc error: code = NotFound desc = could not find container \"831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0\": container with ID starting with 831a8688b5a450dff90d8396c1e1fb966d447dc51ac97a08de33c1e909b800a0 not found: ID does not exist"
Sep 16 20:20:36.287908 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.287834    2583 scope.go:117] "RemoveContainer" containerID="a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b"
Sep 16 20:20:36.288250 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.288233    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-57b69fc66f-s9bjs"]
Sep 16 20:20:36.295382 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.295367    2583 scope.go:117] "RemoveContainer" containerID="9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f"
Sep 16 20:20:36.301329 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.301303    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"]
Sep 16 20:20:36.303269 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.303253    2583 scope.go:117] "RemoveContainer" containerID="a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b"
Sep 16 20:20:36.303570 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:20:36.303554    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b\": container with ID starting with a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b not found: ID does not exist" containerID="a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b"
Sep 16 20:20:36.303655 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.303578    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b"} err="failed to get container status \"a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b\": rpc error: code = NotFound desc = could not find container \"a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b\": container with ID starting with a9c570a97305971f52f57ef44bc0cc3c82cfdac2c94c122748d8a3ebf529f36b not found: ID does not exist"
Sep 16 20:20:36.303655 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.303597    2583 scope.go:117] "RemoveContainer" containerID="9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f"
Sep 16 20:20:36.303846 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:20:36.303829    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f\": container with ID starting with 9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f not found: ID does not exist" containerID="9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f"
Sep 16 20:20:36.303886 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.303851    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f"} err="failed to get container status \"9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f\": rpc error: code = NotFound desc = could not find container \"9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f\": container with ID starting with 9536be1c8ddbd98265b365573ef63ee125c3fea2eac0e0664cb44787737d484f not found: ID does not exist"
Sep 16 20:20:36.309198 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:36.309176    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8464db8854-s2hg6"]
Sep 16 20:20:37.413202 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:37.413164    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" path="/var/lib/kubelet/pods/0434cb1d-5678-4491-940d-301834b2f7bf/volumes"
Sep 16 20:20:37.413849 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:20:37.413825    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" path="/var/lib/kubelet/pods/0c08b1ed-5f36-4944-80d2-52113041133b/volumes"
Sep 16 20:21:29.879986 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:29.879939    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"]
Sep 16 20:21:29.880606 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:29.880195    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy" containerID="cri-o://9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f" gracePeriod=30
Sep 16 20:21:35.129980 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.129955    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:21:35.180989 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.180900    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174316d-c60f-4130-aec4-c9d6e316f0c2-istiod-ca-cert\") pod \"b174316d-c60f-4130-aec4-c9d6e316f0c2\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") "
Sep 16 20:21:35.181460 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.181405    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b174316d-c60f-4130-aec4-c9d6e316f0c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b174316d-c60f-4130-aec4-c9d6e316f0c2" (UID: "b174316d-c60f-4130-aec4-c9d6e316f0c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:21:35.181714 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.181692    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-token\") pod \"b174316d-c60f-4130-aec4-c9d6e316f0c2\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") "
Sep 16 20:21:35.181963 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.181946    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-envoy\") pod \"b174316d-c60f-4130-aec4-c9d6e316f0c2\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") "
Sep 16 20:21:35.182182 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.182166    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-credential-socket\") pod \"b174316d-c60f-4130-aec4-c9d6e316f0c2\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") "
Sep 16 20:21:35.182358 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.182343    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-podinfo\") pod \"b174316d-c60f-4130-aec4-c9d6e316f0c2\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") "
Sep 16 20:21:35.182578 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.182457    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b174316d-c60f-4130-aec4-c9d6e316f0c2" (UID: "b174316d-c60f-4130-aec4-c9d6e316f0c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:21:35.182689 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.182560    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmpp4\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-kube-api-access-tmpp4\") pod \"b174316d-c60f-4130-aec4-c9d6e316f0c2\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") "
Sep 16 20:21:35.182775 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.182687    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-data\") pod \"b174316d-c60f-4130-aec4-c9d6e316f0c2\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") "
Sep 16 20:21:35.183441 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.183375    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "b174316d-c60f-4130-aec4-c9d6e316f0c2" (UID: "b174316d-c60f-4130-aec4-c9d6e316f0c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:21:35.186591 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.184474    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-socket\") pod \"b174316d-c60f-4130-aec4-c9d6e316f0c2\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") "
Sep 16 20:21:35.186591 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.184560    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-certs\") pod \"b174316d-c60f-4130-aec4-c9d6e316f0c2\" (UID: \"b174316d-c60f-4130-aec4-c9d6e316f0c2\") "
Sep 16 20:21:35.186591 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.185139    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-credential-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:21:35.186591 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.185209    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-data\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:21:35.186591 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.185225    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b174316d-c60f-4130-aec4-c9d6e316f0c2-istiod-ca-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:21:35.186591 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.185540    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b174316d-c60f-4130-aec4-c9d6e316f0c2" (UID: "b174316d-c60f-4130-aec4-c9d6e316f0c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:21:35.186591 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.185796    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b174316d-c60f-4130-aec4-c9d6e316f0c2" (UID: "b174316d-c60f-4130-aec4-c9d6e316f0c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:21:35.187639 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.187594    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b174316d-c60f-4130-aec4-c9d6e316f0c2" (UID: "b174316d-c60f-4130-aec4-c9d6e316f0c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 20:21:35.187639 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.187603    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "b174316d-c60f-4130-aec4-c9d6e316f0c2" (UID: "b174316d-c60f-4130-aec4-c9d6e316f0c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:21:35.187639 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.187606    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b174316d-c60f-4130-aec4-c9d6e316f0c2" (UID: "b174316d-c60f-4130-aec4-c9d6e316f0c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:21:35.188712 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.188687    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-kube-api-access-tmpp4" (OuterVolumeSpecName: "kube-api-access-tmpp4") pod "b174316d-c60f-4130-aec4-c9d6e316f0c2" (UID: "b174316d-c60f-4130-aec4-c9d6e316f0c2"). InnerVolumeSpecName "kube-api-access-tmpp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:21:35.286558 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.286529    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:21:35.286558 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.286553    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-token\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:21:35.286558 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.286562    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-envoy\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:21:35.286730 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.286570    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b174316d-c60f-4130-aec4-c9d6e316f0c2-istio-podinfo\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:21:35.286730 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.286578    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmpp4\" (UniqueName: \"kubernetes.io/projected/b174316d-c60f-4130-aec4-c9d6e316f0c2-kube-api-access-tmpp4\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:21:35.286730 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.286586    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b174316d-c60f-4130-aec4-c9d6e316f0c2-workload-socket\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:21:35.543881 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.543784    2583 generic.go:356] "Generic (PLEG): container finished" podID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerID="9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f" exitCode=0
Sep 16 20:21:35.543881 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.543836    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"
Sep 16 20:21:35.543881 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.543849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" event={"ID":"b174316d-c60f-4130-aec4-c9d6e316f0c2","Type":"ContainerDied","Data":"9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f"}
Sep 16 20:21:35.544160 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.543899    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp" event={"ID":"b174316d-c60f-4130-aec4-c9d6e316f0c2","Type":"ContainerDied","Data":"d30bf25f51a220082be4ad8b6b8bc52fadabbc9a73c62825a96aa813da26c799"}
Sep 16 20:21:35.544160 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.543923    2583 scope.go:117] "RemoveContainer" containerID="9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f"
Sep 16 20:21:35.553149 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.553129    2583 scope.go:117] "RemoveContainer" containerID="c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8"
Sep 16 20:21:35.563628 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.563604    2583 scope.go:117] "RemoveContainer" containerID="9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f"
Sep 16 20:21:35.563939 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.563917    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"]
Sep 16 20:21:35.564011 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:21:35.563919    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f\": container with ID starting with 9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f not found: ID does not exist" containerID="9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f"
Sep 16 20:21:35.564011 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.563967    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f"} err="failed to get container status \"9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f\": rpc error: code = NotFound desc = could not find container \"9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f\": container with ID starting with 9a1cecc18d546eae215cf2c1127eb244ceb73b56293c1888bfbe96e1436d010f not found: ID does not exist"
Sep 16 20:21:35.564011 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.563983    2583 scope.go:117] "RemoveContainer" containerID="c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8"
Sep 16 20:21:35.564267 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:21:35.564236    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8\": container with ID starting with c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8 not found: ID does not exist" containerID="c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8"
Sep 16 20:21:35.564307 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.564278    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8"} err="failed to get container status \"c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8\": rpc error: code = NotFound desc = could not find container \"c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8\": container with ID starting with c59b4fc9666836b956c8bb1b7cad931861a0b5cfa8e57bc17bf027464ba67cf8 not found: ID does not exist"
Sep 16 20:21:35.567714 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:35.567689    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-5wzzp"]
Sep 16 20:21:37.411105 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:37.411069    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" path="/var/lib/kubelet/pods/b174316d-c60f-4130-aec4-c9d6e316f0c2/volumes"
Sep 16 20:21:46.478824 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.478747    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"]
Sep 16 20:21:46.479168 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479112    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy"
Sep 16 20:21:46.479168 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479125    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy"
Sep 16 20:21:46.479168 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479135    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="storage-initializer"
Sep 16 20:21:46.479168 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479141    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="storage-initializer"
Sep 16 20:21:46.479168 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479157    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main"
Sep 16 20:21:46.479168 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479163    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479174    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479179    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479187    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="llm-d-routing-sidecar"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479192    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="llm-d-routing-sidecar"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479198    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="storage-initializer"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479203    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="storage-initializer"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479257    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479264    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0434cb1d-5678-4491-940d-301834b2f7bf" containerName="main"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479272    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="llm-d-routing-sidecar"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479278    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c08b1ed-5f36-4944-80d2-52113041133b" containerName="main"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479337    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy"
Sep 16 20:21:46.479362 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479344    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy"
Sep 16 20:21:46.479766 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.479393    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b174316d-c60f-4130-aec4-c9d6e316f0c2" containerName="istio-proxy"
Sep 16 20:21:46.482045 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.482027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.487720 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.487697    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 20:21:46.487893 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.487755    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 20:21:46.487893 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.487756    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 20:21:46.487986 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.487903    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 20:21:46.488081 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.487993    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:21:46.488122 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.488115    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-5dlgk\""
Sep 16 20:21:46.510167 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.510135    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"]
Sep 16 20:21:46.575776 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.575746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.575908 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.575779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.575908 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.575808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmpn\" (UniqueName: \"kubernetes.io/projected/d5853633-4f13-4bdf-8395-e9f1554692ac-kube-api-access-gqmpn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.575986 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.575922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5853633-4f13-4bdf-8395-e9f1554692ac-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.576030 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.575988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.576030 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.576008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.576099 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.576029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5853633-4f13-4bdf-8395-e9f1554692ac-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.576099 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.576048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.676885 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.676859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5853633-4f13-4bdf-8395-e9f1554692ac-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677050 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.676910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677050 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.676928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677050 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.676944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5853633-4f13-4bdf-8395-e9f1554692ac-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677050 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.676961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677050 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.676983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677050 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.677010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677050 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.677049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmpn\" (UniqueName: \"kubernetes.io/projected/d5853633-4f13-4bdf-8395-e9f1554692ac-kube-api-access-gqmpn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677489 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.677464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677588 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.677508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677588 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.677561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677670 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.677596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.677733 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.677716    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5853633-4f13-4bdf-8395-e9f1554692ac-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.679386 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.679357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.679858 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.679839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5853633-4f13-4bdf-8395-e9f1554692ac-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.686327 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.686306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmpn\" (UniqueName: \"kubernetes.io/projected/d5853633-4f13-4bdf-8395-e9f1554692ac-kube-api-access-gqmpn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:46.792813 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:46.792752    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:47.084534 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:47.084508    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"]
Sep 16 20:21:47.086545 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:21:47.086520    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5853633_4f13_4bdf_8395_e9f1554692ac.slice/crio-90e7feea7ed777c266d98b5cfeab9ab69edea86fd5b416908db64fc9e7f1c011 WatchSource:0}: Error finding container 90e7feea7ed777c266d98b5cfeab9ab69edea86fd5b416908db64fc9e7f1c011: Status 404 returned error can't find the container with id 90e7feea7ed777c266d98b5cfeab9ab69edea86fd5b416908db64fc9e7f1c011
Sep 16 20:21:47.600100 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:47.600062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" event={"ID":"d5853633-4f13-4bdf-8395-e9f1554692ac","Type":"ContainerStarted","Data":"8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec"}
Sep 16 20:21:47.600567 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:47.600126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" event={"ID":"d5853633-4f13-4bdf-8395-e9f1554692ac","Type":"ContainerStarted","Data":"90e7feea7ed777c266d98b5cfeab9ab69edea86fd5b416908db64fc9e7f1c011"}
Sep 16 20:21:49.434007 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:49.433965    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"]
Sep 16 20:21:49.611581 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:49.611548    2583 generic.go:356] "Generic (PLEG): container finished" podID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerID="8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec" exitCode=0
Sep 16 20:21:49.611730 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:49.611608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" event={"ID":"d5853633-4f13-4bdf-8395-e9f1554692ac","Type":"ContainerDied","Data":"8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec"}
Sep 16 20:21:50.618605 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:50.618568    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" event={"ID":"d5853633-4f13-4bdf-8395-e9f1554692ac","Type":"ContainerStarted","Data":"a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d"}
Sep 16 20:21:50.646388 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:50.646336    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podStartSLOduration=2.296898503 podStartE2EDuration="4.64631962s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.349421117s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:21:50.643175777 +0000 UTC m=+4587.768980166" watchObservedRunningTime="2026-09-16 20:21:50.64631962 +0000 UTC m=+4587.772124008"
Sep 16 20:21:50.793485 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:50.793446    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:56.793142 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:56.793105    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:21:56.794516 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:56.794482    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:21:57.742986 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:21:57.742951    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" podUID="118a2da6-c581-4038-9f4d-db53ca168bbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:22:04.454624 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:04.454572    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" podUID="118a2da6-c581-4038-9f4d-db53ca168bbe" containerName="main" containerID="cri-o://374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b" gracePeriod=45
Sep 16 20:22:06.793902 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:06.793856    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:22:07.743192 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:07.743158    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" podUID="118a2da6-c581-4038-9f4d-db53ca168bbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:22:16.793531 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:16.793492    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:22:17.743524 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:17.743480    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" podUID="118a2da6-c581-4038-9f4d-db53ca168bbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:22:17.743714 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:17.743617    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:22:26.793441 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:26.793388    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:22:27.743182 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:27.743134    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" podUID="118a2da6-c581-4038-9f4d-db53ca168bbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:22:34.712743 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.712678    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:22:34.815200 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.815170    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz5z5\" (UniqueName: \"kubernetes.io/projected/118a2da6-c581-4038-9f4d-db53ca168bbe-kube-api-access-sz5z5\") pod \"118a2da6-c581-4038-9f4d-db53ca168bbe\" (UID: \"118a2da6-c581-4038-9f4d-db53ca168bbe\") "
Sep 16 20:22:34.815388 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.815215    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a2da6-c581-4038-9f4d-db53ca168bbe-tls-certs\") pod \"118a2da6-c581-4038-9f4d-db53ca168bbe\" (UID: \"118a2da6-c581-4038-9f4d-db53ca168bbe\") "
Sep 16 20:22:34.817602 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.817567    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118a2da6-c581-4038-9f4d-db53ca168bbe-kube-api-access-sz5z5" (OuterVolumeSpecName: "kube-api-access-sz5z5") pod "118a2da6-c581-4038-9f4d-db53ca168bbe" (UID: "118a2da6-c581-4038-9f4d-db53ca168bbe"). InnerVolumeSpecName "kube-api-access-sz5z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:22:34.817713 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.817630    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118a2da6-c581-4038-9f4d-db53ca168bbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "118a2da6-c581-4038-9f4d-db53ca168bbe" (UID: "118a2da6-c581-4038-9f4d-db53ca168bbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:22:34.831173 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.831149    2583 generic.go:356] "Generic (PLEG): container finished" podID="118a2da6-c581-4038-9f4d-db53ca168bbe" containerID="374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b" exitCode=0
Sep 16 20:22:34.831278 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.831188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" event={"ID":"118a2da6-c581-4038-9f4d-db53ca168bbe","Type":"ContainerDied","Data":"374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b"}
Sep 16 20:22:34.831278 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.831213    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"
Sep 16 20:22:34.831278 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.831228    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq" event={"ID":"118a2da6-c581-4038-9f4d-db53ca168bbe","Type":"ContainerDied","Data":"8c44c547b70283a278b1c3eeebe70a6bc1b22d07acb7b428314945ee7b53872d"}
Sep 16 20:22:34.831278 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.831250    2583 scope.go:117] "RemoveContainer" containerID="374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b"
Sep 16 20:22:34.839664 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.839649    2583 scope.go:117] "RemoveContainer" containerID="374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b"
Sep 16 20:22:34.839944 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:22:34.839925    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b\": container with ID starting with 374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b not found: ID does not exist" containerID="374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b"
Sep 16 20:22:34.840013 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.839953    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b"} err="failed to get container status \"374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b\": rpc error: code = NotFound desc = could not find container \"374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b\": container with ID starting with 374446ae86c9cd4e9aa3924fbe69189e33645f9b1fa40c7a69afd314b7bfd11b not found: ID does not exist"
Sep 16 20:22:34.868223 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.868199    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"]
Sep 16 20:22:34.871695 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.871675    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-774dd49f6f-m9swq"]
Sep 16 20:22:34.916905 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.916872    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz5z5\" (UniqueName: \"kubernetes.io/projected/118a2da6-c581-4038-9f4d-db53ca168bbe-kube-api-access-sz5z5\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:22:34.916988 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:34.916907    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a2da6-c581-4038-9f4d-db53ca168bbe-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:22:35.413000 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:35.412965    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118a2da6-c581-4038-9f4d-db53ca168bbe" path="/var/lib/kubelet/pods/118a2da6-c581-4038-9f4d-db53ca168bbe/volumes"
Sep 16 20:22:36.793271 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:36.793222    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:22:46.793431 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:46.793380    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:22:56.794144 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:22:56.794104    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:23:06.793174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:06.793129    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:23:10.884546 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.884512    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"]
Sep 16 20:23:10.885051 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.885030    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118a2da6-c581-4038-9f4d-db53ca168bbe" containerName="main"
Sep 16 20:23:10.885103 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.885056    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="118a2da6-c581-4038-9f4d-db53ca168bbe" containerName="main"
Sep 16 20:23:10.885169 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.885156    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="118a2da6-c581-4038-9f4d-db53ca168bbe" containerName="main"
Sep 16 20:23:10.888389 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.888369    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:10.894260 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.894238    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-zcddt\""
Sep 16 20:23:10.894396 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.894282    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 20:23:10.894974 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.894750    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 20:23:10.894974 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.894800    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-45gvv\""
Sep 16 20:23:10.894974 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.894800    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 20:23:10.903808 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.903785    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"]
Sep 16 20:23:10.919895 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.919870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvwm4\" (UniqueName: \"kubernetes.io/projected/0417d34f-0032-43e3-8715-9f8af8446fc1-kube-api-access-kvwm4\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4\" (UID: \"0417d34f-0032-43e3-8715-9f8af8446fc1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:10.919996 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:10.919947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0417d34f-0032-43e3-8715-9f8af8446fc1-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4\" (UID: \"0417d34f-0032-43e3-8715-9f8af8446fc1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:11.021381 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:11.021339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0417d34f-0032-43e3-8715-9f8af8446fc1-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4\" (UID: \"0417d34f-0032-43e3-8715-9f8af8446fc1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:11.021605 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:11.021471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvwm4\" (UniqueName: \"kubernetes.io/projected/0417d34f-0032-43e3-8715-9f8af8446fc1-kube-api-access-kvwm4\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4\" (UID: \"0417d34f-0032-43e3-8715-9f8af8446fc1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:11.024780 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:11.024746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0417d34f-0032-43e3-8715-9f8af8446fc1-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4\" (UID: \"0417d34f-0032-43e3-8715-9f8af8446fc1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:11.031180 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:11.031159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvwm4\" (UniqueName: \"kubernetes.io/projected/0417d34f-0032-43e3-8715-9f8af8446fc1-kube-api-access-kvwm4\") pod \"prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4\" (UID: \"0417d34f-0032-43e3-8715-9f8af8446fc1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:11.199184 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:11.199094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:11.482573 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:11.482485    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"]
Sep 16 20:23:11.488100 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:23:11.488057    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0417d34f_0032_43e3_8715_9f8af8446fc1.slice/crio-052dc7c4d215c19b1def7c36892047cd756046c31cd29d8b8195969a8b0dc4ca WatchSource:0}: Error finding container 052dc7c4d215c19b1def7c36892047cd756046c31cd29d8b8195969a8b0dc4ca: Status 404 returned error can't find the container with id 052dc7c4d215c19b1def7c36892047cd756046c31cd29d8b8195969a8b0dc4ca
Sep 16 20:23:12.013793 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:12.013749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" event={"ID":"0417d34f-0032-43e3-8715-9f8af8446fc1","Type":"ContainerStarted","Data":"fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5"}
Sep 16 20:23:12.014263 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:12.013805    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" event={"ID":"0417d34f-0032-43e3-8715-9f8af8446fc1","Type":"ContainerStarted","Data":"052dc7c4d215c19b1def7c36892047cd756046c31cd29d8b8195969a8b0dc4ca"}
Sep 16 20:23:12.014263 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:12.013851    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:12.034984 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:12.034935    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" podStartSLOduration=2.034920813 podStartE2EDuration="2.034920813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:23:12.033927552 +0000 UTC m=+4669.159731946" watchObservedRunningTime="2026-09-16 20:23:12.034920813 +0000 UTC m=+4669.160725213"
Sep 16 20:23:16.793885 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:16.793800    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:23:26.794301 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:26.794247    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:23:36.803480 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:36.803441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:23:36.811124 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:36.811095    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:23:41.670748 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:41.670715    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"]
Sep 16 20:23:43.027065 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:43.027034    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:23:56.695390 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:56.695322    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" containerID="cri-o://a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d" gracePeriod=45
Sep 16 20:23:56.804561 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:56.804518    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 16 20:23:57.437183 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.437159    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:23:57.538392 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.538330    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5853633-4f13-4bdf-8395-e9f1554692ac-tls-certs\") pod \"d5853633-4f13-4bdf-8395-e9f1554692ac\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") "
Sep 16 20:23:57.538392 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.538367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqmpn\" (UniqueName: \"kubernetes.io/projected/d5853633-4f13-4bdf-8395-e9f1554692ac-kube-api-access-gqmpn\") pod \"d5853633-4f13-4bdf-8395-e9f1554692ac\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") "
Sep 16 20:23:57.538392 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.538389    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-home\") pod \"d5853633-4f13-4bdf-8395-e9f1554692ac\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") "
Sep 16 20:23:57.538621 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.538539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-dshm\") pod \"d5853633-4f13-4bdf-8395-e9f1554692ac\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") "
Sep 16 20:23:57.538621 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.538610    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-kserve-provision-location\") pod \"d5853633-4f13-4bdf-8395-e9f1554692ac\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") "
Sep 16 20:23:57.538735 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.538660    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-model-cache\") pod \"d5853633-4f13-4bdf-8395-e9f1554692ac\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") "
Sep 16 20:23:57.538735 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.538687    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5853633-4f13-4bdf-8395-e9f1554692ac-cabundle-cert\") pod \"d5853633-4f13-4bdf-8395-e9f1554692ac\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") "
Sep 16 20:23:57.538735 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.538720    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-tmp-dir\") pod \"d5853633-4f13-4bdf-8395-e9f1554692ac\" (UID: \"d5853633-4f13-4bdf-8395-e9f1554692ac\") "
Sep 16 20:23:57.538929 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.538884    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5853633-4f13-4bdf-8395-e9f1554692ac" (UID: "d5853633-4f13-4bdf-8395-e9f1554692ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:57.539049 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.539033    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:23:57.539116 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.539056    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5853633-4f13-4bdf-8395-e9f1554692ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5853633-4f13-4bdf-8395-e9f1554692ac" (UID: "d5853633-4f13-4bdf-8395-e9f1554692ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:23:57.539436 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.539393    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-home" (OuterVolumeSpecName: "home") pod "d5853633-4f13-4bdf-8395-e9f1554692ac" (UID: "d5853633-4f13-4bdf-8395-e9f1554692ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:57.540938 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.540914    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5853633-4f13-4bdf-8395-e9f1554692ac-kube-api-access-gqmpn" (OuterVolumeSpecName: "kube-api-access-gqmpn") pod "d5853633-4f13-4bdf-8395-e9f1554692ac" (UID: "d5853633-4f13-4bdf-8395-e9f1554692ac"). InnerVolumeSpecName "kube-api-access-gqmpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:23:57.541030 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.540929    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-dshm" (OuterVolumeSpecName: "dshm") pod "d5853633-4f13-4bdf-8395-e9f1554692ac" (UID: "d5853633-4f13-4bdf-8395-e9f1554692ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:57.541367 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.541345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5853633-4f13-4bdf-8395-e9f1554692ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5853633-4f13-4bdf-8395-e9f1554692ac" (UID: "d5853633-4f13-4bdf-8395-e9f1554692ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:23:57.552444 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.552420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5853633-4f13-4bdf-8395-e9f1554692ac" (UID: "d5853633-4f13-4bdf-8395-e9f1554692ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:57.563788 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.563765    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5853633-4f13-4bdf-8395-e9f1554692ac" (UID: "d5853633-4f13-4bdf-8395-e9f1554692ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:57.640179 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.640153    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5853633-4f13-4bdf-8395-e9f1554692ac-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:23:57.640179 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.640174    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmpn\" (UniqueName: \"kubernetes.io/projected/d5853633-4f13-4bdf-8395-e9f1554692ac-kube-api-access-gqmpn\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:23:57.640313 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.640186    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:23:57.640313 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.640194    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:23:57.640313 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.640202    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-kserve-provision-location\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:23:57.640313 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.640212    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5853633-4f13-4bdf-8395-e9f1554692ac-cabundle-cert\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:23:57.640313 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:57.640219    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5853633-4f13-4bdf-8395-e9f1554692ac-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:23:58.250143 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.250106    2583 generic.go:356] "Generic (PLEG): container finished" podID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerID="a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d" exitCode=0
Sep 16 20:23:58.250680 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.250167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" event={"ID":"d5853633-4f13-4bdf-8395-e9f1554692ac","Type":"ContainerDied","Data":"a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d"}
Sep 16 20:23:58.250680 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.250227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss" event={"ID":"d5853633-4f13-4bdf-8395-e9f1554692ac","Type":"ContainerDied","Data":"90e7feea7ed777c266d98b5cfeab9ab69edea86fd5b416908db64fc9e7f1c011"}
Sep 16 20:23:58.250680 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.250249    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"
Sep 16 20:23:58.250680 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.250261    2583 scope.go:117] "RemoveContainer" containerID="a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d"
Sep 16 20:23:58.260730 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.260709    2583 scope.go:117] "RemoveContainer" containerID="8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec"
Sep 16 20:23:58.269427 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.269383    2583 scope.go:117] "RemoveContainer" containerID="a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d"
Sep 16 20:23:58.269726 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:23:58.269708    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d\": container with ID starting with a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d not found: ID does not exist" containerID="a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d"
Sep 16 20:23:58.269777 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.269734    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d"} err="failed to get container status \"a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d\": rpc error: code = NotFound desc = could not find container \"a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d\": container with ID starting with a7a755b136c0a35c6d4d110ea12e0c0234731405584f10ae3bf5e7deb15d7e9d not found: ID does not exist"
Sep 16 20:23:58.269777 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.269752    2583 scope.go:117] "RemoveContainer" containerID="8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec"
Sep 16 20:23:58.269991 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:23:58.269972    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec\": container with ID starting with 8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec not found: ID does not exist" containerID="8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec"
Sep 16 20:23:58.270038 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.269997    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec"} err="failed to get container status \"8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec\": rpc error: code = NotFound desc = could not find container \"8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec\": container with ID starting with 8095c141488c77100b872e45a1fa2eb97f945a6fcb7bd30e75c719cc771700ec not found: ID does not exist"
Sep 16 20:23:58.278923 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.278894    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"]
Sep 16 20:23:58.284928 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:58.284907    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5f9f8d4df-cjsss"]
Sep 16 20:23:59.411683 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:23:59.411651    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" path="/var/lib/kubelet/pods/d5853633-4f13-4bdf-8395-e9f1554692ac/volumes"
Sep 16 20:24:54.960599 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.960508    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"]
Sep 16 20:24:54.961174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.960995    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="storage-initializer"
Sep 16 20:24:54.961174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.961014    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="storage-initializer"
Sep 16 20:24:54.961174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.961062    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main"
Sep 16 20:24:54.961174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.961071    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main"
Sep 16 20:24:54.961174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.961151    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5853633-4f13-4bdf-8395-e9f1554692ac" containerName="main"
Sep 16 20:24:54.964196 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.964175    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:54.968230 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.968209    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-x7n4c\""
Sep 16 20:24:54.968377 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.968236    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:24:54.968377 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.968265    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:24:54.968377 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.968236    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:24:54.976055 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.976032    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"]
Sep 16 20:24:54.991117 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.991095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:54.991213 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.991129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:54.991213 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.991153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:54.991295 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.991270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:54.991338 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.991296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:54.991338 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:54.991327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghvp\" (UniqueName: \"kubernetes.io/projected/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-kube-api-access-bghvp\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.092638 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.092607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.092638 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.092641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.092856 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.092671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghvp\" (UniqueName: \"kubernetes.io/projected/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-kube-api-access-bghvp\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.092856 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.092700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.092856 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.092725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.092856 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.092746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.093092 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.093068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.093092 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.093083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.093204 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.093123    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.095063 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.095045    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.095161 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.095145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.102376 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.102350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghvp\" (UniqueName: \"kubernetes.io/projected/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-kube-api-access-bghvp\") pod \"rolling-upgrade-test-kserve-74b79d7f77-2qtl8\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.277091 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.277008    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:24:55.554878 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:55.554792    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"]
Sep 16 20:24:55.559015 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:24:55.558986    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ea8fe3_5bf0_41b5_a7de_6f46b760056e.slice/crio-daec6a9bc725172ce5c0ff64f8045ba6c056110ba4425121ebcd692cb7db230e WatchSource:0}: Error finding container daec6a9bc725172ce5c0ff64f8045ba6c056110ba4425121ebcd692cb7db230e: Status 404 returned error can't find the container with id daec6a9bc725172ce5c0ff64f8045ba6c056110ba4425121ebcd692cb7db230e
Sep 16 20:24:56.516985 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:56.516947    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8" event={"ID":"32ea8fe3-5bf0-41b5-a7de-6f46b760056e","Type":"ContainerStarted","Data":"3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a"}
Sep 16 20:24:56.517368 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:56.517034    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8" event={"ID":"32ea8fe3-5bf0-41b5-a7de-6f46b760056e","Type":"ContainerStarted","Data":"daec6a9bc725172ce5c0ff64f8045ba6c056110ba4425121ebcd692cb7db230e"}
Sep 16 20:24:56.537095 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:56.537042    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8" podStartSLOduration=2.537030182 podStartE2EDuration="2.537030182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:24:56.535022331 +0000 UTC m=+4773.660826730" watchObservedRunningTime="2026-09-16 20:24:56.537030182 +0000 UTC m=+4773.662834569"
Sep 16 20:24:57.278068 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:24:57.278038    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:25:05.277297 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:25:05.277208    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:25:05.289405 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:25:05.289381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:25:05.568153 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:25:05.568067    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:25:27.169506 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:25:27.169478    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:25:27.174778 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:25:27.174758    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:25:27.181759 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:25:27.181740    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:25:27.186974 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:25:27.186945    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsb78_05d5b31c-3de4-44d3-a850-547bf19e0c07/ovn-acl-logging/0.log"
Sep 16 20:25:47.484297 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:25:47.484248    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"]
Sep 16 20:25:53.027287 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:25:53.027249    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" podUID="0417d34f-0032-43e3-8715-9f8af8446fc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:26:00.173128 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.173096    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"]
Sep 16 20:26:00.182465 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:26:00.182430    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:26:00.182465 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:26:00.182465 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 20:26:00.182465 ip-10-0-139-148 kubenswrapper[2583]:  > containerID="3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a" cmd=["/bin/sleep","15"]
Sep 16 20:26:00.182710 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.182482    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8" message="" err=<
Sep 16 20:26:00.182710 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:26:00.182710 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 20:26:00.182710 ip-10-0-139-148 kubenswrapper[2583]:  >
Sep 16 20:26:00.182710 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:26:00.182494    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:26:00.182710 ip-10-0-139-148 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:26:00.182710 ip-10-0-139-148 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 16 20:26:00.182710 ip-10-0-139-148 kubenswrapper[2583]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8" podUID="32ea8fe3-5bf0-41b5-a7de-6f46b760056e" containerName="main" containerID="cri-o://3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a"
Sep 16 20:26:00.182710 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.182535    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8" podUID="32ea8fe3-5bf0-41b5-a7de-6f46b760056e" containerName="main" containerID="cri-o://3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a" gracePeriod=60
Sep 16 20:26:00.526655 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.526629    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:26:00.614456 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614428    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tmp-dir\") pod \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") "
Sep 16 20:26:00.614627 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bghvp\" (UniqueName: \"kubernetes.io/projected/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-kube-api-access-bghvp\") pod \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") "
Sep 16 20:26:00.614627 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614520    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-dshm\") pod \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") "
Sep 16 20:26:00.614627 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614536    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-home\") pod \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") "
Sep 16 20:26:00.614627 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614575    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-model-cache\") pod \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") "
Sep 16 20:26:00.614627 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614626    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tls-certs\") pod \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\" (UID: \"32ea8fe3-5bf0-41b5-a7de-6f46b760056e\") "
Sep 16 20:26:00.614904 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614655    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32ea8fe3-5bf0-41b5-a7de-6f46b760056e" (UID: "32ea8fe3-5bf0-41b5-a7de-6f46b760056e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:00.614904 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614811    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-home" (OuterVolumeSpecName: "home") pod "32ea8fe3-5bf0-41b5-a7de-6f46b760056e" (UID: "32ea8fe3-5bf0-41b5-a7de-6f46b760056e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:00.614904 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614833    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-model-cache" (OuterVolumeSpecName: "model-cache") pod "32ea8fe3-5bf0-41b5-a7de-6f46b760056e" (UID: "32ea8fe3-5bf0-41b5-a7de-6f46b760056e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:00.615025 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614940    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tmp-dir\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:26:00.615025 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614954    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-home\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:26:00.615025 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.614963    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-model-cache\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:26:00.616971 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.616943    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32ea8fe3-5bf0-41b5-a7de-6f46b760056e" (UID: "32ea8fe3-5bf0-41b5-a7de-6f46b760056e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:26:00.617052 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.616995    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-kube-api-access-bghvp" (OuterVolumeSpecName: "kube-api-access-bghvp") pod "32ea8fe3-5bf0-41b5-a7de-6f46b760056e" (UID: "32ea8fe3-5bf0-41b5-a7de-6f46b760056e"). InnerVolumeSpecName "kube-api-access-bghvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:26:00.617052 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.617029    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-dshm" (OuterVolumeSpecName: "dshm") pod "32ea8fe3-5bf0-41b5-a7de-6f46b760056e" (UID: "32ea8fe3-5bf0-41b5-a7de-6f46b760056e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:00.715717 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.715660    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:26:00.715717 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.715685    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghvp\" (UniqueName: \"kubernetes.io/projected/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-kube-api-access-bghvp\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:26:00.715717 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.715696    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ea8fe3-5bf0-41b5-a7de-6f46b760056e-dshm\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:26:00.814571 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.814546    2583 generic.go:356] "Generic (PLEG): container finished" podID="32ea8fe3-5bf0-41b5-a7de-6f46b760056e" containerID="3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a" exitCode=0
Sep 16 20:26:00.814684 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.814594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8" event={"ID":"32ea8fe3-5bf0-41b5-a7de-6f46b760056e","Type":"ContainerDied","Data":"3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a"}
Sep 16 20:26:00.814684 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.814631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8" event={"ID":"32ea8fe3-5bf0-41b5-a7de-6f46b760056e","Type":"ContainerDied","Data":"daec6a9bc725172ce5c0ff64f8045ba6c056110ba4425121ebcd692cb7db230e"}
Sep 16 20:26:00.814684 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.814595    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"
Sep 16 20:26:00.814789 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.814653    2583 scope.go:117] "RemoveContainer" containerID="3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a"
Sep 16 20:26:00.824623 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.824605    2583 scope.go:117] "RemoveContainer" containerID="3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a"
Sep 16 20:26:00.824887 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:26:00.824870    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a\": container with ID starting with 3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a not found: ID does not exist" containerID="3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a"
Sep 16 20:26:00.824925 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.824897    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a"} err="failed to get container status \"3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a\": rpc error: code = NotFound desc = could not find container \"3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a\": container with ID starting with 3fb822536a4c7e500c706b81792481de8a77587da3620c82b5cc05326e9a9c9a not found: ID does not exist"
Sep 16 20:26:00.836021 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.835996    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"]
Sep 16 20:26:00.839535 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:00.839515    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-2qtl8"]
Sep 16 20:26:01.411174 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:01.411139    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ea8fe3-5bf0-41b5-a7de-6f46b760056e" path="/var/lib/kubelet/pods/32ea8fe3-5bf0-41b5-a7de-6f46b760056e/volumes"
Sep 16 20:26:02.506240 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:02.506193    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" podUID="0417d34f-0032-43e3-8715-9f8af8446fc1" containerName="main" containerID="cri-o://fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5" gracePeriod=45
Sep 16 20:26:03.026980 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:03.026947    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" podUID="0417d34f-0032-43e3-8715-9f8af8446fc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:26:13.027779 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:13.027734    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" podUID="0417d34f-0032-43e3-8715-9f8af8446fc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:26:13.028226 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:13.027885    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:26:23.027893 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:23.027807    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" podUID="0417d34f-0032-43e3-8715-9f8af8446fc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:26:32.753136 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.753111    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:26:32.875185 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.875156    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvwm4\" (UniqueName: \"kubernetes.io/projected/0417d34f-0032-43e3-8715-9f8af8446fc1-kube-api-access-kvwm4\") pod \"0417d34f-0032-43e3-8715-9f8af8446fc1\" (UID: \"0417d34f-0032-43e3-8715-9f8af8446fc1\") "
Sep 16 20:26:32.875331 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.875225    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0417d34f-0032-43e3-8715-9f8af8446fc1-tls-certs\") pod \"0417d34f-0032-43e3-8715-9f8af8446fc1\" (UID: \"0417d34f-0032-43e3-8715-9f8af8446fc1\") "
Sep 16 20:26:32.877536 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.877511    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0417d34f-0032-43e3-8715-9f8af8446fc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0417d34f-0032-43e3-8715-9f8af8446fc1" (UID: "0417d34f-0032-43e3-8715-9f8af8446fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:26:32.877643 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.877557    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0417d34f-0032-43e3-8715-9f8af8446fc1-kube-api-access-kvwm4" (OuterVolumeSpecName: "kube-api-access-kvwm4") pod "0417d34f-0032-43e3-8715-9f8af8446fc1" (UID: "0417d34f-0032-43e3-8715-9f8af8446fc1"). InnerVolumeSpecName "kube-api-access-kvwm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:26:32.964717 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.964691    2583 generic.go:356] "Generic (PLEG): container finished" podID="0417d34f-0032-43e3-8715-9f8af8446fc1" containerID="fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5" exitCode=0
Sep 16 20:26:32.964818 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.964728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" event={"ID":"0417d34f-0032-43e3-8715-9f8af8446fc1","Type":"ContainerDied","Data":"fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5"}
Sep 16 20:26:32.964818 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.964736    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"
Sep 16 20:26:32.964818 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.964773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4" event={"ID":"0417d34f-0032-43e3-8715-9f8af8446fc1","Type":"ContainerDied","Data":"052dc7c4d215c19b1def7c36892047cd756046c31cd29d8b8195969a8b0dc4ca"}
Sep 16 20:26:32.964818 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.964794    2583 scope.go:117] "RemoveContainer" containerID="fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5"
Sep 16 20:26:32.973739 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.973719    2583 scope.go:117] "RemoveContainer" containerID="fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5"
Sep 16 20:26:32.973988 ip-10-0-139-148 kubenswrapper[2583]: E0916 20:26:32.973969    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5\": container with ID starting with fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5 not found: ID does not exist" containerID="fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5"
Sep 16 20:26:32.974030 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.973997    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5"} err="failed to get container status \"fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5\": rpc error: code = NotFound desc = could not find container \"fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5\": container with ID starting with fc2d0ead9dded8ff74770036af109cc5a1fbbad9b29e9707739bd43ca5bc63e5 not found: ID does not exist"
Sep 16 20:26:32.975968 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.975951    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0417d34f-0032-43e3-8715-9f8af8446fc1-tls-certs\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:26:32.976033 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.975971    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvwm4\" (UniqueName: \"kubernetes.io/projected/0417d34f-0032-43e3-8715-9f8af8446fc1-kube-api-access-kvwm4\") on node \"ip-10-0-139-148.ec2.internal\" DevicePath \"\""
Sep 16 20:26:32.985806 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.985783    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"]
Sep 16 20:26:32.989708 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:32.989680    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-7bd999f6f5-rpsh4"]
Sep 16 20:26:33.412800 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:26:33.412771    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0417d34f-0032-43e3-8715-9f8af8446fc1" path="/var/lib/kubelet/pods/0417d34f-0032-43e3-8715-9f8af8446fc1/volumes"
Sep 16 20:27:17.059478 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:17.059439    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/0.log"
Sep 16 20:27:17.089499 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:17.089474    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/1.log"
Sep 16 20:27:17.984509 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:17.984454    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/0.log"
Sep 16 20:27:18.015239 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:18.015189    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/1.log"
Sep 16 20:27:19.327168 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:19.327131    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-qw4kr_c7b07186-8e1b-47bd-9caf-1824cc7889a6/authorino/0.log"
Sep 16 20:27:19.341983 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:19.341954    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-2rdd5_8324512c-72c3-4dcf-b68e-e92739454ff8/manager/0.log"
Sep 16 20:27:27.261159 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:27.261118    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-698qk_3af2c548-b0f3-4a74-87f7-eb574626b50d/global-pull-secret-syncer/0.log"
Sep 16 20:27:27.493478 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:27.493443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xtjc2_6a025317-d7c0-4050-8f6b-d6cf69387c55/konnectivity-agent/0.log"
Sep 16 20:27:27.548038 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:27.547969    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-148.ec2.internal_7033f5edf1da39728ed37f3d87a19a2b/haproxy/0.log"
Sep 16 20:27:32.710076 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:32.710038    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-qw4kr_c7b07186-8e1b-47bd-9caf-1824cc7889a6/authorino/0.log"
Sep 16 20:27:32.765677 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:32.765635    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-2rdd5_8324512c-72c3-4dcf-b68e-e92739454ff8/manager/0.log"
Sep 16 20:27:34.344482 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.344371    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f7da67c-519b-4876-a21f-0357b71b7d2b/alertmanager/0.log"
Sep 16 20:27:34.370912 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.370888    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f7da67c-519b-4876-a21f-0357b71b7d2b/config-reloader/0.log"
Sep 16 20:27:34.395425 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.395372    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f7da67c-519b-4876-a21f-0357b71b7d2b/kube-rbac-proxy-web/0.log"
Sep 16 20:27:34.423459 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.423440    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f7da67c-519b-4876-a21f-0357b71b7d2b/kube-rbac-proxy/0.log"
Sep 16 20:27:34.462272 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.462235    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f7da67c-519b-4876-a21f-0357b71b7d2b/kube-rbac-proxy-metric/0.log"
Sep 16 20:27:34.527456 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.527436    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f7da67c-519b-4876-a21f-0357b71b7d2b/prom-label-proxy/0.log"
Sep 16 20:27:34.552984 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.552956    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f7da67c-519b-4876-a21f-0357b71b7d2b/init-config-reloader/0.log"
Sep 16 20:27:34.634930 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.634900    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-tffxw_2e31c156-f5ae-4b96-8e08-c9ec51203b64/cluster-monitoring-operator/0.log"
Sep 16 20:27:34.790220 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.790188    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9487fbf4f-2rv8m_0a1274f6-49a0-4959-b46d-8c6bf82980c7/metrics-server/0.log"
Sep 16 20:27:34.823900 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.823875    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-r4xg7_cca56bad-aaaf-4439-8c80-5bf60dd2dfa0/monitoring-plugin/0.log"
Sep 16 20:27:34.868037 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.868011    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rddf_a2e0b062-6abf-4157-9a36-dcc7c14c1cef/node-exporter/0.log"
Sep 16 20:27:34.910460 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.910399    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rddf_a2e0b062-6abf-4157-9a36-dcc7c14c1cef/kube-rbac-proxy/0.log"
Sep 16 20:27:34.935552 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:34.935530    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rddf_a2e0b062-6abf-4157-9a36-dcc7c14c1cef/init-textfile/0.log"
Sep 16 20:27:35.392423 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:35.392393    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e05c0465-2b7f-448b-895f-23a45971ef52/prometheus/0.log"
Sep 16 20:27:35.436055 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:35.435969    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e05c0465-2b7f-448b-895f-23a45971ef52/config-reloader/0.log"
Sep 16 20:27:35.465941 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:35.465918    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e05c0465-2b7f-448b-895f-23a45971ef52/thanos-sidecar/0.log"
Sep 16 20:27:35.494251 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:35.494223    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e05c0465-2b7f-448b-895f-23a45971ef52/kube-rbac-proxy-web/0.log"
Sep 16 20:27:35.520568 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:35.520545    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e05c0465-2b7f-448b-895f-23a45971ef52/kube-rbac-proxy/0.log"
Sep 16 20:27:35.549585 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:35.549567    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e05c0465-2b7f-448b-895f-23a45971ef52/kube-rbac-proxy-thanos/0.log"
Sep 16 20:27:35.575942 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:35.575917    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e05c0465-2b7f-448b-895f-23a45971ef52/init-config-reloader/0.log"
Sep 16 20:27:35.660941 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:35.660913    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-mnh5b_c033d37c-40f8-46fb-aa0f-cfd1a54f25f5/prometheus-operator-admission-webhook/0.log"
Sep 16 20:27:37.589669 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.589637    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"]
Sep 16 20:27:37.590032 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.589951    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ea8fe3-5bf0-41b5-a7de-6f46b760056e" containerName="main"
Sep 16 20:27:37.590032 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.589962    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ea8fe3-5bf0-41b5-a7de-6f46b760056e" containerName="main"
Sep 16 20:27:37.590032 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.589975    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0417d34f-0032-43e3-8715-9f8af8446fc1" containerName="main"
Sep 16 20:27:37.590032 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.589981    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0417d34f-0032-43e3-8715-9f8af8446fc1" containerName="main"
Sep 16 20:27:37.590169 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.590035    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ea8fe3-5bf0-41b5-a7de-6f46b760056e" containerName="main"
Sep 16 20:27:37.590169 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.590043    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0417d34f-0032-43e3-8715-9f8af8446fc1" containerName="main"
Sep 16 20:27:37.594318 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.594299    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.596947 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.596920    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vcftn\"/\"kube-root-ca.crt\""
Sep 16 20:27:37.597862 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.597844    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vcftn\"/\"openshift-service-ca.crt\""
Sep 16 20:27:37.597920 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.597853    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vcftn\"/\"default-dockercfg-hfqdr\""
Sep 16 20:27:37.602592 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.602568    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"]
Sep 16 20:27:37.668201 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.668158    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/2.log"
Sep 16 20:27:37.675873 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.675844    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-btngq_1a7447f7-35b1-45f4-af73-2ebf91399392/console-operator/3.log"
Sep 16 20:27:37.683785 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.683759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-sys\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.683863 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.683800    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-proc\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.683863 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.683820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-podres\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.683948 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.683877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcssq\" (UniqueName: \"kubernetes.io/projected/c83ef305-b521-486a-8977-d15f7a99c08b-kube-api-access-tcssq\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.683948 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.683930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-lib-modules\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.785192 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.785170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-sys\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.785302 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.785206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-proc\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.785302 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.785229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-podres\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.785302 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.785262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcssq\" (UniqueName: \"kubernetes.io/projected/c83ef305-b521-486a-8977-d15f7a99c08b-kube-api-access-tcssq\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.785302 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.785296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-sys\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.785548 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.785311    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-proc\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.785548 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.785321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-lib-modules\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.785548 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.785359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-podres\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.785548 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.785474    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c83ef305-b521-486a-8977-d15f7a99c08b-lib-modules\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.793694 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.793672    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcssq\" (UniqueName: \"kubernetes.io/projected/c83ef305-b521-486a-8977-d15f7a99c08b-kube-api-access-tcssq\") pod \"perf-node-gather-daemonset-7466t\" (UID: \"c83ef305-b521-486a-8977-d15f7a99c08b\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:37.904919 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:37.904882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:38.179377 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:38.179289    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"]
Sep 16 20:27:38.182103 ip-10-0-139-148 kubenswrapper[2583]: W0916 20:27:38.182067    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc83ef305_b521_486a_8977_d15f7a99c08b.slice/crio-e790e16d5819e5eb7c008b0c88e31ffdb1594c62dece968006b24d2910965511 WatchSource:0}: Error finding container e790e16d5819e5eb7c008b0c88e31ffdb1594c62dece968006b24d2910965511: Status 404 returned error can't find the container with id e790e16d5819e5eb7c008b0c88e31ffdb1594c62dece968006b24d2910965511
Sep 16 20:27:38.274792 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:38.274757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t" event={"ID":"c83ef305-b521-486a-8977-d15f7a99c08b","Type":"ContainerStarted","Data":"e790e16d5819e5eb7c008b0c88e31ffdb1594c62dece968006b24d2910965511"}
Sep 16 20:27:39.280358 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:39.280321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t" event={"ID":"c83ef305-b521-486a-8977-d15f7a99c08b","Type":"ContainerStarted","Data":"a9caff42813eb97f3b0680810e6cd82552c4f5be5ef38b7de588ebf3edc89f9f"}
Sep 16 20:27:39.280764 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:39.280453    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:39.300291 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:39.300246    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t" podStartSLOduration=2.300232558 podStartE2EDuration="2.300232558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:27:39.299256542 +0000 UTC m=+4936.425060931" watchObservedRunningTime="2026-09-16 20:27:39.300232558 +0000 UTC m=+4936.426037010"
Sep 16 20:27:39.965970 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:39.965945    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jnthf_cd6974b2-a024-4ebe-930b-e3df0c1ba35b/dns/0.log"
Sep 16 20:27:40.008155 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:40.008127    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jnthf_cd6974b2-a024-4ebe-930b-e3df0c1ba35b/kube-rbac-proxy/0.log"
Sep 16 20:27:40.046762 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:40.046738    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-76twg_340d58a9-8b5d-416e-8bd3-88d32ba5747f/dns-node-resolver/0.log"
Sep 16 20:27:40.676583 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:40.676541    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bfc5889f8-4mdgf_9d684e18-c88c-4221-9311-6ae86f09dbbf/registry/0.log"
Sep 16 20:27:40.702571 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:40.702547    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ht7r_944c14c7-3c60-43cc-9e04-a0903a555407/node-ca/0.log"
Sep 16 20:27:41.930642 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:41.930612    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/0.log"
Sep 16 20:27:41.959840 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:41.959818    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/1.log"
Sep 16 20:27:42.504089 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:42.504059    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tzklw_ba6bc7d8-815f-4234-8744-3f5f14d850bf/serve-healthcheck-canary/0.log"
Sep 16 20:27:43.158483 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:43.158451    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tbqz7_86222f33-a533-4e08-ba27-ff62049bcba7/kube-rbac-proxy/0.log"
Sep 16 20:27:43.180147 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:43.180124    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tbqz7_86222f33-a533-4e08-ba27-ff62049bcba7/exporter/0.log"
Sep 16 20:27:43.205822 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:43.205799    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tbqz7_86222f33-a533-4e08-ba27-ff62049bcba7/extractor/0.log"
Sep 16 20:27:45.296353 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:45.296269    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-7466t"
Sep 16 20:27:46.312576 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:46.312549    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9498697bd-rc2tm_1c7bcbfd-1ab7-4cd2-9dd1-a3be26ec5b03/manager/0.log"
Sep 16 20:27:47.703674 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:47.703645    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/0.log"
Sep 16 20:27:47.734467 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:47.734443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/1.log"
Sep 16 20:27:49.162028 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:49.161995    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/0.log"
Sep 16 20:27:49.190263 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:49.190237    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-gtqp4_8f4d30db-1bc7-409a-9748-62ea5efc867c/istio-proxy/1.log"
Sep 16 20:27:53.666908 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:53.666878    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-t6nf6_b96d0288-1dff-46a1-a3fe-68b68c696520/migrator/0.log"
Sep 16 20:27:53.689510 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:53.689487    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-t6nf6_b96d0288-1dff-46a1-a3fe-68b68c696520/graceful-termination/0.log"
Sep 16 20:27:54.044818 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:54.044725    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-rq8bn_0ef80565-c6f2-487c-9086-3f98f46e17a4/kube-storage-version-migrator-operator/1.log"
Sep 16 20:27:54.046548 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:54.046524    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-rq8bn_0ef80565-c6f2-487c-9086-3f98f46e17a4/kube-storage-version-migrator-operator/0.log"
Sep 16 20:27:55.292963 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.292934    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9d2x_b2fd7290-33ee-4c35-a7f7-62f0704a5de1/kube-multus-additional-cni-plugins/0.log"
Sep 16 20:27:55.316007 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.315979    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9d2x_b2fd7290-33ee-4c35-a7f7-62f0704a5de1/egress-router-binary-copy/0.log"
Sep 16 20:27:55.336761 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.336742    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9d2x_b2fd7290-33ee-4c35-a7f7-62f0704a5de1/cni-plugins/0.log"
Sep 16 20:27:55.360395 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.360377    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9d2x_b2fd7290-33ee-4c35-a7f7-62f0704a5de1/bond-cni-plugin/0.log"
Sep 16 20:27:55.407838 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.407817    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9d2x_b2fd7290-33ee-4c35-a7f7-62f0704a5de1/routeoverride-cni/0.log"
Sep 16 20:27:55.435040 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.435016    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9d2x_b2fd7290-33ee-4c35-a7f7-62f0704a5de1/whereabouts-cni-bincopy/0.log"
Sep 16 20:27:55.458077 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.458055    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9d2x_b2fd7290-33ee-4c35-a7f7-62f0704a5de1/whereabouts-cni/0.log"
Sep 16 20:27:55.569790 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.569725    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pcbx2_601823a6-99aa-4683-b1db-06c7e34c3d86/kube-multus/0.log"
Sep 16 20:27:55.694599 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.694573    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wvcsr_14abbfab-5c2d-4416-a29a-939762f17972/network-metrics-daemon/0.log"
Sep 16 20:27:55.712053 ip-10-0-139-148 kubenswrapper[2583]: I0916 20:27:55.712032    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wvcsr_14abbfab-5c2d-4416-a29a-939762f17972/kube-rbac-proxy/0.log"