Over the past day, the two 32GB RTX 5090s in my always-on home server have been running tensor parallelism at TP=2 with SGLang, serving an NVFP4 floating-point quantized Qwen-3.8-27B. Looking over the monitoring logs from 08-23: this 27B model processed about 224 million prefill tokens in a day, of which 182 million were prefix cache hits taking almost zero compute, while newly generated decode tokens came out to roughly 2.25 million. Based on the U.S. Energy Information Administration (EIA) residential electricity rate of $0.1844/kWh and a total system full-load draw of around 500W, the machine spent only about 3.8 hours actually emitting tokens throughout the day, which works out to a monthly electricity bill of roughly $11.
If you compare this generative workload against commercial API subscription tiers (like a GPT-5.6 Luna paired with a $200 Codex plan), the average daily cost is about $1, or around $30 a month. Looking strictly at the monthly ledger, even for heavy daily use processing hundreds of millions of tokens, self-hosting a 3.8-hour daily active load at an electricity cost of around $11/month only saves about $20 compared to a subscription. But the upfront hardware investment is right in front of you—just the two 5090s plus the rest of the rig cost $6,000–8,000. Relying on a ~$20 monthly electricity savings margin to amortize $6,000–8,000 in hardware would take 25–33 years to break even; you simply cannot justify it purely on cost savings. So the reason to self-host isn’t saving money. What I’m after is compliance and privacy with data completely residing on local media, uncensored customization of model behavior, dynamic hot-swapping of full fine-tunes and LoRA weights, rock-solid certainty that it will keep working even completely offline, and an entire inference software stack under my absolute control. Looking at the benchmark numbers, dual 5090s running 27B-class models deliver ample decode throughput: pushed to 8-way concurrency, each stream still delivers 141 tok/s, making it the sweet spot for consumer hardware today. Of course, the ultimate criteria for hardware selection remains your actual workload utilization rate; blindly chasing single-stream speed divorced from utilization is an easy way to make poor configuration choices.
To get a clear picture of the actual throughput capabilities of 2×5090, I ran concurrent stress tests using the same standard mathematical reasoning prompt. Under baseline conditions with thinking mode disabled and max_tokens set to 1024, I ramped up load from 1-way concurrency up to 8-way concurrency, recording total throughput and per-stream decode rates across the different tiers:
| Concurrency | Total Throughput (tok/s) | Per-Stream Decode (tok/s) | Scaling Factor |
|---|---|---|---|
| 1 | 268 | 286 | 1.00× |
| 2 | 461 | 253 | 1.72× |
| 4 | 692 | 196 | 2.58× |
| 8 | 962 | 141 | 3.59× |
Total throughput represents the aggregate across all concurrent requests on the machine (total tokens divided by wall-clock time, including prefill), while per-stream decode is the pure generation rate of a single request (excluding prefill). At concurrency=1, the two differ only by the amortized TTFT, which is why 268 is slightly lower than 286.
Running multi-task workloads on dual cards in parallel is smooth: as concurrency increases, total system throughput scales almost linearly. Going from 268 tok/s at 1-way concurrency to 962 tok/s at 8-way concurrency represents a 3.59x scaling factor, showing that within 8 streams, dual-card TP=2 data exchange across a consumer PCIe bus does not become a bottleneck. At the same time, even with concurrency cranked up to 8 streams, each individual stream continues to output at 141 tok/s. Compared to natural human reading speed (around 200–300 words per minute), 141 tok/s leaves massive headroom; the actual system bottleneck shifts away from the model and onto downstream consumption capacity or upstream dispatch cadence.
Switching over to speculative decoding and long-context scenarios, the dual-card setup continues to deliver impressive generation speeds. Paired with the DFlash2 algorithm, measured speculative accept steps fall between 3.3–5.6, achieving 215–256 tok/s on math tasks and 222–232 tok/s on code generation tasks. Compared to the standard DFlash2 benchmark numbers from the official SGLang cookbook on a single 5090 (206.1 tok/s at 8192 input, 1024 output, concurrency 1), our local dual-card TP=2 measured single-stream throughput of 286 tok/s holds a distinct advantage. On a long-context test suite consisting of five 16k-token documents, average throughput reached 156.1 tok/s with thinking mode enabled and 195.4 tok/s with thinking mode disabled, with structured JSON extraction tasks hitting a measured peak of 334 tok/s.
If we set a practical usability threshold for 27B at no less than 100 tok/s per-stream decode speed and no less than 500 tok/s total system throughput, 2×5090 handles it with ease: at 4-way concurrency it delivers 196 tok/s per stream and 692 tok/s total throughput, and at 8-way concurrency it still maintains 141 tok/s per stream, comfortably exceeding the baseline across the board. The RTX 5090 hits the sweet spot for hosting 27B-class models because it ticks four hardware boxes simultaneously: Blackwell architecture’s natively integrated FP4 tensor cores providing efficient NVFP4 compute, 1792 GB/s memory bandwidth, 32GB of physical VRAM per card, and an MSRP of $2,000 per card. Higher-positioned workstation cards offer zero memory bandwidth advantage, while cheaper previous-gen cards run straight into bottlenecks in quantization support and VRAM capacity. Detailed deployment configurations and steady-state operational data are provided in the appendix at the end.
If you consider using the previous-generation flagship RTX 4090 for self-hosting, the most immediate issue is that it simply isn’t fast enough. LLM token generation speed is primarily bounded by memory bandwidth. The 4090 comes equipped with 24GB GDDR6X memory with a bandwidth of only 1008 GB/s—exactly 56% of the RTX 5090 (1792 GB/s). With memory bandwidth acting as a hard ceiling, the 4090 is roughly 40% slower than the 5090 in single-stream decode speed right off the bat.
An even bigger issue lies in the quantization formats. The latest NVFP4 format relies strictly on the dedicated FP4 tensor core hardware units built into the Blackwell architecture. The 4090 is based on the previous-generation Ada Lovelace architecture and lacks FP4 hardware instruction support; running 4-bit quantization forces you to fall back to traditional INT4 formats (such as AWQ, GPTQ, or AutoRound) or to use FP8, which eats significantly more VRAM. Data from the RaZeR paper and quantization benchmark studies show that NVFP4, with its floating-point representation and dynamic scaling mechanisms, achieves a dynamic range of 12—far exceeding INT4’s 7. This allows it to preserve the long-tail characteristics of weight distributions much more accurately, resulting in model output quality that consistently outperforms INT4. Furthermore, before performing tensor multiplications, INT4 requires dequantizing weights back into 16-bit floats, introducing extra computational overhead for nothing.
VRAM capacity is similarly constrained at every turn. A 27B model’s base weights in NVFP4 format take 20.14GB; if you add the 3.38GB DFlash2 speculative decoding draft model, total memory consumption hits 23.5GB. Squeezing a speculative draft model into the 4090’s 24GB VRAM leaves virtually zero headroom for the KV cache. If you switch to INT4 on a 4090, the model weights take roughly 14–16GB, leaving only 8–10GB of VRAM per card for KV cache, which quickly runs out as soon as context lengths grow or concurrency climbs. If you switch to FP8, the model weights expand to 27–28GB—a single 24GB card cannot even load the model, requiring at least two cards (48GB total) just to boot.
Small VRAM capacity gets magnified into another issue in multi-GPU scenarios: to reach the same total pool of VRAM, you have to slot more 4090s. Reaching 96GB of total VRAM takes just three 5090s, but requires four 4090s. More cards drive up platform costs and easily run into non-standard tensor parallelism splits: slotting three 4090s (72GB) hits TP=3 splitting constraints, leaving portions of memory idle; slotting four (96GB) is a setup meant for 70B+ models, making it pure overkill and waste for 27B. Conversely, while a single 4090 can technically fit 27B under INT4, the remaining 8–10GB for KV cache bottoms out instantly under long contexts or concurrency. Dual 4090s at best represent a bare-minimum floor for running 27B, and you still cannot escape the compromised memory bandwidth and degraded quantization quality.
Not to mention current market conditions. The RTX 4090 has been completely discontinued. While its launch MSRP in 2022 was $1,599, in the 2026-08 market, brand-new 4090s have surged to around $2,755, with used cards trading in the $2,200–2,350 range. Paying a higher premium than a 5090 to buy a discontinued, last-gen card with less VRAM, lower bandwidth, and no modern floating-point quantization support makes no sense—technically or financially.
The workstation-grade RTX PRO 6000 Blackwell comes equipped with 96GB of GDDR7 ECC memory, which might intuitively suggest stronger inference performance. But in a 27B self-hosting scenario, performance and cost are completely inverted. The PRO 6000’s memory bandwidth is 1792 GB/s—identical to the ~$2,000 MSRP RTX 5090. Because the core bottleneck of autoregressive decoding lies squarely on memory read bandwidth, with identical bandwidth the PRO 6000 cannot accelerate single-stream decode speed by even a fraction. The core value of 96GB VRAM is providing a massive KV cache pool: in FP8 mode for a 27B model, model weights take roughly 28GB of VRAM, leaving about 65GB entirely available for the KV cache pool. But such a massive cache pool is only useful in extreme workloads featuring massive concurrency paired with ultra-long contexts; for individual daily development or small-team inference, it is pure excess.
Pricing is even more exorbitant. Statistics from Tom’s Hardware show that the retail price of the PRO 6000 rose from $8,565 at its March 2025 launch to $13,250 in June 2026, climbing all the way to $16,000 in August 2026—a cumulative increase of 87%. Actual retail listings on Newegg sit similarly high at $12,099. Shelling out a procurement budget 8x the MSRP of a 5090 yields zero proportional speedups for 27B inference.
Multi-card interconnects are likewise limited to PCIe Gen5 P2P, with unidirectional communication bandwidth around 55 GB/s. On top of that, it suffers from a head-scratching firmware issue. A thread on the NVIDIA Developer Forums (365739) documents numerous firmware crash reports: in environments running vLLM inference services 24×7, after months of stable operation, cards suddenly trigger full-board resets with kernel logs throwing GSP firmware-related Xid 62, 119, 120, and 154 errors. This issue is widespread across multiple driver branches, including 570, 580, and 595. Once a failure occurs, recovery requires a physical power cycle via the chassis PSU; as of 2026-08-24, no official patch has been released. For self-hosted nodes demanding 24×7 high availability, this firmware vulnerability represents a direct operational roadblock.
If you don’t plan to purchase and maintain physical hardware yourself and instead rent compute in the cloud, there are three primary routes: public cloud GPU rentals, Google Cloud TPUs, and AWS Neuron dedicated compute instances. When you run the actual numbers, however, both billing and supply show significant mismatches.
First, look at the most common option, cloud GPU rentals: according to market data from 2026-05, third-party compute platforms (like Vast, Spheron, and RunPod) rent RTX 5090s for roughly $0.51–0.99/hour (Spheron lists at $0.76/hour, RunPod at $0.99/hour), but spot inventory is perpetually tight. On enterprise compute platforms, Lambda offers PCIe H100s at an on-demand rate of $3.29/hour and SXM H100s at $4.29/hour (market-wide rates range from $2.19–4.29/hour), with A6000 48GB cards at $1.09/hour. Converted to uninterrupted 24×7 monthly operation, a single 5090 instance runs roughly $370–715 per month, while a single H100 instance reaches $2,300–3,100 per month. The more awkward reality is the compute supply mismatch: checking the self-service consoles of mainstream AI cloud platforms like Lambda in 2026-08 reveals only enterprise compute options like B200 ($6.99), H100 SXM ($4.29), A100 ($1.99), and A6000 ($1.09)—with no self-service instances for consumer or workstation GPUs like the 5090 or PRO 6000 that combine high bandwidth with sensible pricing. On mainstream cloud platforms, you simply cannot rent a hardware combination with price-performance comparable to a local setup.
Google Cloud TPU is another avenue, but the hardware is likewise not sold at retail. TPU v1 through v7 chips have never been available to individuals; although Google opened direct hardware sales for the first time in 2026 by supplying 400,000 TPUv7 chips exclusively to Anthropic via Broadcom (a deal valued at around $10 billion, slated for delivery in 2027), ordinary developers and enterprises can only access them via cloud rental. The official on-demand pricing for TPU v5e is $1.20/chip-hr, with a 1-year committed use price of $0.84, a 3-year committed use price of $0.54, and measured spot instance prices around $0.5779. Renting a v5e-4 instance containing 4 chips (providing 64GB of memory) results in an on-demand monthly bill of $3,504, or about $2,453 per month with a 1-year commitment. On the software side, the official vLLM TPU backend provides production-grade support for standard 9B–32B dense models, but its support status for the GDN hybrid attention architecture used in Qwen-3.8 remains unconfirmed; SGLang’s TPU support is maintained only in a separate sglang-jax repository and restricted to TPU v6e/v7 hardware architectures.
AWS Neuron dedicated inference chips are likewise rented exclusively as EC2 instances. Taking us-east-1 on-demand pricing as an example, the 32GB memory inf2.xlarge instance runs $0.7582/hour (roughly $553/month), while the 192GB memory inf2.24xlarge instance costs $6.49/hour (roughly $4,740/month). Regarding framework support, vLLM has provided official support via NxD Inference starting from version 0.3.3, claiming compatibility with the V1 API; supported models include Llama 2/3.1/3.3, Llama 4, and Qwen 2.5/3, but adaptation for Qwen-3.8-27B’s GDN hybrid architecture remains officially unverified. SGLang offers no official Neuron backend, leaving only the community-maintained mini-sglang-neuron. In practice, Neuron requires roughly 15 minutes of ahead-of-time (AOT) compilation the first time it loads a 15GB model; the r/aws subreddit on Reddit has seen continuous complaints regarding deployment complexity and runtime crashes since May 2024; and AWS Free Tier accounts cannot launch inf2 instances by default.
The core advantage of the public cloud is providing an elastic buffer for volatile workloads. But faced with constant, continuous 24×7 high-load inference scenarios, the monthly bill for cloud GPU instances runs 6x to 50x higher than local physical electricity expenses.
Deciding whether self-hosting or cloud renting is more economical comes down to utilization. When you aggregate hardware procurement amortization, depreciation, and actual electricity costs and compare them against hourly public cloud rental rates, the cost crossover point between owned hardware and on-demand renting falls within a 26%–45% monthly duty cycle (equivalent to 192–328 hours of actual high-load operation per month). If your monthly utilization is above this range, owning physical hardware effectively drives down per-token inference costs; if your load falls below this range, relying on on-demand rentals or direct commercial API calls offers far superior capital efficiency.
In my own daily workflow, the inference service runs 24×7 always-on, processing about 224 million prefill tokens and generating roughly 2.25 million new decode tokens daily, with about 3.8 hours of actual high-load compute per day. Strictly on cost, this workload actually falls below the threshold where self-hosting makes financial sense; but I chose to purchase 2×5090s to gain total control over the inference stack, not to save money. For most independent developers and small-to-midsize engineering teams, however, daily usage follows a characteristically bursty pattern: concentrated calls during working hours, silence during nights and weekends, and prolonged idling between project cycles, resulting in real hardware utilization typically below 30%. Deploying an always-on self-hosted node in a low-utilization scenario means hardware depreciation and baseline idle power draw quickly become long-term idle waste.
So don’t rush to buy hardware before measuring your actual consumption. You can spin up an SGLang service in an existing development environment or test node and monitor it continuously for 7–14 days via the exposed /metrics endpoint, collecting key metrics such as steady-state TTFT latency, KV cache allocation ratios, prefix cache hit rates, and concurrency queue depth. Calculate your actual duty cycle and peak concurrency scale from real workload metrics, and map them against the hardware matrix to make your decision—preventing your hardware setup from drifting away from your actual operational needs.
Returning to the ledger from the opening: a local self-hosted monthly electricity bill for a 3.8-hour daily load is roughly $11—lower than a $30 commercial subscription tier. Looking strictly at operating expenses, self-hosting already comes out ahead. So why invest the effort to build your own service? A friend in a tech chat group put it aptly: “The fish at the supermarket is cheap and good, but why do so many people still go fishing? They aren’t doing it just to eat fish; they enjoy the process.” The logic behind self-hosting is the same: what I want is the compliance and privacy guarantee of data staying entirely on local physical media, freedom from third-party content policy restrictions, full control over full-parameter fine-tunes and dynamic LoRA weights, dependable operation even when offline and air-gapped, and a full-stack inference environment entirely under my control. In workflows involving sensitive private data, customized agent behaviors, and strict security isolation requirements, these qualities form the true core value of self-hosting.
For deployment selection with 27B-class models, you can follow a three-step decision framework:
For self-hosting 27B-class models, the technical roadmap is clear: dual 5090 interconnects, NVFP4 floating-point quantization, an optimized SGLang build, evaluating utilization against real workload metrics, and embracing the “good enough” performance envelope. Self-hosting buys you independent, total control, and dual 5090s provide a rock-solid, abundant compute foundation for models of this scale.
This article was researched and conceived by Qwen-3.8-27B and written by Gemini 3.7 Flash.
Qwen-3.8-27B adopts a GDN hybrid attention architecture (linear attention mechanism), which is friendly to memory footprints and attention computation overhead. The total NVFP4 4-bit floating-point quantized weights take 20.14GB, occupying only 11.02GB per card under TP=2 dual-card tensor parallelism. The inference engine is a local vendor build of SGLang featuring a DFlash2 quantized lm_head selector. Under this configuration, the KV cache pool holds 706,621 tokens, maximum context reaches 262,144 tokens, single-stream decode hits 282 tok/s (single-card baseline is 256 tok/s; inter-card PCIe interconnect does not drag down decode), and DFlash2 speculative decoding accept length remains steady at 5.2–5.4.
Steady-state metrics (production logs over an approximate 24-hour window): initial cold-start TTFT was around 78 seconds (JIT compilation), dropping to around 0.2 seconds thereafter; median decode throughput was 183 tok/s with a peak of 664 tok/s; prefix cache hit rate averaged 96% throughout the day, with a real-time gauge of 99.3%. Daily token accounting: roughly 224 million prefill input tokens, of which 182 million hit the prefix cache (actual compute volume was roughly 42.5 million); roughly 2.25 million decode output tokens. Workload profile: requests are dispatched by agents on demand rather than streaming tokens continuously 24 hours a day; decode active time was about 3.8 hours, with the machine mostly idling the rest of the time.
Sampling parameters: temperature=1.0, top_p=0.95, top_k=20, enable_thinking=false disabling built-in thinking. The upstream model is DeepSeek Flash V4 (hosted on Ollama Cloud, non-local); under blind evaluation scenarios, this 27B model’s overall reasoning and generation capabilities sit in the same tier.