Distributed LLM Inference Over Consumer Hardware: A GPU-Pooling Experiment

Consumer motherboards typically support one, occasionally two, full-bandwidth GPU slots. Combining more GPUs than that normally requires server-class hardware — multi-socket boards with sufficient PCIe lanes — which puts it out of reach for anyone working with hardware accumulated from previous builds rather than purchased as a matched set. This leaves a common situation: several GPUs, no way to combine them in a single chassis, and no financial case for a server motherboard to run inference workloads that idle most of the day.

This raised a natural question: if PCIe lanes were the limiting resource, could a network connection substitute for one? Could two separate machines, each contributing a GPU, function as a single inference target?

The scope of the idea expanded naturally from there. If two machines on a home network could pool GPU capacity, the same principle could in theory apply at a larger scale — a building's worth of otherwise idle consumer GPUs acting as a shared local inference cluster, without a datacenter or a grid-connection queue. That version of the idea is speculative and untested. What follows is the smaller, testable version: two consumer PCs, mismatched GPUs, connected over a standard gigabit switch, evaluated against a specific hypothesis set before any hardware was touched.

Mechanism: llama.cpp's RPC Backend

The feature that makes this possible is llama.cpp's RPC backend, built specifically for splitting a model's layers across networked machines with different GPUs.

The RPC worker runs on the machine contributing GPU capacity:

./build/bin/rpc-server -H 0.0.0.0 -p 50052

The host machine — the one driving inference — connects to that worker as though it were a local device:

llama-server -m model.gguf -ngl 99 --rpc 192.168.1.42:50052 --host 0.0.0.0 --port 8080

From the host process's perspective, the remote machine is simply another device in the same memory pool. By default, llama.cpp distributes weights and KV cache across all available devices — local and remote — in proportion to each device's reported free memory. Manual control is available via --tensor-split, which accepts explicit per-device ratios (e.g., --tensor-split 4,3,3).

A client application — a laptop issuing requests, for example — has no visibility into this split. It communicates with the host machine over a standard HTTP API and is unaware that inference is distributed across a network boundary.

Two implementation details are worth noting:

Build consistency is required across machines. The RPC protocol is not guaranteed stable across llama.cpp versions, and a mismatch between the host and worker build produces connection failures or tensor-shape errors that do not obviously point to a version problem.
A Linux-only setup is not required. llama.cpp builds natively on Windows with CUDA support, and prebuilt release binaries ship with RPC already enabled — no compiler toolchain is necessary. Docker was excluded from consideration for the benchmark phase specifically: on Windows, containerization routes through WSL2, which defaults to NAT networking and complicates both LAN reachability and VRAM accounting — exactly the variables under measurement.

Experimental Design

Given the goal of producing a defensible answer rather than an anecdote, hypotheses were defined before any measurement was taken:

#	Hypothesis	Predicted outcome
H1	A model that fits entirely on the stronger single GPU will run slower, not faster, when distributed	Distributed tok/s < best single-GPU tok/s
H2	Distributed inference is substantially less energy-efficient than single-machine inference, due to duplicated system overhead	J/token roughly 1.5–2.5× the single-machine figure
H3	Distributed inference only provides a net benefit when the model cannot fit on either GPU individually	Distributed tok/s >> single-GPU-with-CPU-offload tok/s
H4	Prefill throughput degrades less than decode throughput under distribution, because prefill is compute-bound and batched while decode is latency-bound per token	pp t/s penalty < tg t/s penalty

H1 and H2 were deliberately framed as the unremarkable, expected outcomes. If the experimental setup failed to reproduce them, that would indicate a measurement error rather than a genuine finding.

A three-tier model ladder was used rather than a single model size, since one model size cannot isolate the relevant variables:

Tier 1 (small): fits comfortably on the smaller GPU. Distribution cannot improve this case by construction; any measured slowdown is attributable entirely to coordination overhead.
Tier 2 (mid): fits on the larger GPU only. Tests whether distribution outperforms simply using the stronger card alone.
Tier 3 (large): fits on neither GPU individually, but fits across both combined. This is the only tier where distribution can plausibly provide a benefit, and the correct baseline for comparison is single-GPU-with-CPU-offload, not single-GPU-native.

The test architecture kept the control machine (a laptop, in this case) entirely out of the inference path — its role was limited to SSH orchestration and result collection:

                    ┌──────────────────────────┐
                    │      control laptop      │
                    │   ssh, collects results  │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────▼─────────────┐
                    │      gigabit switch      │
                    └──────┬────────────┬──────┘
                           │            │
          ┌────────────────▼──┐      ┌──▼─────────────────┐
          │   PC A — host     │      │   PC B — worker    │
          │ llama-bench/-cli  │◄────►│   rpc-server       │
          │ layers 1..n       │ TCP  │   layers n+1..end  │
          │                   │ 50052│                    │
          └─────────┬─────────┘      └─────────┬──────────┘
                    │                          │
              ┌─────▼─────┐              ┌─────▼─────┐
              │ smart plug│              │ smart plug│
              │ wall watts│              │ wall watts│
              └───────────┘              └───────────┘

Including the control machine in the inference path would have introduced a third network hop, making it impossible to attribute measured overhead specifically to the A↔B link. Wall power meters — not GPU-reported power draw alone — were included on both machines, since nvidia-smi accounts only for the GPU itself and excludes CPU load, PSU losses, and fan power, all of which are duplicated in a two-machine configuration and are directly relevant to the joules-per-token comparison.

Throughput was tracked as two distinct figures rather than one blended number: pp t/s (prompt processing / prefill — batched, compute-bound) and tg t/s (token generation / decode — sequential, latency-bound). Collapsing these into a single "tokens per second" figure would obscure the distinction the experiment was specifically designed to surface.

Result

The initial measurement produced the following comparison: single-machine inference reached approximately 2,400 tokens/sec; the distributed configuration across both machines produced approximately 550 tokens/sec — a roughly 4.4x reduction in throughput.

This result should be reported with an important caveat rather than presented as a settled finding. A decode throughput of 2,400 tokens/sec is implausible for consumer GPU hardware, where decode typically runs in the tens-to-low-hundreds of tokens/sec range. The figure is far more consistent with prefill (pp) throughput than decode (tg) throughput. This distinction materially changes the interpretation: a 4.4x penalty on prefill is the expected and comparatively uninteresting result, since prefill genuinely is bandwidth-bound — it transmits the full hidden-dimension activation for the entire prompt across the network in one operation, which can total megabytes, versus the roughly 8KB per token that decode transmits. A 4.4x penalty on decode, by contrast, would exceed the roughly 2x penalty predicted by simple pipeline-bubble reasoning (each GPU idle approximately half the time while awaiting the other) and would warrant further investigation into serialization overhead or split-point count.

This ambiguity was not resolved before the investigation moved to a related question. As a result, the confirmed finding here is limited to the qualitative direction predicted by H1 — single-machine inference outperformed the distributed configuration — without a fully validated quantitative breakdown by throughput phase.

Interpretation

The result does not indicate that distributed inference across consumer hardware is without value; it indicates that speed is the wrong axis on which to evaluate it. Splitting a model that already fits on the strongest available card introduces network overhead with no compensating benefit — this outcome is close to a mathematical certainty rather than an empirical surprise. The actual value proposition, consistent with H3, is exclusively in Tier 3: models too large for any single available GPU, where the relevant comparison is against CPU-offload rather than against an unconstrained single-GPU baseline.

This is consistent with prior work in the same space. Petals, which distributed a 176B-parameter model (BLOOM) across volunteer consumer GPUs over the public internet, established the same principle at a larger scale: distributed pooling does not compete with a single strong GPU on raw speed. Its value is converting an infeasible workload into a feasible, if slower, one.

The energy dimension (H2) reinforces this conclusion rather than complicating it. Running two machines to serve one workload duplicates CPU load, PSU losses, and idle draw. If this penalty is significant at two nodes, it does not improve with scale — it compounds. This is the figure most relevant to any larger-scale extension of the concept, and it should be modeled explicitly before assuming that scale resolves the inefficiency.

Practical Application

A related, more modest question followed from this work: whether adding a second GPU to a single machine — rather than networking two separate machines — is worthwhile. Two 3070s in one system provide a combined 16GB VRAM pool versus 8GB on a single card, extending feasible model size from roughly 9B parameters to the 20–34B range. The same caveat applies here as at the network level: running an already-fitting model across two GPUs in one box does not produce a speed improvement, since layer-split inference remains sequential — computation passes through layer N on GPU 0, hands off to layer N+1 on GPU 1, and that PCIe handoff adds latency without removing any. The practical comparison, for anyone optimizing for capacity rather than reusing existing hardware, is against purchasing a single larger-VRAM card (a used 3090, for instance), which achieves the same capacity gain without split overhead.

Conclusion

GPU pooling — whether across a network, across PCIe lanes in a single chassis, or in the original larger-scale hypothesis — functions as a capability mechanism, not a performance mechanism. It expands the set of models that can be run at all; it does not make already-runnable models faster. The distinction matters for anyone evaluating this approach: the correct comparison is never "is distributed faster than single-GPU," but rather "is distributed-but-slower better than not-runnable-at-all" for a specific model that exceeds any single available card's capacity.