My NVIDIA Jetson AGX Orin (64 GB RAM, 456 GB disk, JetPack 7.0, Ubuntu 24.04) is the inference node of my homelab cluster — and deliberately only that: an Ollama server behind Tailscale, no Telegram, no key storage.
The architecture idea: least privilege for devices too
The cluster consists of a master VPS (orchestration, gateways, config master) and the AGX as a pure inference client. The configuration (.env with all provider keys) exists only on the master — the AGX receives an automatic read-only copy via inotify trigger, scp transfer, and md5 verification. Manual changes on the AGX are overwritten at the next push.
Why this matters: if someone compromises the AGX, they get no write access to the central configuration. Same principle as the agent-mailbox design — follow the attack surface downward.
Tailscale connects everything
All nodes sit in a Tailscale mesh. The monitor VPS is additionally publicly reachable (watchdog + dashboard), the AGX stays behind the tailnet. SSH jumps are defined; no open port knocking in the home network.
Measured performance (no marketing numbers)
| Metric | Value |
|---|---|
| qwen3.8:27b on AGX — TTFT | 0.89 s |
| qwen3.8:27b — throughput | 9.8 tok/s |
| 200 tokens take | 21.5 s |
| glm-5.3 (cloud) for comparison | 244 tok/s (25x faster) |
The lesson: GPU clock and model quantization beat TOPS numbers. A 17 GB model with Q4 quantization on 64 GB RAM runs smoothly — but local is local, and 9.8 tok/s is entirely sufficient for cron jobs and background inference.
What I’ll measure next: tokens/s across several models (identical prompts), power draw in watts over days via jtop logging, and the Orin Nano vs. AGX comparison on identical models.