Clustering two AMD Strix Halo machines over RDMA for local LLM inference

Category: Engineering post
Status: discussion
Sources: setup guide, HN discussion
Summary: A practitioner setup guide documents clustering two AMD Strix Halo machines (Framework Desktop boards, 128 GB unified memory each) over RDMA using ConnectX-5 100G NICs to run larger open-weight models such as DeepSeek V4 Flash and GLM 5.2 via tensor parallelism. The guide notes the PCIe 4.0 x4 slot (about 64 Gbps) bottlenecks the 100G NIC and that Strix Halo memory bandwidth (around 300 GB/s) trails Apple Silicon (600+ GB/s), with Thunderbolt offered as a lower-latency alternative interconnect.
Comments: HN commenters reported Strix Halo board prices have risen sharply since launch, narrowing the cost gap to Apple hardware, and debated whether the local-inference build is cost-effective against cloud subscriptions given the bandwidth limits.
Why it matters: It is a concrete reference for building multi-node local inference on consumer AMD hardware, where memory bandwidth and interconnect choice, not raw compute, govern usable model size and speed.