변호사의 8-GPU Nvidia V100 서버가 이제 DeepSeek V4.1 Flash의 가장 많이 사용되는 전문가(expert)들을 NVFP4에 근접한 정밀도로 실행하며, 이는 기존 2비트 대비 향상된 것이다.

A lawyer with a private AI server built on eight used Nvidia V100 GPUs has posted an update. He has moved about 80% of the workload from 2-bit to 4-bit NVFP4 precision, added 1TB of RAM, and installed an RTX 3090. The server runs DeepSeek V4.1 Flash with a legal harness, the workflows and checks he wraps around the model, for his law practice. The cost is performance: code output dropped from 160–215 to 150–180 tokens per second. However, the crashes that affected the previous build are gone. He had originally blamed PCIe for the crashes, but the likely culprit was a bad PSU.

The poster is a trial lawyer in Columbia, South Carolina. After the original story, a commenter using the handle ProprietaryLegalAI replied three times, linking to a code repository, a website, and a LinkedIn profile that identify him as Nick Mermiges, founder of PLI Labs and proprietarylegal.ai. His other firm, PLI, is a technology consultancy rather than a law firm, and it now offers private, on-premises LLM serving as a service.

He built the software side with Claude Code, which wrote the kernels and ran the benchmarks under his direction. The Opus 5.5 and Fable 5.2 models handled the optimizations, custom kernels, and re-quantization. He said he was surprised by how well Fable built custom Volta kernels and how it re-quantized the experts that his legal harness triggers. From about February to July, he built a legal harness around Claude that worked, while his attempts at a local-model version failed. When DeepSeek V4.1 Flash arrived, he was able to port the harness successfully, though he says it is not as good as frontier models.

He reaffirmed that "the documents never leave the office." He upgraded the system's RAM from 256GB to 1TB and added a newer GPU, even though the V100s are on "borrowed time" because R580 is the last Nvidia driver branch to support them. The project's stated intention was to offload more of his workload to the system, so this upgrade and the precision change fit that goal. His previously stated aim of handling more than one request at a time has also been met.

The original "Volta sPLIt" build has become the "Volta Max uPLIft." The name still plays on his company's initials, PLI, which stand for Proprietary Legal Intelligence. He now describes the build as having "near-NVFP4 precision." Earlier this week, however, he had described NVFP4 on our forum as "essentially full precision." NVFP4 is Nvidia's 4-bit floating-point format, and Nvidia publishes its own NVFP4 version of DeepSeek V4.1 Flash. According to the project's README, that version matches the precision of the model's official weights at 4.25 bits per weight.

DeepSeek V4.1 Flash is built from "experts," the parts of the model that do most of the work for each word. At official precision, the model did not fit on this hardware. It has 384 routed experts in each of 40 layers, or 15,360 in total. By his estimate, that comes to about 269 GiB, while eight 32GB V100s provide 256 GiB in total before anything else is loaded. The 2-bit build's weights totaled 248.8GB, and the new build's Hugging Face package is about 556GB.

His workaround keeps every expert on the cards at 2-bit and upgrades the most-used experts to NVFP4 until the cards are full. This came to about 38.7%, or 5,949 experts, at NVFP4. "Most-used" refers to a ranking based on four kinds of text, weighted equally, one of which is his own legal harness traffic.

The payoff is that the NVFP4 experts do most of the work. In tests with one request at a time, 80% to 86% of decode routes, meaning the expert calls made while the model generates text, landed on NVFP4 experts. The share was lowest for general text and highest for source code. He predicts 78% to 85% for the two-request setup he now runs. The remaining 15% to 22% of calls still go to 2-bit experts.

Measured against Nvidia's version of the model as the reference, the results were good. The same-next-token rate, meaning how often a build picks the same next token as the reference, rose to 95.1% from 92.2%. KL divergence, which measures how far a build's probabilities diverge from the reference's, fell to 0.079 nats from 0.196. On 513 multiple-choice questions from a standard knowledge test, the number of answers that changed fell from 79 to 8.

He notes that these measurements capture fidelity, not accuracy. Accuracy was about the same across all three cases. The fidelity tests also ran on the build's first release, not on the two-request setup he runs now.

The trade-off is lower performance. Code output is 3% to 30% slower, depending on prompt length, and legal writing is 9% to 23% slower. He had hoped to maintain about 80 tokens per second for legal and prose work. The benchmark came in at 86 to 106 tokens per second for legal writing and 78 to 86 for prose. On the upside, running two requests at once raises total output to 159.0 tokens per second, compared with 118.9 for a single request. The first token also arrives sooner at every prompt length. At 256K tokens, it takes 108.4 seconds instead of 118.7.

Each request needs a "slot," and each slot costs something. Every slot keeps its own state on the cards, which leaves room for fewer NVFP4 experts: 6,215, 5,949, and 5,663 for one, two, and three slots, respectively. Each added slot may push another one to two percentage points of calls back to 2-bit, which could reduce output quality.

**DeepSeek V4.1 Flash on eight Tesla V100s: 2-bit vs. near-NVFP4 build, by prompt length**

| Prompt length | Wait for first token (s), 2-bit / near-NVFP4 | Code (tokens/s), 2-bit / near-NVFP4 | Legal writing (tokens/s), 2-bit / near-NVFP4 |

|---|---|---|---|

| 1K tokens | 1.2 / 1.0 | 215.0 / 150.1 | 117.6 / 105.5 |

| 32K tokens | 17.2 / 16.0 | 200.5 / 165.1 | 116.5 / 93.4 |

| 128K tokens | 59.2 / 54.3 | 182.8 / 151.4 | 110.9 / 87.3 |

| 256K tokens | 118.7 / 108.4 | 160.0 / 154.6 | 103.1 / 93.5 |

*Builder's measurements, Nov. 5 (2-bit) and Nov. 12 (near-NVFP4); greedy decoding, thinking off, one request at a time.*

The move to more RAM came this week with eight 128GB modules running at 2,933 MT/s. The original eight 32GB modules had to run at DDR4-2666 because of ECC errors. After the RAM upgrade, the host reports two NUMA nodes, and two boots hung at the first transfer between the boards. He resolved this by reordering the cards in software. The GPUs remain capped at 1,522 MHz, as before.

To track down the crashes, he swapped PCIe slots, cards, and cables until he concluded that the cause was "probably one bad PSU." He also added fans, but since temperatures did not change, this only showed that airflow was not the problem. After he changed the power delivery, all nine long chats read clean. He also acknowledged that his cable management is not something to judge, and that any "advice [there] is welcome and will be ignored."

He already runs two DGX Sparks with the same model at 1M context for agent work, handling three or four sessions at once, and he is not stopping there. The next step is scaling. Eight more V100s and a second server with 512GB of 3,200 MT/s ECC RAM are waiting in the wings. The second box would run another high-precision DeepSeek model, serving as an extra agent or running tasks in parallel. He also plans to move DeepSeek's small draft model off the V100s and onto the RTX 3090. That has not happened yet.