LazyWeights: Decode-on-Demand
Edge LLMs

Arnab Sanyal1 Gourav Datta2 Sandeep P. Chinchali1
1The University of Texas at Austin2Case Western Reserve University
arXiv preprint
arXiv Code Models Device logs BibTeX

TL;DR — The integer codes of a quantized LLM are far from uniform, yet every deployed format stores each one in the same number of bits. LazyWeights keeps llama.cpp's formats, tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so the GPU kernels run unchanged and DRAM moves only coded bytes. At equal perplexity budgets the models are 5.7–31.4% smaller than the best llama.cpp format.

LazyWeights overview: entropy-aware GPTQ and a per-matrix Huffman bundle in the cloud; on the device, a Huffman decode unit between the memory controller and the L2 cache; bits per weight fall from 16 to 3.4

Fig. 1. Entropy-aware quantization and a per-matrix Huffman bundle are built in the cloud. On the device, a decode unit between the memory controller and L2 restores the fixed-width codes, so the GPU sees ordinary packed integers. Without the unit, the GPU kernels decode in software inside GEMV, as the reference.

Abstract

Memory decides which LLMs an edge device runs, and how fast: capacity on a 4 GB Jetson Nano, bandwidth on an 8 GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, llama.cpp's K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform.

LazyWeights keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7–31.4% smaller than the best llama.cpp format.

A bit-exact 0.21 mm2 ASIC decoder unit (1 GHz, 7 nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, LazyWeights needs 58–64% less time and 53–57% less energy per token than CPU llama.cpp, and runs a streamed Mistral-7B 6.6× faster at 3.89 bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).

Equal bits for unequal codes

On the edge, memory is the bottleneck twice over. The 4 GB Jetson Nano cannot even hold Mistral-7B, and the 8 GB Orin Nano Super is bound by DRAM bandwidth. Every deployed weight format, llama.cpp's K- and I-quants included, gives each integer code the same width, although the codes of a matrix are far from uniform. LazyWeights spends the bits where the information is: it keeps the format, steers the codes toward a concentrated histogram, and codes them losslessly.

Code selection: llama.cpp gives each sub-block its own step and rounds each weight alone, so codes spread; LazyWeights shares one widened step per super-block and lets GPTQ feed each column's error forward, so the codes concentrate

Fig. 2. Code selection (histograms illustrative). LazyWeights keeps the format but shares one step per super-block (γ) and widens it (c), so the codes concentrate. GPTQ feeds each column's error to the remaining columns, so the coarser grid keeps its quality.

1 · Select the codes

scripts/kq_quant.py

GPTQ runs into llama.cpp's own block formats (Q4_K, Q5_K, Q6_K, IQ4_XS), so the output is a valid GGUF that unmodified llama.cpp loads. One shared, widened step per super-block concentrates the codes; an optional entropy term joins GPTQ's code choice.

2 · Code them

lazyweights/bundle.py

Each matrix gets one length-limited Huffman table (11-bit codes). A 64×256 tile is 32 lanes of two rows, with the lanes' 32-bit words interleaved, so one aligned read fetches the next word of every lane. 4-bit codes are coded in pairs. Scales and minimums stay as llama.cpp stores them.

3 · Decode on demand

hw/huff/rtl

A decode unit between the memory controller and L2 restores the packed codes, so GPU kernels run unchanged and DRAM moves only coded bytes. Each lane compares an 11-bit peek against per-length limits, so there is no 2048-entry lookup table.

The bundle and the decode unit

(a) Bundle layout: a 64 by 256 tile of 4-bit codes split into 32 lanes, each one Huffman stream of 32-bit words interleaved in memory; (b) the decode unit: T engines of 32 lanes sharing an 8-word read port through a masked round-robin arbiter, each lane with a FIFO, a 64-bit buffer, an 11-bit peek, length-limit comparators and a symbol table

Fig. 3. (a) Bundle layout: each lane is one Huffman stream of 32-bit words; the lanes' words are interleaved, so one aligned read fetches the next word of every lane, and per-tile lane offsets let any tile start on its own. (b) The decode unit: T engines of 32 lanes share an 8-word read port through a masked round-robin arbiter. Each lane adds the code length's offset to index the symbol table and shifts its buffer by the length.

The runtime that runs the coded models on the Jetsons (kernels/llmrt.cu) fuses decode with GEMV, captures each pass in CUDA graphs, and supports prompt-lookup speculation and streaming from USB 3. Where the decode unit is not in silicon, the runtime stands in for it with software decode, or with a decode-free fixed-width “twin” whose measured time is used as the unit's.

Results

5.7–31.4%
smaller than the best llama.cpp format at 1% and 3.5% perplexity budgets
−36% / −38%
Orin latency / energy per token with the decode unit (−54% / −44% with speculation)
6.6×
faster streamed Mistral-7B on the 4 GB Jetson Nano, at 3.89 bits per weight
0.21 mm2
four-engine decoder at 1 GHz in 7 nm, bit-exact on an AMD ZCU102 FPGA
ModelBudgetFormatb/wΔPPLSavingGreedyPrompt-lookup speculation
ms / mJΔtΔEms / mJΔtΔE
Phi-3-mini≤1%IQ4_XS + imatrix4.311+0.65%35.0 / 69127.4 / 495
LazyWeights4.046+0.11%6.2%31.4 / 627−15/−10%−9%17.9 / 366−39/−35%−26%
≤3.5%Q3_K_M + imatrix4.108+2.91%44.0 / 84027.6 / 512
LazyWeights3.649+2.97%11.2%31.4 / 616−37/−29%−27%18.3 / 370−42/−34%−28%
Mistral-7B≤1%Q5_K_M5.668+0.44%83.3 / 178461.6 / 1276
LazyWeights3.886+0.94%31.4%53.0 / 1107−41/−36%−38%41.5 / 871−37/−33%−32%
≤3.5%IQ3_M + imatrix3.628+1.01%77.7 / 161171.6 / 1397
LazyWeights3.422+1.46%5.7%52.9 / 1068−44/−32%−34%38.5 / 786−56/−46%−44%
SmolLM-1.7B≤1%Q5_K_M5.721+0.65%26.4 / 48114.3 / 246
LazyWeights5.162+0.88%9.8%18.6 / 356−33/−29%−26%10.1 / 194−33/−29%−21%
≤3.5%IQ4_XS + imatrix4.388+3.09%23.7 / 37516.5 / 254
LazyWeights3.934+3.27%10.3%15.3 / 287−40/−36%−23%7.6 / 143−57/−54%−44%

Operating points on the Jetson Orin Nano Super (Table III). The upper row of each pair is the llama.cpp budget baseline in its own format. b/w counts all overheads; ΔPPL is WikiText-2 against fp16; saving is 1 − bours/bllama.cpp. Times and energies are per token, and each runtime uses its own drafter. LazyWeights runs with the decode unit at the measured twin time; Δt is bandwidth-bound / conservative, and ΔE assumes a clock-gated decoder.

Rate-quality frontiers of LazyWeights and llama.cpp on Phi-3-mini, Mistral-7B and SmolLM-1.7B: LazyWeights reaches each perplexity level at fewer total bits per weight

Fig. 4. Rate–quality frontiers. Dots are coded models, including llama.cpp's imatrix codes behind our Huffman stage. Lines show the lowest ΔPPL reachable at or below a rate, and shading marks where coding wins. Rings are the released operating points; dashed lines are the 1% and 3.5% budgets and fp16.

(a) Per-step speedup and energy factors from the llama.cpp baseline to LazyWeights on the Orin and the Nano; (b) software decode rates, and decode-unit area against rate for Huffman and rANS

Fig. 5. (a) Factors from the llama.cpp baseline (greedy) to LazyWeights with the unit and our drafter. Each step makes one change and is drawn on a log scale; the top half is the Orin and the bottom half the Nano. (b) Decode rate: software decoders on the left; on the right, ASAP7 area against rate for 1–16 engines, with Huffman compared to rANS. Bands show the rate each device's DRAM delivers.

Decoder hardware

BlockASAP7, 1 GHzNangate45, 500 MHz
µm2mWslackµm2slack
Comparator lane1,5663.59+117 ps23,462+0.5 ns
Table lane (2048 entries)14,07694.4−31 ns––
Arbiter, 32 lanes1,4430.83met†18,639+1.5 ns
Arbiter, 64 lanes9,174–−0.29 ns––
Engine (32 lanes + arbiter)51,550115.7–––

Logic synthesis, pre-layout (Table V). † after OpenROAD repair. The Orin's four-engine unit is 0.206 mm2; the Nano needs one engine.

Comparators, not tables. A 2048-entry lookup table per lane is nine times the area of a lane that compares an 11-bit peek against per-length limits, and it misses timing by 31 ns.

Huffman over rANS. rANS is never smaller (0.0 to +1.0% more bits), and at equal throughput (2 G weights/s) it needs 4.0× the area with pair symbols, or 1.15× with single codes.

On an FPGA, bit for bit. On the AMD ZCU102, four engines close timing at 112.5 MHz. All 29 matrices (484M weights) decode bit-exact, with cycle, stall and word counts equal to the C model on every chunk: 14.3 G weights/s at 0.26 nJ per weight.

What each piece is worth

ModelBudgetFullγ = 0RTN
Phi-3≤3.5%3.649, +2.974.046, +0.113.646, +12.07
Mistral≤1%3.886, +0.944.241, +0.323.886, −0.04
≤3.5%3.422, +1.463.837, +2.613.422, +2.78
SmolLM≤1%5.162, +0.885.621, +0.315.162, +0.57
≤3.5%3.934, +3.274.321, +2.243.934, +6.14

Ablations (Table VI): b/w, ΔPPL (%) at the operating-point settings. γ = 0 uses llama.cpp's own per-sub-block fit; RTN replaces GPTQ with round-to-nearest on the same grid.

Runtime changeMeasured onBefore → after
Speculative attentionSmolLM, 8-token pass44.4 → 31.2 ms
Phi-3, 8-token pass77.5 → 52.5 ms
Nano kernelsSmolLM ≤1%474 → 152 ms
SmolLM ≤3.5%282 → 134 ms
Phi-3585 → 261 ms
Ordered reductionNano, 2-token pass418 → 301 ms
CUDA graphsOrin, per greedy token−6 to −11%

Runtime optimizations (Table II), each measured before and after the change.

Models and code

ModelHugging Face repositoryPoints (≤1% / ≤3.5%)
Phi-3-mini-4k-instructLazyWeights-Phi-3-mini-4k-instructIQ4_XS_g0_lam0_c1.25 / IQ4_XS_g1_lam0_c1.25
Mistral-7B-Instruct-v0.2LazyWeights-Mistral-7B-Instruct-v0.2IQ4_XS_g1_lam0 / IQ4_XS_g1_lam0_c1.5
SmolLM-1.7B-InstructLazyWeights-SmolLM-1.7B-InstructQ5_K_M_g1_lam0 / IQ4_XS_g1_lam0

Each operating point comes in two forms: a GGUF that runs in unmodified llama.cpp, and its coded runtime bundle for the CUDA runtime. The code repository reproduces every table and figure, from GPTQ and the llama.cpp baselines through the Jetson runs to the RTL, synthesis and FPGA build; the raw Jetson logs are a release asset.

BibTeX

@misc{sanyal2026lazyweightsdecodeondemandedgellms,
  title         = {{LazyWeights: Decode-on-Demand Edge LLMs}},
  author        = {Sanyal, Arnab and Datta, Gourav and Chinchali, Sandeep P.},
  year          = {2026},
  eprint        = {2611.17108},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2611.17108}
}