Persistent State Machines: LLM Attention with INT4 In-Memory Cells
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Persistent State Machines (PSMs) are presented as a formal discrete framework for LLM attention, with computation broadcast to stationary in-memory cells that evaluate local deterministic state transitions.
- The paper supplies complete mathematical proofs for quantization error bounds, a multi-phase discrete Softmax construction under bounded-logits, DFA equivalence with spatial factorization, and membership in DSPACE(O(n)).
- A 1024-cell array (d=128) implemented on a Zynq-7000 xc7z020 FPGA hit estimated dynamic power below 1.0 mW and normalized dynamic energy of 3.81 × 10^-5 pJ/op via hardware-level self-activation gating.
- A 256-cell sub-array was integrated into a full SoC on an UltraScale+ xcvu9p with AMBA AXI4 interconnects and AMD Xilinx PCIe Gen3 x1 Bridge (XDMA v4.2), closing timing at 62.5 MHz with WNS of +1.854 ns under the AWS Cloud FPGA Developer environment.
- The integrated SoC consumed only 0.67% of the device's logic slices and 0.00% of DSP blocks, indicating substantial headroom for scaling.
- Functional simulation with over 1,000 random vectors confirmed bit-exact agreement with a fixed-point software reference; all energy figures are tool estimates with no physical board measurement.
- The work is filed as Japanese Patent Application No. 2026-177318 (Patent Pending).
Why it matters: The sub-1mW dynamic power and 0.67% slice occupancy on the xcvu9p establish in-memory stationary-cell architectures for LLM attention as a working reference design on commodity FPGA, with >99% of logic resources still available for further scaling by AI hardware researchers.
