The fundamental physics of data movement are forcing a shift to new memory architectures for AI inference. Sudeep Bhoja, co-founder and CTO of d-Matrix, joins Vik to detail the company's journey from a non-consensus bet on memory-bound inference to a full-rack 3D-stacked DRAM solution. Sudeep explains the pivot from analog to digital in-memory compute, the system-level disaggregation strategies for their Corsair card, and how the next-generation Raptor chip integrates with Nvidia's ecosystem.

"Essentially, we got 16 times the capacity of SRAM, while we retained SRAM bandwidth."
— Sudeep Bhoja, d-Matrix

Presented by Crusoe: Fine-tune and serve open models without managing a single GPU. When we sat down with Crusoe’s product team, Austin asked: what will you do with my data? Their answer: They don’t touch or see it. Your training data is encrypted so even Crusoe’s own admins can’t see it, and the fine-tuned model is yours. Deploy on Crusoe in one click, or download and run it anywhere. Upload a dataset, pick an open model, and pay per token rather than per GPU hour. Run a few versions, compare them, and download the one that works best. Accounts are free, and get $5 in credit when you sign up. Try Serverless Fine-Tuning: https://semidoped.com/crusoe

Key Takeaways:

- The core problem in AI inference is a 1000x energy gap. Compute costs femtojoules per operation, but moving data from memory costs picojoules per bit.

- d-Matrix's first analog in-memory compute chip failed because it couldn't maintain 32-bit accumulation precision, forcing a pivot to a digital architecture.

- The Corsair card's digital in-memory compute delivers 150 TB/s of on-chip SRAM bandwidth — over 8x more than an Nvidia Blackwell GPU's 18 TB/s HBM.

- The next-gen Raptor chip stacks a logic die on a DRAM die, delivering 16x the capacity (32 GB) of the prior SRAM design while retaining ~100 TB/s bandwidth.

- This 3D stacking is over 10x more energy-efficient than HBM4, consuming just 0.37 pJ/bit versus an estimated 5 pJ/bit for an HBM4-plus-GPU system.

- Choosing the stacking order is a 'pick your poison' trade-off. d-Matrix put logic-on-top for better thermals, accepting a harder power delivery challenge through the DRAM.

- The go-to-market product is a full rack with 144 Raptor packages, aggregating 2.3 TB of high-bandwidth memory designed to drop into Nvidia's VR rack infrastructure.

Chapters:
0:00 Intro: The Cost of a Token
3:28 The 2019 Bet on Inference
7:43 Nighthawk: Analog In-Memory Compute
12:08 Why Analog Failed
19:02 Corsair: Digital In-Memory Compute
26:04 System Strategy: Speculative Decoding
37:05 Raptor: 3D Stacked DRAM
39:40 The 'Free Lunch' of 3D Stacking
45:00 Trade-off: Logic-on-Top vs DRAM-on-Top
54:14 Go-to-Market: The Nvidia VR Rack
1:03:17 The Future: Lightning & 4-High Stacks
1:12:45 The End State of Inference

Follow Chipstrat:
Newsletter: https://www.chipstrat.com
X: https://x.com/chipstrat

Follow Vik:
Newsletter: https://www.viksnewsletter.com/
X: https://x.com/vikramskr

Follow Semi Doped:
Get more of Austin and Vik daily, free: https://daily.semidoped.com/

3D Stacked DRAM for Inference: d-Matrix's Sudeep Bhoja

Semi Doped

3D Stacked DRAM for Inference: d-Matrix's Sudeep Bhoja