MediaChipset

Technology

Inference is a memory problem.
We built a memory machine.

Generating a token means reading every weight once. At small batch sizes, how fast you can read memory, not how fast you can multiply, decides how fast you can think.

The balance

Compute per byte of bandwidth.

Training-class GPUs ship roughly 400 FLOP of compute per byte of memory bandwidth (≈410 FLOP/B for H200). That ratio suits gradient computation. For token generation it means the arithmetic units sit idle unless hundreds of requests are batched together, and every user waits for the batch.

IPROVF1 is balanced at 20.8 FLOP/B: 1000 TFLOPS over 48 TB/s. Decode becomes compute-bound at a batch of about 21, so the device runs at full efficiency on interactive, low-batch traffic.

Back of the envelope

At batch 1, decode speed is capped by how often the model's weights can be streamed from memory:

tokens/s ≈ memory bandwidth ÷ model bytes

13B FP8 on IPROVF1: 48 TB/s ÷ 13 GB ≈ 3,700 tok/s
13B FP8 on H200:    4.8 TB/s ÷ 13 GB ≈ 370 tok/s

An upper bound that ignores KV-cache reads and overheads; it shows why bandwidth sets the ceiling.

Architecture

One die, a 3D-stacked memory, nothing in the way.

100 LLM cores sit beneath a 64 GB 3D-stacked DRAM. Weights and KV cache stream from memory; on-chip SRAM is spent on double-buffering, which turns bandwidth into delivered throughput.

IPROVF1 block diagram 64 GB 3D-stacked DRAM connects to the IPROVF1 die at 48 TB/s. The die contains 100 LLM cores, an attention engine, 128 MB SRAM, a paged KV cache and dynamic W8A8 quantization. A 1.0 TB/s die-to-die link connects to a peer device, and PCIe Gen5 x16 at 128 GB/s connects to the host server. 64 GB 3D-stacked DRAM weights + KV cache stream from here 48 TB/s IPROVF1 die 100 LLM cores 250,000 dual-mode PEs Attention engine hardware FlashAttention score tile stays on die 128 MB SRAM double-buffered weight & KV streaming Paged KV cache 16-token blocks 32k+ token context Dynamic W8A8 per-channel weight × per-token activation scale Runtime-provisionable compute, 10–100 % in steps 1.0 TB/s D2D IPROVF1 peer → VF2 PCIe Gen5 ×16 · 128 GB/s Host server

Native W8A8 at exactly double rate

FP8 = 2× BF16 is a property of the multiplier: one dual-mode PE issues two FP8-E4M3 MACs or one BF16 MAC per cycle. The FP8 GEMM speed-up measures 1.92× when compute-bound.

Attention as one instruction

The engine replaces matrix-multiply / softmax / matrix-multiply with online softmax over an on-chip score tile, so the largest tensor in a long-sequence layer is never written out and re-read.

Linear scale-out

A 1.0 TB/s die-to-die link pairs two devices into IPROVF2 with twice every resource. The same architecture scales into IPROVF4 and IPROVF48 for frontier-scale models.

Software stack

From your framework to silicon.

Models enter through the frameworks your team already runs. The compiler and kernel library lower them onto the device; every family member runs the same compiler, runtime and model formats.

Validated against the reference GPU implementation on text LLMs, vision-language and vision-language-action models, in both BF16 and FP8 W8A8. Prefill and decode throughput track the reference at ≈1.00× and ≈0.97× (median).

Measured on the IPROVF-Sim validation stack.

  1. PyTorch · Hugging Face · vLLMmodels & serving
  2. MediaChipset Graph Compilergraph optimization & lowering
  3. MediaChipset Kernel Librarytuned FP8 & BF16 kernels
  4. MediaChipset Runtime & HALhardware abstraction
  5. MediaChipset Kernel DriverLinux
  6. IPROVF1PCIe Gen5 ×16