NorthPole puts memory beside compute

Most of the energy spent running a neural network is not spent doing arithmetic. It is spent moving numbers. In a conventional accelerator, weights and activations sit in banks of memory and must be shuttled to the arithmetic units, and on every chip generation the arithmetic gets faster while the data movement does not. The result is the well-known memory wall: a processor that can compute trivially but cannot feed itself. A paper in the 19 October 2023 issue of Science, ‘Neural inference at the frontier of energy, space, and time’ by Dharmendra Modha and colleagues at IBM Research, describes a chip designed to delete that wall rather than route around it.

The chip, NorthPole, is explicitly brain-inspired and silicon-optimized. The brain intertwines compute with memory; a neuron processes and stores in the same place, and its connectivity is dense and local. NorthPole translates that into a non-von Neumann architecture in which small processing tiles are woven directly into a large memory fabric, so that compute and memory are intertwined on the die and off-chip memory is eliminated. Externally, the chip presents itself as an active memory part rather than a conventional accelerator, a framing that matters for how it slots into a system.

The engineering result is striking because of when it was built. NorthPole was fabricated on a 12-nm process, a node older than the leading-edge parts it competes against. The die packs 22 billion transistors into a memory-near-compute spatial design with a two-dimensional mesh of cores and on-chip global wiring. On standard inference benchmarks the authors report that the chip outperforms contemporary specialized accelerators on ResNet-50 and YOLOv4, including chips built on more advanced process nodes, and they extend the same argument to BERT-base. The headline claim is not a transistor-node war but a systems claim: architecture can recover what process generation would otherwise dictate.

The idea is nearly two decades in the making. Modha’s group spent years on spiking-neuron simulators such as the TrueNorth board and its successor, and NorthPole is the distillation of that programme into a part made for ordinary machine-learning inference rather than for neuroscience simulation. It is worth being precise about what it is and is not. NorthPole does not run spiking networks; it executes conventional dense neural networks on a spatial architecture. The brain-inspired part is the principle, memory near compute and local connectivity, not the biology of the signals.

The paper’s limitations are the standard ones for a research chip. The part is a prototype, not a product, and the benchmarks are the inference-friendly ones where the memory wall bites hardest. On large language models, where weights dwarf what a single die can hold, the case is weaker, and IBM’s own later work on NorthPole for small LLMs is an attempt to extend the argument rather than a proof that it already holds. The architecture also presumes a fixed, scheduled dataflow; it is not a general-purpose processor.

That said, the conceptual move is the contribution. The industry had decided that faster inference meant a newer node and a bigger batch. NorthPole argues that the binding constraint is the distance a number travels, and that a chip which treats memory as the fabric of computation rather than as a neighbouring component can beat much newer silicon on the very workloads that matter for deployment. A decade of neural-network accelerators has mostly been a tribute to von Neumann; this one at least tries to be a rebellion, and the numbers on a 12-nm die suggest the rebellion is not purely decorative.