Transcribing speech on an analogue chip

Digital accelerators spend most of their energy moving data. A neural network’s weights live in memory, and every forward pass ships them to and from the compute units, which is why inference on a large model is dominated not by arithmetic but by the traffic between memory and processor. A team at IBM Research, writing in Nature on 23 August 2023, describes a different machine: a 14-nanometre inference chip that does the arithmetic where the weights live, in the analogue domain, on arrays of phase-change memory devices.

The physics is old and the engineering is new. Phase-change memory stores a weight as the conductance of a tiny volume of chalcogenide glass, set by the last heat pulse it received. Feed a voltage pulse into the array and the currents emerging on the other side are, by Ohm’s law and Kirchhoff’s laws, the sum of conductance times input voltage. That is a matrix-vector multiply, the kernel of every neural network layer, computed in one analogue step with no clocks and no fetch. The chip pairs 34 large phase-change arrays with digital input, analogue peripheral circuits and analogue-to-digital output on a two-dimensional mesh, and it deliberately omits on-chip digital cores. The weights never leave the memory they are stored in.

The experiments are the part that matters. The team ran two real inference workloads on the chip: speech recognition on a conventional speech corpus, and speech transcription on an end-to-end recurrent neural network transducer. In both tasks the analogue chip matched the accuracy of the digital models it was derived from, while the estimated energy efficiency, compared with a digital inference process doing the same job, was roughly fourteen times better. That headline number deserves a precise reading. It is energy per inference, not speed, and it is an estimate for the deployed workload rather than a measured wall-clock saving on a commercial product.

What makes the paper a milestone rather than another laboratory curiosity is the size of the models. Earlier in-memory chips had demonstrated the principle on toy networks of a few thousand weights, where the gaps between analogue and digital hardly matter. This implementation runs network layers large enough to be useful for speech tasks, with the non-idealities of real memory devices, conductance drift, device-to-device variation, read noise, handled by calibration and by copying trained weights into the arrays. The gap between a proof of concept and a chip you could put inside a hearing aid is mostly a gap in model size, and this paper closed it.

The honest limitations are visible in the design. Analogue compute is noise-limited; effective precision is a few bits rather than thirty-two, and drift means weights must periodically be refreshed or re-mapped to stay accurate. The chip is an inference device, not a training device, and it speaks one class of workload well rather than every neural network. None of that is a defect. The whole point of analogue inference is that the workload it targets, running already-trained models on edge hardware, is precisely where energy per inference is the bottleneck and where a handful of bits of precision are enough.

A year ago, in-memory analogue chips were a story about physics. This paper makes them a story about engineering: a manufactured 14-nanometre part, with digital peripherals wrapped around an analogue core, running models that matter. The route to a speech chip that lasts a week on a coin cell runs through parts like this, and the next question is not whether the principle works but how quickly a fab can build the arrays with the uniformity this approach demands.