Running Models on Devices: The Edge Inference Stack

The deployment of large-scale neural networks has traditionally relied on the vast compute clusters of data centres. In these environments, power consumption is measured in kilowatts and cooling is managed by industrial HVAC systems. The model sits on a server, receives a request over the internet, processes it through thousands of parallel processors, and returns a response. This arrangement allows for high throughput, but it introduces latency, necessitates an internet connection, and centralises private data.

Modern mobile devices are attempting to invert this relationship. By running models locally on the device, often referred to as edge inference, manufacturers can offer features that function offline and preserve user privacy. However, the physical constraints of a smartphone differ fundamentally from those of a server rack. A mobile phone must operate within a thermal envelope of roughly five watts before its chassis becomes uncomfortable to hold or its processor begins to throttle its clock speed to prevent damage.

The primary obstacle is not merely the number of calculations required, but the movement of data. In a neural network, every operation requires the processor to fetch weights from memory, perform a calculation, and store the result. On a phone, the energy cost of moving a single piece of data from the main system memory to the processor can be orders of magnitude higher than the energy required to actually perform the mathematical operation. This imbalance creates a ceiling for performance known as the memory bandwidth wall.

Engineering a model to run on a handheld device is therefore a task of extreme compression and data management. It involves a suite of techniques designed to shrink the footprint of a model so it fits within the limited random-access memory (RAM) of a phone, while ensuring the processor can access those weights fast enough to provide a seamless experience. This transition from cloud-based gigantism to local efficiency is reshaping the architecture of mobile silicon and the software stacks that sit atop them.

The shift from cloud to local silicon

For the past decade, the trend in machine learning has been toward larger models. As the number of parameters in a transformer-based model grows, its ability to generalise and perform complex tasks tends to improve. This growth has been supported by the scaling of cloud infrastructure, where memory and power are relatively elastic. When a model becomes too large for one chip, it is partitioned across several.

The shift toward local silicon is driven by three primary constraints: latency, cost, and sovereignty. Latency is the most immediate factor for interface design. Round-trip communication with a server can take hundreds of milliseconds, making real-time applications like voice translation or camera processing feel sluggish. By processing data on the device’s Neural Processing Unit (NPU), the delay is reduced to the speed of the local bus.

Cost is a structural concern for service providers. Running inference on a server for millions of users incurs a constant operational expense in electricity and hardware maintenance. Shifting that compute load to the user’s own device effectively offloads the cost of power and hardware to the consumer. Furthermore, local processing ensures that sensitive data, such as private messages or biometric information, never leaves the device. This provides a level of data sovereignty that is difficult to guarantee in a cloud-based model.

Principles of neural network weights and precision

To understand how models are compressed for mobile devices, one must understand what a model actually is. A neural network is essentially a massive collection of numbers called weights, arranged in layers. When a model processes an input, such as a word or a pixel, it performs a series of matrix multiplications. The input is multiplied by these weights to produce an output.

In standard scientific computing, these weights are usually stored in 32-bit floating-point format (FP32). This format uses a high degree of precision to represent very small or very large numbers. While FP32 is excellent for training a model, where small adjustments to weights are necessary to help the network learn, it is inefficient for running a model that has already been trained.

The energy required to move data across a chip often exceeds the energy used to compute the result by a factor of one hundred.

Each 32-bit weight occupies four bytes of memory. A model with seven billion parameters stored in FP32 would require 28 gigabytes of RAM. Most modern smartphones have between eight and twelve gigabytes of RAM, much of which is already occupied by the operating system and active applications. To run a large model on a phone, the precision of these weights must be reduced without destroying the model’s ability to function.

Quantisation and the mechanics of weight compression

Quantisation is the process of converting the weights of a neural network from a high-precision representation to a lower-precision one. The most common target for mobile devices is 8-bit integer (INT8) or 4-bit integer (INT4). By moving from a 32-bit float to a 4-bit integer, the size of the model is reduced by a factor of eight. A seven-billion-parameter model that previously required 28 gigabytes of memory can be squeezed into less than four gigabytes.

The mechanism of quantisation involves mapping a wide range of floating-point values into a smaller set of discrete buckets. In an 8-bit system, there are only 256 possible values. In a 4-bit system, there are only 16. During quantisation, the software finds the minimum and maximum values in a layer of the network and divides that range into equal intervals. Each original weight is then assigned to the nearest bucket.

This process inevitably introduces a small amount of error, known as quantisation noise. If a weight was originally 0.7654 and the nearest available buckets are 0.75 and 0.80, the weight must be rounded. In many cases, neural networks are remarkably resilient to this loss of precision. The collective behaviour of millions of weights often compensates for the slight inaccuracies of individual ones.

Engineers use two primary methods to mitigate this noise:

• Post-training quantisation involves taking a finished model and applying the compression. This is fast but can lead to a slight drop in accuracy.
• Quantisation-aware training involves simulating the effects of lower precision during the training phase itself. This allows the model to learn weights that are specifically robust to being rounded, preserving more of the model's original intelligence.

The memory bandwidth wall in mobile architecture

Even if a model is small enough to fit in a phone's RAM, it must still be loaded into the processor to perform calculations. This is where the memory bandwidth wall becomes the limiting factor. The speed of a processor is often much higher than the speed at which the system memory can deliver data to it.

In a large language model, the inference process is typically auto-regressive. This means the model generates one token at a time, and for every single token, it must read every single weight in the model. If a model has seven billion parameters and each parameter is stored as a 4-bit integer, the processor must read roughly 3.5 gigabytes of data from the RAM just to produce one word. To produce words at a readable speed of ten words per second, the memory must be able to move 35 gigabytes of data every second.

Mobile memory, typically LPDDR5 or LPDDR5X, has a theoretical peak bandwidth that seems sufficient, but this bandwidth is shared across the entire system. The camera, the display, and the operating system are all competing for the same data bus. When the memory controller is saturated, the processor sits idle, waiting for the next batch of weights to arrive. This idleness does not save power; the mere act of keeping the memory bus active generates significant heat.

This thermal constraint is the ultimate arbiter of performance. As the memory and processor work at high capacity, they generate heat that cannot be dissipated by a fan. If the temperature exceeds a certain threshold, the system-on-a-chip (SoC) will reduce its voltage and clock speed to cool down. For the user, this results in a sudden drop in the speed of the model. Designing the edge inference stack is therefore a delicate balance between the mathematical requirements of the model and the physical realities of the hardware’s thermal and electrical limits.

The architecture of silicon specialisation

Modern mobile processors are no longer homogenous blocks of silicon. The central processing unit (CPU) is a generalist, designed to execute a vast array of instructions with low latency. It relies on complex branch prediction and large caches to handle unpredictable software tasks. However, the mathematical operations required for large language models and computer vision are remarkably predictable. They consist almost entirely of massive matrix multiplications. For this, the CPU is inefficient; its overhead per instruction is too high when the goal is to perform billions of similar additions and multiplications per second.

The Neural Processing Unit (NPU) is the hardware response to this inefficiency. Unlike a CPU, which processes a few pieces of data at high clock speeds, an NPU employs a systolic array architecture. In this configuration, data flows through a grid of processing elements, each performing a small part of a larger calculation before passing the result to its neighbour. This reduces the need to frequently access the main register file or memory, as intermediate results stay within the array.

While a Graphics Processing Unit (GPU) also offers high throughput, it is designed for the high-precision floating-point arithmetic required for rendering. NPUs are tuned for lower precision, such as 8-bit or even 4-bit integers. By narrowing the bit-width of each number, silicon designers can fit more processing elements into the same area. The result is a chip that can perform more operations per watt than a CPU, provided the software can translate the model weights into these lower-precision formats.

The high cost of moving bits

In a typical inference cycle, the energy spent on the actual arithmetic is relatively small. The primary energy sink is the movement of data between the main memory (DRAM) and the processor. To perform a single operation, the system must fetch the model weights and the input data from memory, move them across the data bus, and store them in local registers.

The energy cost of moving a single bit of data from external memory can be orders of magnitude higher than the cost of a mathematical operation on that bit.

Physicists and hardware engineers track this through the lens of energy per bit. Calculations suggest that a 32-bit floating-point addition might require a few picojoules, while fetching those same 32 bits from DRAM across a few millimetres of circuit board could require hundreds of times more energy. This disparity explains why quantisation is not merely an optimisation but a requirement for mobile devices. By compressing a model from 16-bit floats to 4-bit integers, the system reduces the amount of data crossing the bus by a factor of four. This directly translates to lower power consumption and less heat, even if the total number of mathematical operations remains the same.

Memory hierarchy plays a vital role in mitigating this cost. Engineers attempt to keep as much of the model as possible in the Static Random-Access Memory (SRAM) located directly on the processor die. SRAM is much faster and more energy-efficient than DRAM, but it is also much more expensive and takes up significant physical space. Most mobile NPUs have only a few megabytes of local cache, meaning larger models must still rely on constant, energy-expensive trips to the main memory.

Sustaining the thermal envelope

A mobile phone is a closed system with no active cooling. Heat must be conducted away from the SoC through the internal frame and out through the glass or metal casing. When an inference task begins, the chip can briefly run at high speeds using its thermal mass to absorb the initial spike in temperature. This is known as a burst. However, if the inference task is sustained, such as during a real-time video filter or a long voice-to-text transcription, the heat builds up faster than it can be dissipated.

Managing this thermal envelope requires a sophisticated interplay between the hardware and the operating system. The system monitors temperature sensors across the die. When a specific limit is reached, the power management integrated circuit (PMIC) reduces the voltage supplied to the NPU. Because power consumption scales quadratically with voltage, a small reduction in voltage can lead to a significant drop in heat. However, this also necessitates a lower clock frequency, which the user perceives as a slowing of the model.

Efficient inference stacks use dynamic frequency scaling to prevent these sharp drops. Rather than running at maximum speed until the chip overheats, the software may cap the performance at a level that can be sustained indefinitely. This provides a consistent, albeit slower, user experience and prevents the device from becoming uncomfortably hot to the touch. The goal is to find the equilibrium point where the heat generated by the memory bus and the processing cores matches the rate at which the phone's surface can radiate that heat into the surrounding air.

The abstraction of deployment

The diversity of hardware presents a significant challenge for software developers. A model must be able to run on an array of different SoCs, each with its own NPU architecture, memory bandwidth, and thermal characteristics. Providing a custom implementation for every chip is impractical. Consequently, the industry has settled on several intermediate frameworks to bridge the gap between high-level model code and low-level hardware instructions.

• CoreML and Android Neural Networks API (NNAPI) act as abstraction layers that negotiate between the app and the specific hardware available on the device.
• TFLite and ONNX Runtime provide cross-platform engines that can execute models on CPUs, GPUs, or NPUs using a plug-in architecture known as delegates.

These frameworks perform a process called graph lowering. The original model, usually defined in a high-level language like Python, is a directed graph of mathematical operations. The deployment framework takes this graph and optimises it for the specific target hardware. It might fuse multiple operations into one, such as combining a convolution and an activation function, to reduce memory accesses. It also handles the casting of data types, ensuring that a model trained in 32-bit precision can be executed efficiently using the 8-bit integer units of the NPU.

However, these abstractions are not perfect. A model optimised for one specific silicon architecture may perform poorly on another, even if the raw specifications of the chips are similar. The compiler must decide how to partition the workload between the CPU and the NPU, a decision that depends on the current thermal state of the device and the presence of other background tasks.

The balance of privacy and latency

The primary motivation for moving inference from large server farms to the edge is twofold: privacy and latency. When a model runs locally, the user’s data never leaves the device. This eliminates the risks associated with transmitting sensitive information over the network and storing it on a remote server. For applications involving biometric data, health records, or private conversations, this is a significant security advantage.

Latency is the second driver. Even with a fast internet connection, a round trip to a data centre introduces a delay that can make interactive applications feel sluggish. Local inference removes the network overhead, providing an instantaneous response. However, this comes at the cost of raw power. A server-side model can be vastly larger and more capable than anything a mobile phone can host. Developers must therefore choose between a highly capable but remote model and a smaller, less accurate, but faster and more private local one.

The decision to run a model on-device involves a trade-off between the depth of the intelligence provided and the sovereignty of the user's data.

The latency advantage of the edge can also be negated by the limitations of the hardware. If a model is too large for the device’s RAM, the system may be forced to swap data in and out of storage, which is much slower than DRAM. In such cases, the local model might actually take longer to produce a result than a remote model would, despite the network delay. This makes the selection of model size and the efficiency of the inference stack critical to the perceived performance.

Current status and outlook

The field of edge inference is currently defined by a clear consensus on certain methods and significant disagreement on others. It is established that quantisation is essential and that the memory bottleneck is the primary constraint on mobile performance. There is also broad agreement that dedicated silicon, such as the NPU, is the most energy-efficient way to handle these workloads.

However, the industry remains divided on the best way to handle the software-to-hardware interface. Some manufacturers prefer highly proprietary, vertically integrated stacks that offer maximum performance on their own hardware but poor portability. Others advocate for open standards that allow for easier deployment across different devices but may not fully exploit the unique features of a specific chip.

The picture would be fundamentally altered by two potential developments. First, the emergence of non-volatile memory technologies that can perform calculations within the memory itself (compute-in-memory) would eliminate the energy cost of moving data across the bus. Second, a shift towards sparse neural networks, where the majority of weights are zero, could drastically reduce the amount of data that needs to be stored and processed. Until such technologies mature, edge inference will remain a game of managing bits and heat within the narrow confines of a handheld device.