Our friends at Google have been busy working on improving LiteRT performance on the Raspberry Pi platform. We are pleased to share this work with you now, providing a robust framework for your general-purpose AI applications.
With special thanks to Google software engineers Lu Wang and Terry Heo, and Igalia graphics software engineer José María Casanova.
Imagine building a fully autonomous robot that can see, hear, and react to its environment in real time, completely offline on a single compact device like a Raspberry Pi. Edge AI unlocks this exact autonomy. It enables developers to build highly secure, self-contained systems — like intelligent robots and local AI agents — with zero cloud dependencies, ultra-low latency, and total data privacy.
We’ve made running edge AI on Raspberry Pi a breeze with LiteRT, Google’s high-performance, production-proven, on-device inference runtime. LiteRT allows you to seamlessly deploy everything from classical ML models to state-of-the-art LLMs right out of the box. By delivering optimized execution and hyper-efficient memory usage on both the CPU and GPU, LiteRT maximizes your Raspberry Pi’s full computing potential.
This hardware–software synergy shines brightest when paired with Gemma, Google’s family of lightweight open models. To show you what’s possible, we’ll demonstrate how running Gemma and LiteRT on a Raspberry Pi 5 can power the Reachy Mini robot to perceive and react to its environment entirely locally in real time. Read on to get started with your own deployment.
Discover the agentic capability of Gemma
Gemma models are well suited for building autonomous agents, smart cameras, and social robots that can reason and execute complex, multi-step workflows directly on your Raspberry Pi. To accommodate different hardware constraints, the Gemma family of models provides several highly efficient options:
- Gemma 3 270M: A hyper-efficient and compact base model designed for task-specific post fine-tuning, enabling high-speed, low-latency features like sentiment analysis or entity extraction in resource-constrained environments.
- EmbeddingGemma 300M: A state-of-the-art text embedding model that produces high-quality embeddings on-device, great for Retrieval Augmented Generation (RAG), semantic search, and classification.
- Gemma 3 1B: A lightweight and multilingual text-only model that balances compact size with strong generative capabilities, making it ideal for a wide range of on-device reasoning, summarization, and content creation tasks.
- Gemma 4 E2B: Tailored specifically for mobile and tight edge environments, it features memory-mapped per-layer embeddings and is ideal for continuous monitoring, fast text/image/audio inference, and edge-based speech processing where saving RAM is absolutely critical.
- Gemma 4 E4B: The sweet spot for performance and size. This model delivers noticeably stronger reasoning capabilities and frontier-level edge performance while remaining compact. It is the perfect choice for complex multi-step planning without overwhelming the Raspberry Pi’s resources.
Gemma performance on Raspberry Pi’s CPU
Through LiteRT-LM, a specialized orchestration layer on top of LiteRT, developers can seamlessly deploy Gemma right out of the box. Under the hood, sophisticated CPU acceleration via LiteRT and XNNPACK ensures the Gemma family of models is highly optimized for resource efficiency and low-latency execution directly on the Raspberry Pi.
| Model | Framework (CPU) | Prefill (tokens/sec) | Decode (tokens/sec) | Peak memory (MB) |
|---|---|---|---|---|
| Gemma 4 E2B | LiteRT-LM (QAT) | 99 | 9 | 1432 |
| llama.cpp (Q4_0) | 24 | 4 | 4406 |
- Hardware: Raspberry Pi 5 (8 GB RAM variant)
- Benchmark setup: 1024 prefill tokens and 256 decode tokens; the CPU runs with four threads
- llama.cpp is benchmarked using
llama-bench
withgemma-4-E2B-it-Q4_0.gguf
- LiteRT-LM is benchmarked using
litert lm benchmark
withgemma-4-E2B-it.litertlm
On a Raspberry Pi 5, LiteRT-LM delivers robust performance for Gemma 4 E2B, achieving 99 tokens/sec for prefill and 9 tokens/sec for decode, all while maintaining a remarkably low peak memory footprint of just 1432 MB. This brings Gemma’s highly responsive, general-purpose intelligence to Raspberry Pi.
Thanks to Gemma 4 E2B’s highly efficient tokenizer, which packs more text into fewer tokens (averaging ~4.2 characters per token), LiteRT-LM achieves an impressive end-to-end generation speed of ~27.3 characters per second, roughly 300 words per minute (wpm), in the Reachy Mini voice demo. This throughput makes Gemma 4 E2B excellent for real-time speech and translation tasks, delivering text at twice the speed of normal human speech (~150 wpm).
Explore more ready-to-use open models to run on Raspberry Pi from the LiteRT Hugging Face Community.
Execute on Raspberry Pi’s GPU with LiteRT
On Raspberry Pi 5, the quad-core Arm Cortex-A76 CPU is a raw computing powerhouse, delivering ~153.6 GFLOPS (FP32) and up to ~2.0 TOPS (INT8). In comparison, the integrated Broadcom VideoCore VII GPU is clocked at 800 MHz and offers a peak of ~76.8 GFLOPS (FP32) and ~0.24 TOPS (INT8).
While the CPU possesses a massive capacity advantage, the GPU introduces heterogeneous parallel execution, a paradigm critical for real-time edge applications. Rather than saturating the CPU, developers can delegate tasks across both processors to optimize overall system and thermal efficiency. For example, by offloading continuous vision or audio models to the VideoCore VII GPU, it preserves high-priority CPU cycles for overall system monitoring, pipeline orchestration, or computationally demanding LLM inference.
As such, we have enabled GPU inference on Raspberry Pi 5 with LiteRT’s WebGPU (Vulkan) backend via ML Drift. This integration allows you to run a wide range of computer vision, audio, and embedding models directly from the LiteRT Hugging Face Community with exceptional, low-latency performance. This includes seamless support for popular MediaPipe models, Ultralytics YOLO models, Moonshine, and much more.
The table below demonstrates the CPU and GPU latency of running classic computer vision and segmentation models via LiteRT:
| Model | Task | Input size | CPU latency (ms) | GPU latency (ms) |
| MediaPipe Selfie Segmenter | Segmentation | 256 × 256 | 7.88 | 26.36 |
| YOLO26n | Object detection | 640 × 640 | 101.26 | 375.73 |
| EfficientNet-Lite0 | Image classification | 224 × 224 | 29.15 | 87.85 |
| Moonshine-tiny | Speech recognition | 5 sec | 148.67 | 409.03 |
- Hardware: Raspberry Pi 5 (8 GB RAM variant)
- LiteRT CPU runs with four threads
- LiteRT GPU running with WebGPU (Vulkan) on updated Mesa V3DV Vulkan drivers
Deep dive: Reachy Mini pipeline powered by LiteRT
The Reachy Mini pipeline is a powerful showcase of low-latency, real-time edge AI inference running entirely on Raspberry Pi 5. By leveraging LiteRT, the system splits intensive vision and language workloads into a concurrent, dual-processing architecture across the CPU and GPU.
Here is how the parallel architecture works under the hood to ensure seamless interactions:
- Object detection (YOLO on GPU): Camera frames are streamed to the Raspberry Pi, where a YOLO detector (
yolox-tiny
) runs continuously on the GPU, avoiding resource contention and freeing up the CPU. - Speech recognition (Moonshine on CPU): When the user speaks, the ASR component transcribes the audio into text directly on the CPU.
- Reasoning and action (Gemma 4 E2B on CPU): The Gemma 4 E2B model processes the resulting transcript alongside the latest visual metadata to generate low-latency streaming responses, such as speech replies and physical robotic gestures.
- Text-to-speech (TTS on CPU): The TTS component synthesizes the generated text into audio in streaming. The system streams the synthesized voice back to the Reachy Mini robot.
See the full source code from the Reachy demo in the LiteRT Samples Github repo.
Agentic coding with LiteRT on Raspberry Pi
LiteRT provides a comprehensive suite of tools that covers the entire development cycle: conversion, quantization, benchmarking, and inference. For a fast, frictionless setup, the most straightforward approach is to use the LiteRT CLI tool. Rather than requiring developers or coding agents to manually manage multiple independent libraries, the LiteRT CLI aggregates core edge workflows into a single, unified command set.
You can now supercharge your development cycle by adding the LiteRT CLI skill and other advanced LiteRT skills into your AI coding agent, such as Google Antigravity. This empowers agents to autonomously orchestrate and execute complex, multi-stage machine learning workflows on your behalf. For example, you can easily build your own voice translator completely offline on a Raspberry Pi, like the Gemma Translator shown below.
An ultra-lean binary footprint for IoT devices
For resource-constrained IoT devices, minimizing storage and memory overhead is critical. Without special optimization, generic AI runtimes often bundle heavy desktop or server dependencies. In contrast, LiteRT is engineered specifically for on-device deployment, maintaining an exceptionally lean and modular distribution.
The table below compares the download footprint required to run LLM inference on a Raspberry Pi (Arm64 Linux).
| Tool | Download size | Architecture and dependency |
| LiteRT CLI (with LLM inference) | ~25 MB | Modular and on-demand: A lightweight pure-Python CLI frontend (~100 KB) that dynamically installs only the specific runtime modules (e.g. inference or conversion) optimized for the target hardware. |
| Ollama | ~1.44 GB | Monolithic: A single, pre-compiled binary that statically bundles all execution runtimes and heavy server-class acceleration drivers, regardless of the actual capabilities of the target host. |
Running your first model
You can install the LiteRT CLI and run your first model on a Raspberry Pi 5 with just a few simple commands.
1. Install the LiteRT CLI
To get started, install the LiteRT CLI via pip
(ideally within a virtual environment):
pip install litert-cli
2. Run the model
Download and run any compatible model directly from the LiteRT Hugging Face Community. The code snippet below demonstrates how to execute Gemma 4 E2B (e.g. gemma-4-E2B-it-litert-lm) on Raspberry Pi 5.
Run the model by providing your Hugging Face authentication token:
export HUGGING_FACE_HUB_TOKEN=
litert lm run \
--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
gemma-4-E2B-it.litertlm \
--attachment=image.jpg \
--prompt="You are Reachy Mini. Identify the main object in front of you, state its location (Left/Right/Center), and suggest one arm action in 10 words or less."
What’s next?
We are excited to share that LiteRT integration and Gemma models are coming soon to Hailo AI accelerators! This update will allow you to seamlessly offload model inference to the Raspberry Pi AI HAT+ and AI HAT+ 2, delivering massive hardware acceleration benefits through the exact same LiteRT workflows you use today.
Explore our resources and start your journey with LiteRT:
- Official documentation: Access installation guides, API references, and quick-start tutorials on the LiteRT Developer Site.
- GitHub repos: Find the latest source code, implementation details, and updates on the LiteRT and LiteRT-LM GitHub repos.
- Samples and templates: Check out the LiteRT-Samples GitHub repo for reference code. Kick-start your own application using the Google AI Edge Gallery app.
- Ready-to-use models: Download optimized, open-weight models like the lightweight and powerful Gemma 4 E2B directly from the LiteRT Hugging Face Community.
We value your input. Please share your thoughts, feedback, or feature requests by opening an issue on our GitHub issue tracker. Share your cool Raspberry Pi + LiteRT + Gemma projects with @googlegemma. We can’t wait to see what you build!
Acknowledgements
Google: Changming Sun, Chintan Parikh, Cormac Brick, Daisuke Majima, Dillon Sharlet, Erin Walsh, Frank Barchard, Glenn Cameron, Ian Ballantyne, Jingjiang Li, Jun Jiang, Kimish Patel, Lu Wang, Matthias Grundmann, Rodney Witcher, Sachin Kotwani, Sasha Denisov, Scott Loftin, Shuangfeng Li, Somdatta Banerjee, Terry (Woncheol) Heo, Volodymyr Kysenko, Weiyi Wang, Yi-Chun Kuo, Yu-hui Chen, and the gTech team
Raspberry Pi: Ashley Whittaker, Naushir Patuck, and Sarah Cunningham
Igalia: José María Casanova
Hailo: Eldad Rubinstein
Ultralytics: Francesco Mattioli, Lakshantha Dissanayake, and Onuralp Sezer
Moonshine AI: Pete Warden
Appendix
| Model | Info density (chars/tok) | Framework (CPU) | Model size (MB) | Prefill (tokens/sec) | Decode (tokens/sec) | Generation speed (chars/sec) | Peak memory (MB) |
|---|---|---|---|---|---|---|---|
| Gemma 4 E2B | ~4.2 | LiteRT-LM | 2468 | 99 | 9 | ~27.3 | 1432 MB |
| llama.cpp | 2709 | 58 | 7 | ~12.8 | 4406 MB | ||
| Gemma 3 270M | ~2.8 | LiteRT-LM | 278 | 433.17 | 22.58 | ~34.4 | 680 MB |
| llama.cpp | 278 | 462 | 39 | ~49.2 | 685 MB |
When running Gemma 3 270M, LiteRT-LM delivers impressively fast performance of 433 prefill tokens/sec and 23 decode tokens/sec. This speed makes it exceptionally well suited for low-latency, real-time applications, such as the fluid interactive behaviors required by the Reachy Mini robot.
2 comments
Jump to the comment form
Hugo
I don’t know what version you compare, and I applaud the progress on litert-lm, but llama.cpp has much better inference and prefill time on my Rpi5. Also my llama cpp is 15 Mb with static library, so I guess your 1,2 Gb llama is using some libraries for GPU that doesn’t seem useful anyway as you as infering on CPU only? And it has much better support for harness with chat and completion openai compatible profiles.There are QAT also for llama.cpp. I’m not saying this is the best solution, but it seems to me your numbers are lagging behind what we can get for a few month now. And I didn’t even add the results with MTP numbers, and other predictions systems.
crumble
Ollama not llama.cpp.
Ollama is delivered with libs for Nvidia and AMD. The only small Asset for CPU only is for Windows on ARM.
This is, because they want to teach us, that AI needs a lot of resources. So they build a release for every line change, write install scripts that cannot continue failing downloads and use extremely large chunk size for model downloading. Of course the failing cannot continue as well.
Their cunning plan has only one week part. The local models do not contain a machine GUID. You can simply copy the whole folder to another machine or point to a network folder.
Facts Only
* Google developed LiteRT as an on-device inference runtime for Raspberry Pi.
* The Raspberry Pi 5 features an Arm Cortex-A76 CPU and a Broadcom VideoCore VII GPU.
* The Gemma model family includes Gemma 3 270M, EmbeddingGemma 300M, Gemma 3 1B, Gemma 4 E2B, and Gemma 4 E4B.
* LiteRT-LM enables Gemma 4 E2B to achieve 99 tokens/sec prefill and 9 tokens/sec decode on Raspberry Pi 5.
* Gemma 4 E2B has a peak memory footprint of 1432 MB.
* LiteRT supports GPU inference via a WebGPU (Vulkan) backend using ML Drift.
* The Reachy Mini robot pipeline uses YOLO (GPU), Moonshine (CPU), Gemma 4 E2B (CPU), and TTS (CPU).
* The LiteRT CLI tool is available via pip.
* The LiteRT CLI download size is approximately 25 MB, while Ollama is approximately 1.44 GB.
* Support for Hailo AI accelerators is scheduled for future release.
Executive Summary
Google and Raspberry Pi have integrated LiteRT, an on-device inference runtime, to enable edge AI capabilities on the Raspberry Pi 5. This framework supports a range of Gemma lightweight open models, including Gemma 3 (270M, 1B) and Gemma 4 (E2B, E4B), allowing for offline reasoning, speech processing, and vision tasks without cloud dependency. By utilizing a dual-processing architecture, developers can offload vision models to the Broadcom VideoCore VII GPU while reserving the Arm Cortex-A76 CPU for LLM inference and system orchestration.
The system is demonstrated via the Reachy Mini robot, which uses a pipeline of YOLO for object detection, Moonshine for speech recognition, and Gemma 4 E2B for reasoning. While LiteRT-LM shows high prefill and decode speeds for the E2B model compared to some benchmarks, external user feedback suggests that alternative frameworks like llama.cpp may offer superior performance in certain configurations. Future updates intend to extend this integration to Hailo AI accelerators for further hardware acceleration.
Full Take
The strongest version of this narrative presents a significant leap in "cognitive autonomy" for edge devices, moving from simple pattern recognition to complex reasoning (agentic AI) entirely offline. By leveraging heterogeneous computing—splitting workloads between CPU and GPU—it provides a blueprint for low-latency, privacy-preserving robotics.
However, this is a vendor-led presentation where the load-bearing evidence for "performance" is provided by the vendor's own benchmarks. The comparison between LiteRT and Ollama focuses on download size—a metric of convenience—rather than a head-to-head performance analysis of the underlying inference engines. The user comments highlight a critical tension: the gap between curated corporate benchmarks and real-world community performance (e.g., llama.cpp), suggesting the "performance" claim may be context-dependent.
Patterns detected: ARC-0061 Authority Game
The driving paradigm is the shift toward "Local AI," where the value proposition is the removal of cloud dependencies. While framed as a win for privacy and latency, the underlying move is to embed proprietary runtimes (LiteRT) into the hardware ecosystem, creating a streamlined "golden path" for developers that favors Google's model ecosystem.
This benefits developers seeking rapid deployment but risks creating a dependency on a specific toolchain. If the community's independent tools (like llama.cpp) remain more efficient, the "official" path may become a gilded cage of convenience over performance.
Bridge Questions:
1. How do the benchmarks change when using non-optimized, real-world prompts versus controlled benchmark sets?
2. To what extent does the LiteRT ecosystem limit the ability to swap in non-Google open-weight models?
3. Does the reduction in binary size actually translate to a meaningful performance gain for the end-user, or is it a marketing metric?
Counterstrike Scan: A bad actor would use "technical superiority" and "privacy" as a Trojan horse to push a specific software stack, using selective benchmarks to marginalize superior open-source alternatives. The actual content matches the "vendor advertorial" pattern, though it remains a legitimate technical announcement.
Sentinel — Human
The text reads like a technically informed publication detailing specific performance benchmarks for running advanced AI models on edge hardware, supported by internal references and external discussion.