In this guide, our friends from Google walk you through how to install LiteRT and run high-performance on-device AI models using the LiteRT CLI tool. You can read more about the work Google has done to improve LiteRT performance on the Raspberry Pi platform here.
Step 1. Raspberry Pi OS Setup
This step is meant for those starting with a fresh Raspberry Pi device or those looking to install an updated version of Raspberry Pi OS.
- Download the Raspberry Pi Imager tool from raspberrypi.com/software.
- Launch the application.
- Select Raspberry Pi 5 in the ‘Choose device’ menu.
- Select Raspberry Pi OS (64-bit) under ‘Choose OS’.
- Insert your SD card and select it under the ‘Choose storage’ column.
- Enter a hostname (e.g. dino-pi), username (e.g. dino), and password (e.g. dinopi).
Note: Customize the hostname, username, and password to your preferences.
- Enable SSH.
Note: You may wish to skip enabling Raspberry Pi Connect.
- Click Save, then select Yes to begin the write process.
- Click Finish once writing and verification are complete.
Step 2. Hardware and connection
- Remove the SD card from your laptop and connect it via the SD card slot on your Raspberry Pi.
- Connect the Raspberry Pi and your laptop together via an Ethernet cable.
Note: You may need to use an adapter if your laptop does not have a matching port.
- Connect the Raspberry Pi to a power outlet using its power supply.
Step 3. Accessing the Raspberry Pi
- Open your terminal and ping your Raspberry Pi to confirm it is reachable on your network.
Note: Replace ‘dino-pi’ with the hostname chosen during installation (Step 1.6).
ping dino-pi.local
- SSH to the Raspberry Pi via SSH.
Note: When prompted, enter the password chosen during installation (Step 1.6).
ssh [email protected]
- Verify the hardware architecture is aarch64.
uname -m
Step 4. Set up Hugging Face account and access
- If you don’t already have one, create an account on Hugging Face.
- Go to your Hugging Face settings, click ‘Access Tokens’, then click ‘Create New Token’. This is referred to as
going forward. - Follow the instructions for generating a token with the token type of Read.
Step 5. LiteRT CLI installation
- Update the system and refresh the available software packages.
sudo apt update
sudo apt full-upgrade -y
- Download and install the
uv
tool manager to handle your AI environments.
curl -LsSf https://astral.sh/uv/install.sh | sh
- Refresh your shell environment to ensure it recognizes the newly installed tools.
source $HOME/.local/bin/env
- Use the
uv
manager to install the LiteRT CLI, which can run and benchmark various models such as language models (via LiteRT-LM) or vision models.
uv venv --clear --python=3.13 --seed
source .venv/bin/activate
uv pip install litert-cli-nightly
Step 6.a. Run large language models
- Navigate to the desired Gemma model (e.g. gemma-4-E2B-it-litert-lm).
Note: You may be required to accept the Gemma model terms for certain models.
- Export your token from Step 4 and run the
litert lm
command to start a conversation.
export HUGGING_FACE_HUB_TOKEN=
litert lm run \
--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
gemma-4-E2B-it.litertlm \
--prompt="What is the capital of France?"
Example output
dino@dino:~ $ litert lm run \
--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
gemma-4-E2B-it.litertlm \
--prompt="What is the capital of France?"
Downloading gemma-4-E2B-it.litertlm from litert-community/gemma-4-E2B-it-litert-lm...
gemma-4-E2B-it.litertlm: 100%|██████████████████████| 2.59G/2.59G [02:06<00:00, 20.4MB/s]
The capital of France is **Paris**.
Step 6.b. Run classic machine learning models
Find, download, and run the desired LiteRT model (e.g. efficientnet_b1). For example, you can download and run EfficicientNet for image classification:
litert download litert-community/efficientnet_b1 --output efficientnet
litert run efficientnet/efficientnet_b1.tflite --input
Example output
dino@dino:~ $ litert run efficientnet/efficientnet_b1.tflite --input
shark.JPEG
...
Outputs:
linear (Top 5 Predictions):
1: index 3 (tiger shark, Galeocerdo cuvieri) - score 7.4043
2: index 2 (great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias) - score 4.7619
3: index 4 (hammerhead, hammerhead shark) - score 3.3590
4: index 395 (gar, garfish, garpike, billfish, Lepisosteus osseus) - score 1.7937
5: index 0 (tench, Tinca tinca) - score 1.2683
Step 7. Run model with Raspberry Pi’s GPU (optional)
LiteRT uses WebGPU for GPU acceleration. It takes advantage of the Dawn WebGPU implementation, which can run on a Vulkan driver. Raspberry Pi 5 uses the V3DV open source Vulkan driver shipped with Mesa.
Experimental WebGPU support is available as of today. It can be enabled with the V3D_WEBGPU_OVERRIDE=1
environment variable using the updated official Raspberry Pi OS Mesa package.
WARNING: The GPU provides lower performance than the CPU. This is expected behavior, as WebGPU support is experimental in the V3DV Vulkan driver. Raspberry Pi’s CPU currently outperforms the GPU on this workload.
- Run LiteRT-LM using your Raspberry Pi’s GPU.
export V3D_WEBGPU_OVERRIDE=1
litert download litert-community/efficientnet_b1 --output efficientnet
litert run efficientnet/efficientnet_b1.tflite \
--input \
--gpu
Note: You need to set export V3D_WEBGPU_OVERRIDE=1
to activate GPU optimization.
What’s next?
We are excited to share that LiteRT integration and Gemma models are coming soon to Hailo AI accelerators! This update will allow you to seamlessly offload model inference to the Raspberry Pi AI HAT+ and AI HAT+ 2, delivering massive hardware acceleration benefits through the exact same LiteRT workflows you use today.
Explore our resources and start your journey with LiteRT:
- Official documentation: Access installation guides, API references, and quick-start tutorials on the LiteRT Developer Site.
- GitHub repos: Find the latest source code, implementation details, and updates on the LiteRT and LiteRT-LM GitHub repos.
- Samples and templates: Check out the LiteRT-Samples GitHub repo for reference code. Kick-start your own application using the Google AI Edge Gallery app.
- Ready-to-use models: Download optimized, open-weight models like the lightweight and powerful Gemma 4 E2B directly from the LiteRT Hugging Face Community.
We value your input. Please share your thoughts, feedback, or feature requests by opening an issue on our GitHub issue tracker. Share your cool Raspberry Pi + LiteRT + Gemma projects with @googlegemma. We can’t wait to see what you build!
Acknowledgements
Google: Changming Sun, Chintan Parikh, Cormac Brick, Daisuke Majima, Dillon Sharlet, Erin Walsh, Frank Barchard, Glenn Cameron, Ian Ballantyne, Jingjiang Li, Jun Jiang, Kimish Patel, Lu Wang, Matthias Grundmann, Rodney Witcher, Sachin Kotwani, Sasha Denisov, Scott Loftin, Shuangfeng Li, Somdatta Banerjee, Terry (Woncheol) Heo, Volodymyr Kysenko, Weiyi Wang, Yi-Chun Kuo, Yu-hui Chen, and the gTech team
Raspberry Pi: Ashley Whittaker, Naushir Patuck, and Sarah Cunningham
Igalia: José María Casanova
Hailo: Eldad Rubinstein
Ultralytics: Francesco Mattioli, Lakshantha Dissanayake, and Onuralp Sezer
Moonshine AI: Pete Warden
2 comments
Jump to the comment form
Alexi
Awesome! Using the AI HAT with USB bootdrive, instead of SD Card is better?
Benji
I wonder how it perform in comparaison of Ollama running the same model on the same device…
Facts Only
* Google and Raspberry Pi provide instructions for installing LiteRT on Raspberry Pi 5.
* Installation requires Raspberry Pi OS (64-bit) and a 64-bit aarch64 hardware architecture.
* The setup process involves using the Raspberry Pi Imager tool, an Ethernet connection, and SSH access.
* LiteRT CLI installation is managed via the uv tool manager and requires Python 3.13.
* Users must create a Hugging Face account and generate a Read access token to access models.
* LiteRT-LM is used to run large language models, such as gemma-4-E2B-it-litert-lm.
* The LiteRT CLI also supports classic machine learning models, such as EfficientNet for image classification.
* GPU acceleration is available via experimental WebGPU support using the V3DWEBGPUOVERRIDE=1 environment variable.
* Current performance data indicates the Raspberry Pi 5 CPU outperforms its GPU for these specific workloads.
* Future integration is planned for Hailo AI accelerators, including the Raspberry Pi AI HAT+ and AI HAT+ 2.
Executive Summary
LiteRT provides a framework for running high-performance on-device AI models on the Raspberry Pi 5. The deployment process involves a standardized pipeline: configuring a 64-bit Raspberry Pi OS, utilizing the uv manager for Python environment isolation, and authenticating via Hugging Face to pull optimized models. The system supports both Large Language Models (LLMs), specifically the Gemma 4 family, and classic computer vision models like EfficientNet.
While GPU acceleration is technically possible through experimental WebGPU and Vulkan drivers, it is currently less efficient than CPU execution on this hardware. The ecosystem is positioned for expansion, with upcoming support for Hailo AI accelerators designed to offload inference to dedicated hardware (AI HAT+), which is expected to provide significant performance gains over current general-purpose processor methods.
Full Take
This technical guide presents the strongest narrative of "democratized AI," moving powerful inference from centralized clouds to local, low-cost hardware. By providing a streamlined CLI and utilizing open-weight models like Gemma 4, the barrier to entry for edge AI is significantly lowered for hobbyists and developers.
However, the content functions as a classic vendor-driven ecosystem play. By tightly integrating the software stack (LiteRT), the model source (Hugging Face), and the hardware (Raspberry Pi/Hailo), the providers create a seamless "golden path" that encourages users to stay within their proprietary or partnered environments. The admission that the GPU currently underperforms the CPU is a rare moment of transparency, yet it serves to prime the reader for the inevitable upsell to the Hailo AI HAT+ hardware.
Patterns detected: ARC-0042 Authority Game
The underlying paradigm is the shift toward "Edge Intelligence," where the goal is to reduce latency and privacy risks by removing the cloud. While this enhances user agency, it also shifts the burden of compute and energy to the end-user. The second-order effect is a growing reliance on specific hardware-software bundles to achieve advertised performance.
If this were a coordinated influence campaign, the playbook would involve creating a "frictionless" entry point to make a specific software ecosystem the default standard for edge computing, thereby locking in developers before competing standards mature. The content here is a standard technical onboarding guide, though it aligns structurally with the goal of ecosystem capture.
Bridge Questions:
1. How does LiteRT's performance and resource overhead compare to decentralized alternatives like Ollama or llama.cpp on the same hardware?
2. To what extent does the reliance on Hugging Face tokens and specific Google-optimized models limit the ability to run truly independent, local-first AI?
3. Will the move toward dedicated AI HATs create a fragmented hardware market where software is only optimized for specific accelerators?
Sentinel — Human
This text functions as a detailed technical installation guide, characterized by clear steps and specific command-line instructions, exhibiting strong signs of human authorship aimed at practical instruction.