Featured projects
Today, Meta introduced Muse Glimmer, an open-weight, 30-billion-parameter model distilled from Meta’s Muse Spark for on-device agentic workflows. Alongside, ExecuTorch is adding end-to-end support for running Muse Glimmer on NVIDIA GPUs and Macs with Apple silicon.
Why ExecuTorch?
Most local AI frameworks rewrite models in other non-Python languages. That scaled well when LLMs were standard text transformers, but today’s models are becoming more complex – novel architectures, multimodal inputs and outputs, advanced decoding algorithms like DFlash (parallel diffusion-based speculative decoding) for low latency. Reimplementing these across different backends doesn’t scale.
ExecuTorch takes a different approach. As machine learning engineers and researchers, you implement the model (and its decoding strategy) in PyTorch. Once you’re ready for deployment, you export to ExecuTorch, and the framework handles backend-specific lowering, Triton on CUDA, MLX-native and custom Metal on Apple silicon. Ahead-of-time compilation optimizes the full execution path end-to-end, not just individual ops.
This is how we ship Muse Glimmer’s text and image inputs, direct GGUF export, native K-quant execution, 128K+-token context, and DFlash speculative decoding features. We have released prebuilt PTE artifact bundles that you can download and run on supported NVIDIA GPUs or Macs with Apple silicon using the ExecuTorch runtime.
Quickstart
Getting the PTEs
A PTE is the serialized artifact produced ahead of time from a model’s PyTorch graph by the ExecuTorch Python stack, and optimized for a target backend.
Download (Preferred)
We have published verified PTEs on Hugging Face for NVIDIA CUDA and Apple Silicon (Metal). This includes text-only and text-plus-image artifacts, with and without DFlash speculative decoding. Download them here: link.
Build your own
Starting with a prebuilt PTE is the fastest way to get running. To build your own, follow the ExecuTorch Muse Glimmer README, and select the backend, modality, context length, and whether to use DFlash. ExecuTorch exports directly from the released GGUF checkpoints through its torch.export-based ahead-of-time stack. CUDA export compiles and autotunes Triton kernels for the detected GPU architecture. For the best results, export on the same GPU architecture that will run the artifact.
Executing the PTEs
1. Build the runtime
ExecuTorch ships CMake presets for both the CUDA and MLX backends for this model runner(s). Follow ExecuTorch installation instructions here, and then use CMake to build the runners with or without speculative decoding for the PTE you selected. Both, with and without DFlash, runners support text and image modalities and are compatible with the example llm_server in ExecuTorch for agentic use cases.
Build the runtime ###
After installing ExecuTorch, build the runners for your backend:
$ cd examples/models/muse-glimmer
$ cmake --workflow --preset muse-glimmer-cuda # use muse-glimmer-mlx for macOS
This builds solo_runner, dflash_runner, and the serving worker.
2. Run the PTEs
Here are some examples of how to run the PTEs, once you have built the runners.
Example 1: Standalone infereance on cmdline ###
$ PROMPT='<|start|>user<|message|>Describe this image: <|eot|><|start|>assistant'
$ cmake-out/examples/models/muse-glimmer/dflash_runner \
--model_path artifacts/dflash-vision/model.pte \
--data_path artifacts/dflash-vision/aoti_cuda_blob.ptd \
--tokenizer_path assets/hf/tokenizer.json \
--image_path image.jpg --prompt "$PROMPT" \
--block_length 4 --n_draft 3 --temperature 0 --max_new_tokens 256
Example 2, step 1/2: Start the agent server ###
$ python -m executorch.examples.models.muse_glimmer.serving.serve \
--model-path artifacts/dflash-vision/model.pte \
--data-path artifacts/dflash-vision/aoti_cuda_blob.ptd \ # only for cuda
--tokenizer-path assets/hf/tokenizer.json --hf-tokenizer assets/hf \
--worker-bin cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \
--tool-parser atem --max-context 131072
APIs at http://127.0.0.1:8000/v1
Example 2, step 2/2: Start your agent (use Pi as example) ###
$ pi \
--provider muse-glimmer-local \
--model muse-glimmer \
--thinking high \
--tools read,bash,edit,write
This will automatically start your pi agent by using your local muse glimmer server.
Register muse_glimmer-local in ~/.pi/agent/models.json first.
See the README.md for more details.
Use cases enabled
Image understanding with Muse Glimmer, with and without speculative decoding
Figure 1: Muse Glimmer text-image input experiment on M5 Pro (64 GiB). Solo achieves 21.6 tok/s, while our speculative decoding set up (DFlash) reaches 33.0 tok/s, a 52.8% performance improvement without quality regression
Muse Glimmer powering Pi Coding Agent through ExecuTorch
Figure 2: Muse Glimmer agent pipeline on an M5 Pro (64 GB) using the Pi coding agent. The agent creates a bird-themed game, iteratively refining details through extended reasoning, calling tools to create files, installing required packages, writing and running tests, and proactively asking the user about next steps and additional requirements
Performance
Figure 3: Muse Glimmer performance on ExecuTorch using text-only input with varying context on NVIDIA A100 (as a proxy for RTX cards) and Apple Mac with an M5-max measuring prefill and decode performance in tokens/second with and without DFlash using coding prompt, which also has a good acceptance rate for this model, as seen in the decode charts
Under the Hood
Muse Glimmer now runs end-to-end on ExecuTorch on both NVIDIA GPUs and Apple Silicon GPUs. Here are some of the key capabilities and optimizations we built.
Enabling DFlash speculative decoding
- We optimized target and draft interoperability through weight sharing, exporting both into a single PTE.
- The DFlash block dimension is exported dynamically, allowing one PTE to support runtime-selectable block lengths.
- The runtime supports both greedy decoding and rejection sampling.
Supporting GGUF loading and k-quant
- We export straight from the GGUF released with the Muse Glimmer.
- We map Q4_K/Q5_K/Q6_K to packed INT4/5/6 with dp4a GEMV kernels on CUDA, and to repacked or fused Metal kernels on MLX.
- On MLX, for performance, at repack time we merge adjacent sub-blocks whose scale and min are identical into a larger group size, up to 128, whenever the merge is lossless.
Agentic harness and LLM serving
- One model load serves multiple isolated conversations, through per-session mutable-state rebinding we added to both backends.
- We added Harmony chat templating with reasoning routing.
- We added a parser for the model’s XML tool-call format, including multiple calls in one turn.
Backend-specific performance optimizations
CUDA
- We capture decode into a CUDA graph, reducing per-kernel launch overhead into one submission.
- Packed K-quant kernels accelerate low-batch decode, while length-aware split-K FlashDecoding++ paths optimize single-token decode and small DFlash verification blocks.
MLX
- RMSNorm, RoPE, SDPA, KV-cache updates, and quantized linear operations are lowered to MLX-native or custom Metal implementations.
- GGUF K-quant weights use either repacked MLX-native operations or fused Metal kernels.
Supporting long context
Muse Glimmer supports a 128K+ token context, and is efficient in how its KV-cache grows: only 13 of its 52 layers are global; the other 39 are sliding-window. ExecuTorch supports this efficiently, making the long context use cases practical on edge devices.
What’s next
- This initial release supports text and image inputs; video input is not yet supported. It is a work in progress.
- No cross-session prefix sharing or checkpointing or continuous batching as of now. These are all actively being worked on to make ExecuTorch even more suitable for agentic workflows.
Try Muse Glimmer with ExecuTorch and let us know what you think on Discord. If you run into any issues, feel free to open a Github Issue.
References
Muse Glimmer in ExecuTorch | Muse Glimmer on Hugging Face | ExecuTorch Documentation | ExecuTorch on Github
Facts Only
* Meta introduced Muse Glimmer, a 30-billion-parameter open-weight model.
* Muse Glimmer is distilled from Meta’s Muse Spark.
* The model supports text and image inputs and agentic workflows.
* ExecuTorch provides end-to-end support for running the model on NVIDIA GPUs and Apple silicon Macs.
* The model features a context length of 128K+ tokens.
* DFlash is a parallel diffusion-based speculative decoding algorithm used to reduce latency.
* ExecuTorch utilizes a torch.export-based ahead-of-time stack to produce serialized artifacts called PTEs.
* Prebuilt PTEs are available on Hugging Face for CUDA and Metal backends.
* The model uses GGUF checkpoints and supports K-quant execution (Q4K, Q5K, Q6K).
* The model utilizes a sliding-window KV-cache where 39 of 52 layers are sliding-window and 13 are global.
Executive Summary
Meta has released Muse Glimmer, a 30-billion-parameter open-weight model designed for on-device agentic workflows. To facilitate deployment, Meta is utilizing ExecuTorch, a framework that allows researchers to implement models in PyTorch and export them to backend-specific formats for NVIDIA GPUs and Apple silicon. This approach avoids the need to rewrite models in non-Python languages, supporting complex architectures and multimodal inputs natively.
The deployment emphasizes efficiency through DFlash speculative decoding and K-quantization, which significantly improve token-per-second performance on edge hardware. For example, on an M5 Pro, DFlash increases image-understanding speeds from 21.6 to 33.0 tok/s. While the current release supports text and image modalities and long-context windows via a hybrid sliding-window KV-cache, video input and cross-session prefix sharing remain under development. The ecosystem is positioned to power local agents, such as the Pi coding agent, by combining specialized runtime runners with an LLM serving worker.
Full Take
The strongest version of this narrative is that Meta is democratizing high-performance, multimodal AI by removing the "translation tax" usually associated with moving models from research (PyTorch) to edge hardware (CUDA/Metal). By integrating the model, the quantization method (GGUF), and the execution framework (ExecuTorch), they are creating a streamlined pipeline for local agentic intelligence.
However, this is a classic vendor-driven technical announcement. The evidence for performance gains is provided by the vendor using their own hardware (M5 Pro) and their own software stack to validate their own model. The narrative frames "reimplementing across different backends" as a scalability failure to position ExecuTorch as the inevitable solution, while the performance metrics serve as the load-bearing justification for adopting the entire Meta-ecosystem stack.
Patterns detected: ARC-0043 Authority Game
The underlying paradigm is the shift toward "Edge AI Sovereignty," where the value moves from the cloud API to the local runtime. The unstated assumption is that users possess the specific high-end hardware (e.g., M5 Max, A100) required to realize these "edge" gains. This echoes the historical pattern of hardware-software bundling used to create ecosystem lock-in.
The benefit accrues to developers who gain powerful local tools, but the second-order consequence is a tighter coupling between model architecture and specific hardware optimizations, potentially making "open weights" less portable if they rely on proprietary runtime "blobs" for efficiency.
Bridge Questions:
1. How does Muse Glimmer’s performance compare when run on non-Meta optimized frameworks?
2. If the efficiency relies on specific "aoticudablobs," to what extent is the model truly "open" in its execution?
3. What are the privacy and security trade-offs of moving agentic tool-calling (bash, edit, write) to local execution?
Counterstrike Scan: A coordinated campaign would push this by emphasizing the "death of the cloud" and the "liberation of local AI" to drive adoption of a specific hardware/software stack. The actual content is a technical product announcement; while it promotes a specific ecosystem, it provides enough implementation detail to remain a standard technical release rather than a psychological operation.
Sentinel — Human
The article reads like a highly detailed technical announcement by an involved researcher, effectively explaining a complex system's architecture and deployment strategy through practical examples.