From the paper
to your prompt.
Get the weights, pick a runtime and put fixed-capacity history to work.
Your first
generation.
Start with the original model and portable PyTorch backend. Run these commands on your chosen compute host.
Full installation guide1. Prepare the environment
Install a compatible PyTorch build for your platform, then install the repository’s bundled Transformers dependency and runtime.
git clone https://github.com/rockmatrix-ai/EdgeRock.git
cd EdgeRock
python -m venv .venv
source .venv/bin/activate
# Install PyTorch for your platform before continuing.
python -m pip install huggingface_hub
python -m pip install ./wheels/transformers-5.7.0.dev0-py3-none-any.whl -e .
hf download rockmatrix-ai/EdgeRock --local-dir checkpoints/EdgeRock-4B \
--include '*.safetensors' '*.json' '*.jinja' 'tokenizer*' 'vocab*' 'merges*' \
--exclude 'evaluation_data/*'
cp config.json configuration_edgerock.py modeling_edgerock.py checkpoints/EdgeRock-4B/2. Generate a response
The portable backend runs on a CUDA device with PyTorch. For a CPU run, change --device cuda to --device cpu; CPU execution uses FP32 and can be substantially slower.
python -I run.py demo --model checkpoints/EdgeRock-4B \
--backend torch --device cuda \
--prompt 'Compute 17 × 23.' --max-new-tokens 5123. Pick the right execution path
For optimized Hopper execution, use the repository’s pinned Linux CUDA environment and Dockerfile. For text serving, streaming responses and batch generation, follow the examples in the usage guide.
Choose for
your constraints.
Hardware, memory and architecture support come first.
The release matrix describes the available paths and their tested scope. Use the package notes for the exact dependencies and supported operators.
| Path | Where to start | What to know |
|---|---|---|
| Portable PyTorch | Original runtime ↗ | CPU or supported CUDA. Use the matching model modules and chat template. |
| Optimized CUDA | Hopper runtime ↗ | Linux GPU host. Pinned dependencies; new shapes may compile on first use. |
| Compact PyTorch | NF4 ↗ · FP8 ↗ · INT8 ↗ | Text inference adaptations. Limited capability screening is separate from the full report. |
| GGUF / Ollama | Custom RMA backend ↗ | Validated on Linux x86 CUDA. Requires custom operators; stock runtimes are insufficient. |
| MLX | 4-bit adaptation ↗ | Validated on Linux CUDA. Apple Metal execution remains unvalidated. |
| ONNX | CUDA ↗ · CPU ↗ | Custom RMA execution. Mobile ONNX is not validated by these packages. |
| Android / LiteRT-LM | GPU development preview ↓ | INT4 text inference on a selected Adreno device, with a custom runtime and Gallery app. |
Successful conversion and functional execution do not imply benchmark parity, universal device support or production readiness.
A smaller model.
A closer connection.
Android development previewExplore local text inference with the full 4B model in an INT4 package.
The LiteRT-LM preview includes a custom GPU RMA path and a matching Gallery build. It is a development route for a specific tested device, rather than general Android compatibility.
INT4 weights · Custom GPU runtime · Text inference
- Tested device
- SM-S9280 / Adreno / Android 16
- Model package
- 2.622 GB
- Median decode rate
- 5.483 tokens/s
Three Gallery 48 runs: 210 actual input tokens and 256 output tokens, with decode rates 6.130, 5.483 and 5.473 tokens/s. FP32 compute/state, no speculative decoding. Gallery 49 separately fixes initialization timing; its check is not pooled into this median. These are device-specific text-inference checks, not a capability benchmark.
Make your
comparison useful.
Start from the workload your application actually runs: input and output lengths, concurrency, state precision and your hardware budget.
Keep precision and reuse explicit
For the current primary comparison protocol, use BF16 weights, KV and recurrent-state storage, including prefill. Keep FP32 update accumulation separate. Disable cross-request prefix reuse on both sides, and evaluate without INT8, speculation or host offload.
Separate latency from throughput
Report decode-only latency separately from end-to-end output throughput. State whether prefill, loading, warmup and tokenization are included. Repeat timing in fresh processes and record peak device allocation.
The published report and some runtime documentation describe an earlier precision and prompt-sharing profile. Check the exact release and configuration before comparing it with a newer BF16-state, independent-prompt result.
A clear path
through the details.
Why use the bundled Transformers wheel?
The runtime uses EdgeRock-specific configuration and model modules. Install the matching dependency and copy the supplied modules into the checkpoint directory as shown in the repository guide.
Can I use a standard chat-completions client?
The repository includes a lightweight text API with chat completions and SSE streaming. Follow its serving example for endpoints and supported options. It is a local runtime you host, rather than a hosted Rock Matrix API service.
Is every package ready for production?
Each package has its own validation scope. Original model evaluations, quantization screening, functional runtime checks and the Android preview answer different questions. Review the relevant release notes before deployment.
How do I report an issue?
Share your runtime revision, model package, hardware, command and a small reproduction in the GitHub issue tracker. Keep credentials and private prompts out of public issues.
Building something ambitious?
Bring us a concrete workload or a difficult deployment constraint.