Getting MiniLM-L6-v2 Onto SiMa's Modalix MLA

Exporting a HuggingFace BERT model, rewriting its ONNX graph for hardware compatibility, and deploying it to run sentence embeddings in ~7 milliseconds.

MiniLM-L6-v2 is a small, fast sentence-embedding model - the kind of thing that powers semantic search, duplicate-question detection, and document similarity. On its own it already runs in tens of milliseconds on a CPU. Point it at SiMa's Modalix Machine Learning Accelerator instead, and that drops to under 7ms - but only after the model gets translated into a dialect the chip actually understands.

The MLA doesn't run arbitrary ONNX. It supports a fixed set of roughly 60 int8 operators, and a stock BERT export from HuggingFace is full of things outside that set - dynamic shapes, Gather-based embedding lookups, Gemm, NaN guards that never actually fire. Getting from a HuggingFace checkpoint to a working MLA binary means exporting to ONNX, surgically rewriting the graph operator-by-operator, quantizing to int8, compiling to a hardware binary, and shipping the result to the DevKit.

The Six Steps

Two ideas do most of the work: freeze every shape to static values at export time (the MLA compiler requires fixed input shapes), and move anything the chip can't run - the embedding table, the attention mask construction - off-chip onto the CPU.

How the Chip Is Actually Organized

The Modalix DevKit isn't a single accelerator - it's three compute domains working together, and knowing which one does what explains most of the surgery decisions later.

Prerequisites & Environment

You'll need an x86_64 Linux dev machine with Docker, the SiMa DevKit reachable over Ethernet, and the SDK container already running with the Model Compiler inside it.

# Enter the running SDK container
docker exec -it <container_id> bash

# Activate the Model Compiler virtual environment
source /sdk-extensions/model-compiler/bin/activate

# Confirm the version
python3 -c "import afe; print('AFE version:', afe.__version__)"
# Expected: AFE version: 2.1.0

# SSH into the DevKit (password: edgeai)
ssh sima@192.168.0.28

# Confirm pyneat is available
python3 -c "import pyneat; print('pyneat version:', pyneat.__version__)"
# Expected: pyneat version: 0.2.0

Step 1: Export to ONNX

The Model Compiler doesn't take HuggingFace .bin or PyTorch .pt files directly - it wants ONNX (its most mature import path). Every shape has to be frozen at export time; the MLA compiler has no tolerance for dynamic dimensions.

# Run the export script (see export_minilm.py for the full source)
# Loads sentence-transformers/all-MiniLM-L6-v2, exports with static
# shapes (batch=1, seq_len=128), opset 17, and simplifies with onnxsim
python3 export_minilm.py
# -> minilm-l6-v2.onnx (~90 MB)
# -> minilm-l6-v2.sim.onnx (~90 MB, after onnxsim simplification)

Step 2: Graph Surgery

This is where most of the real work happens. The exported ONNX graph has to be audited against the MLA's supported-operator list, and every unsupported op removed, replaced, or moved off-chip.

# Audit the model for unsupported operators
sima-neat-sdk_audit_model --model_path minilm-l6-v2.sim.onnx --dtype int8 --json_output

Six surgeries turn the audit results into a compilable graph:

# Run the surgery script (see surgery_minilm.py for the full source)
python3 surgery_minilm.py
# -> minilm-l6-v2.surg.onnx (~41 MB)

# Validate: compare FP32 outputs of the original vs. surgically modified model
python3 validate_surgery.py
# Expected: max absolute difference < 1e-5

Step 3: Quantization

Quantization converts weights and activations from float32 down to int8 - the MLA's native compute precision. That's a 4× reduction in memory and a meaningful speedup, at a cost of roughly 0.1% accuracy loss on sentence similarity tasks for this model.

# quantize_clean.py — quantization config for this model
from afe.apis.defines import (
    default_quantization, quantization_scheme,
    RequantizationMode, CalibrationMethod
)

# 8-bit activations (symmetric, per-tensor)
act_scheme = quantization_scheme(quantize_activations=True,
                                  quantize_weights=False, num_bits=8)

# 8-bit weights (asymmetric, per-channel)
wt_scheme = quantization_scheme(quantize_activations=False,
                                 quantize_weights=True, num_bits=8)

quant_config = default_quantization \
    .with_activation_quantization(act_scheme) \
    .with_weight_quantization(wt_scheme) \
    .with_requantization_mode(RequantizationMode.sima) \
    .with_calibration(CalibrationMethod.from_str("mse"))

quant_model = loaded_net.quantize(
    calibration_data=calib_data,
    quantization_config=quant_config,
    any_shape_on_mla=True,             # BERT's 3D tensors, not 4D image tensors
    automatic_layout_conversion=True,  # ONNX is NCHW, MLA expects NHWC
    model_name="minilm-l6-v2",
)

Step 4: Compilation

Compilation schedules operators across the three compute domains, tiles tensors to fit the MLA's 128 KB local SRAM, maps memory addresses, and produces .elf binaries the MLA can execute directly.

# THE KEY INSIGHT: pass a DIRECTORY path, not a file path
quant_model.compile(
    BUILD_DIR,       # Directory path -> produces model_mpk.tar.gz
    batch_size=1,
)

Compile time is roughly 3 minutes (quantize + compile combined). The output directory contains the deployable .tar.gz alongside the raw .elf binary and a set of pipeline-stage config JSONs (quantization/tessellation, buffer concat, MLA processing, post-processing).

Step 5: Deploy to the DevKit

Pull the MPK archive out of the container, then copy it onto the DevKit over SSH.

# Copy the archive out of the container onto the host
docker cp <container>:/workspace/build_minilm_v5/minilm-l6-v2_mpk.tar.gz \
    /host/path/minilm-l6-v2_mpk_nopooler.tar.gz

# Copy it onto the DevKit (password: edgeai)
scp minilm-l6-v2_mpk_nopooler.tar.gz sima@192.168.0.28:/home/sima/models/

# Verify the archive - it MUST contain at least one .elf file
tar tzf /home/sima/models/minilm-l6-v2_mpk_nopooler.tar.gz

Step 6: Run Inference on the DevKit

With the archive in place, load the model and push tensors through it directly.

# Run the inference script (see minilm_inference.py for the full source)
# Loads the MPK with mla_only=True, pushes hidden_states + attention_bias,
# pulls the raw int8 output, and dequantizes it manually on the CPU
python3 minilm_inference.py

For context, the same model takes ~80ms on one Xeon CPU core, ~15ms across 8 cores, and ~5ms on a T4 GPU - the GPU is marginally faster but draws roughly 75W against the MLA's ~5W.

Troubleshooting Common Failure Modes

Command Quick Reference

All six steps, one command block per machine, in the order you'd actually run them.

# ── HOST, ENTER CONTAINER ────────────────────────────
docker exec -it <container_id> bash

# ── INSIDE THE CONTAINER ─────────────────────────────
source /sdk-extensions/model-compiler/bin/activate
python3 export_minilm.py
sima-neat-sdk_audit_model --model_path minilm-l6-v2.sim.onnx --dtype int8 --json_output
python3 surgery_minilm.py
python3 validate_surgery.py
python3 compile_v2.py
exit

# ── HOST -> DEVKIT ───────────────────────────────────
docker cp <container>:/workspace/build_minilm_v5/minilm-l6-v2_mpk.tar.gz .
scp minilm-l6-v2_mpk_nopooler.tar.gz sima@192.168.0.28:/home/sima/models/
ssh sima@192.168.0.28

# ── ON THE DEVKIT ────────────────────────────────────
tar tzf /home/sima/models/minilm-l6-v2_mpk_nopooler.tar.gz
python3 minilm_inference.py

Quick Glossary

• MLA - Machine Learning Accelerator, the neural network compute engine on Modalix.
• CVU / EV74 - Computer Vision Unit, handles tensor pre- and post-processing.
• Tessellation - slicing large tensors into tiles that fit the MLA's local memory.
• MPK - Model Pack, the deployable archive containing .elf binaries and pipeline config JSONs.
• Scale / Zero Point - the two numbers defining int8↔float32 mapping: float = (int8 - zp) × scale.
• Orphaned initializer - a weight tensor left in the graph file after surgery disconnected it from any node.