Running XIAOMI-OCR-0 natively in looksneat
XIAOMI's 0.8B OCR model runs through Rust and MLX in a 645 MB package. A synthetic invoice shows the first integration's text recognition and prompt-driven extraction.
- Published
- Reading time
- 9 min read
The latest addition to my local looksneat build is XIAOMI-OCR-0, a document model with 0.8 billion parameters. It runs inside the Rust program through MLX on Apple Silicon. The pinned 4-bit weights and tokenizer take about 645 MB to download.
Alongside text recognition, I can ask it a question about a document or request particular fields. That is useful for scripts that need an invoice number or a total rather than the entire page. My first small extraction example also produced a made-up company name and date, so I have a concrete reason to keep checking what comes back.
The examples below use a release-mode development build from November 5. The XIAOMI implementation is now available on looksneat’s public main branch.
XIAOMI-OCR-0
XIAOMI released the weights and technical report on September 30, 2026. The model builds on Qwen3.5-0.8B-Base, with training for document parsing, text recognition, and questions about document content. The code and model weights use Apache-2.0.
The model card describes an approximately 170-million-sample OCR training corpus. Part of the data preparation compares annotations from several existing OCR models. When their outputs disagree, candidate tables and formulas can be rendered back into images and checked against the source. That gives the training process a way to catch structural mistakes that a plain text comparison could miss.
In looksneat, text extraction, document parsing, table and formula recognition, and custom prompts fit behind the existing --model, --task, and --prompt options.
My earlier PaddleOCR-VL integration already accepted a custom prompt, but its task-specific checkpoint often returned ordinary OCR instead of following a free-form instruction. XIAOMI’s training includes field extraction and visual question answering. It gives me another model to try when the requested result is more specific than a transcription.
Invoice extraction
I used the synthetic fixture from the new integration test. There are only two lines, with no seller, address, or date to infer:
Ordinary OCR read it correctly:
looksneat --model xiaomi-ocr-0 --max-tokens 128 \
--format json invoice.png | jq -r '.pages[0].blocks[0].text'
INVOICE 12345
TOTAL 42.50
Then I asked for the two fields as JSON:
looksneat --model xiaomi-ocr-0 --max-tokens 128 \
--prompt 'Extract the invoice number and total. Return only a JSON object with keys invoice_number and total, both strings.' \
invoice.png
The response contained the correct values. It also added fields I had not requested, including details that were absent from the image. This is an excerpt of the returned object:
{
"title": "Invoice",
"source": "Company Name",
"date": "2023-04-05",
"invoice_number": "12345",
"total": "42.50"
}
The full response also invented an address and contact details. It wrapped the object in a Markdown code fence despite the request for only a JSON object.
I repeated the run with this instruction:
Read only the visible text. Return exactly two JSON string fields: invoice_number and total. Do not add other keys, explanations, or information that is not visible.
That returned:
{"invoice_number": "12345", "total": "42.50"}
The stricter prompt worked on this image. I have not established that it fixes extraction on real invoices. These runs used the community 4-bit conversion through my native port, so they also do not isolate the original model’s behavior from quantization or implementation differences.
For an application, I would validate the field names and types, reject unexpected fields, and preserve the image for checking the values. A JSON parser would have accepted the invented date just as readily as the correct total.
JSON output
There are two JSON formats involved here. --format json selects looksneat’s own document format. It records the model, source hash, image dimensions, and recognized blocks. That outer document is serialized by Rust.
A JSON object requested through --prompt is generated text inside a block. The current integration does not enforce a JSON schema on it. Combining --format json with a JSON extraction prompt gives me a valid outer document whose block text may still contain a code fence, extra fields, or an incorrect answer.
A script can parse looksneat’s document to retrieve the text, then validate the generated result against the application schema. Checking whether the amount appears in the image requires a separate comparison with the source.
Document parsing
The other tasks use XIAOMI’s published task instructions:
looksneat --model xiaomi-ocr-0 --task document report.png
looksneat --model xiaomi-ocr-0 --task table table.png
looksneat --model xiaomi-ocr-0 --task formula equation.png
The document prompt asks for the main body in Markdown, in reading order, with headers and footers ignored. Formulas use LaTeX. Tables use OTSL, a sequence of tokens that describes table cells and their relationships. A consumer expecting an HTML table needs a conversion step.
This first provider processes the whole image in one generation call. A document comes back as one text block. Table and formula tasks label the block accordingly, but there are no region coordinates or confidence values. The model’s reading order is represented by the order of its generated text.
That differs from looksneat’s PaddleOCR document mode, which detects page regions with PP-DocLayoutV3, crops them, and recognizes each retained region separately. That path can return located blocks. XIAOMI’s first integration does not use it, even though both models appear in the same CLI.
For coordinates, I would keep using the spatial OCR engines or the Paddle document pipeline. XIAOMI’s current provider is suitable for trying whole-image parsing and prompted extraction. It rejects the Paddle-specific layout switches, and it does not implement lines, spotting, chart, or seal tasks.
Recent commits also added image batches and PDF input to looksneat. On macOS, PDF pages are rasterized lazily, with an ordered buffer for parallel rasterization. Recognition remains serial, and the recognizer keeps its weights and tokenizer loaded across pages. XIAOMI uses that shared input path while recognizing each page as one whole image.
Single-image JSON retains schema version 2. Batches and PDFs use version 3, with source provenance on each page. The image preprocessing described below still applies to each rasterized page before XIAOMI sees it.
Native implementation
The recent committed work in looksneat added line recognition, annotated images, page orientation and unwarping, and controls for PaddleOCR’s layout pipeline. The latest commits extend that work with bounded multi-page processing and lazy PDF rasterization. XIAOMI adds a separate model implementation alongside those engines.
It has a twelve-layer vision encoder and a twenty-four-layer Qwen3.5 language decoder. Eighteen decoder layers use gated-delta recurrence; six use full attention. Reusing the PaddleOCR-VL decoder would give the wrong computation, so the new provider has its own model code and decoding state.
The port uses a community MLX conversion with affine 4-bit quantization in groups of 64. Some tensors remain at higher precision. The main projections stay packed for MLX’s quantized matrix multiplication instead of expanding the entire model into floating-point weights.
The conversion’s normalization weights already include Qwen’s RMSNorm offset. Adding the offset again would change the computation. The native code consumes the converted weights directly. The recurrence also keeps its running state in float32 and uses a chunked scan while processing the input sequence.
These implementation details sit behind the same command-line interface as the other OCR engines. No Python worker, Ollama process, or separate model server runs during inference. The model architecture is XIAOMI’s and Qwen’s work; the conversion is community work. My addition is the Rust/MLX runtime and its integration into looksneat.
Model setup
The provider’s setup command is:
looksneat --model xiaomi-ocr-0 --setup
It downloads two files from a pinned revision: 625,228,929 bytes of model weights and 19,989,339 bytes of tokenizer data. looksneat checks their sizes and SHA-256 hashes before using them. XIAOMI setup does not fetch the Paddle layout detector.
The combined 645 MB is a download size. I have not measured a representative memory or latency profile for this integration. The invoice checks ran on an M2 Ultra with 128 GiB of unified memory and macOS 15.8.1.
Images first receive a 2,048-pixel long-edge bound, then resizing on a 32-pixel grid with a minimum area of 65,536 pixels. That preprocessing changes how much detail the model sees. The default generation limit is 4,096 tokens; reaching it produces a warning on stderr. A long page can need a larger limit, and increasing it still requires checking that the output finishes correctly.
Evaluation
XIAOMI reports 96.83 on OmniDocBench v1.6 and 95.24 on Real5-OmniDocBench. Its official repository explicitly identifies the headline document-parsing results as two-stage results.
The evaluation pipeline detects layout regions with PaddleX’s PP-DocLayoutV3 at a threshold of 0.25, recognizes the crops, and assembles the page. It also converts OTSL tables to HTML before evaluation. My whole-image path omits those stages and uses quantized weights. I have not reproduced those scores for it.
The local tests check narrower things. The opt-in model test renders the small invoice, looks for its number and total, and checks source metadata and the absence of fabricated coordinate or confidence fields. Unit tests cover preprocessing, prompt formatting, recurrent computation, and decoding behavior. They help catch errors in the runtime; they do not measure document accuracy across a collection of real pages.
For the next evaluation, I would use pages with known text and fields, including cases where a requested field is missing. The invented date in the invoice example gives me a specific failure to test. Comparing the native 4-bit output with an upstream reference is also still needed before making numerical equivalence claims.
I described the existing OCR engines and the reason for building these runtimes in Neat discoveries in AI. The Data Berlin recording shows the project before this XIAOMI addition.