Model preparation

This page lists the models in the local AI bundle and explains how to prepare, run, and check them offline. It's for developers who build Lumen with local transcription, speaker labels, generated notes, and semantic retrieval.

Included models#

The local AI bundle for Apple Silicon includes these revision-pinned models. Developer builds bundle them. A small distribution build downloads the same files, once, from the same Hugging Face repositories and revisions when the person chooses Get started or Download; see Development. Lumen never downloads models on its own.

For upstream sources, see the Whisper conversion, the Gemma conversion, and EmbeddingGemma 2. catalog.json and the search catalog contain the exact pins, and requirements.lock pins the Python dependencies.

Prepare the local AI bundle#

Before you begin#

• Use an Apple Silicon Mac running macOS.
• Install uv from its official distribution.
• Make sure you have enough disk space. The current prepared bundle occupies about 6.8 GB: 6.1 GB of model files and a 674 MB Python runtime. Allow additional space for source preparation, staged app copies, and audio.

Prepare, build, and open the app#

1. Download and package the bundle:

   sh

   bash scripts/prepare-local-models.sh --download

   The script downloads standalone CPython 3.11.13, locked dependencies (no PyTorch), and pinned model snapshots to the ignored build/local-ai directory. It checks model LFS files against upstream SHA-256 values. It packages safetensors, config, and tokenizer files plus the explicitly pinned Silero ONNX artifact, inventories model files and adapters, and includes license notices.

2. Build and sign the app, then open it. Wait for signing to finish before you open the app.

   sh

   ./scripts/build-app.sh release
   open build/Lumen.app

The app bundle includes its own Python, so users don't need a system Python environment.

When the app starts, it verifies the bundled adapters and then prepares local search vectors. Imported adapters take precedence for their supported operations.

Note: A base build without the local AI bundle keeps keyword search, recording, and manual notes. It doesn't fetch missing models.

Inference behavior#

Speech#

Silero VAD (ONNX, through sherpa-onnx) first identifies voice regions locally on CPU. Whisper uses MLX, local decoded PCM, and chunks of at most 30 seconds, and returns word timestamps. Whisper doesn't require an external ffmpeg executable. Silent and non-speech chunks return no transcript.

The detector and boundary padding reduce silence hallucinations, but they don't guarantee error-free recognition.

Live recording feeds only finalized chunks. The first transcript normally follows a roughly 25-second chunk plus processing time.

Speaker analysis#

After you stop recording, the speaker package uses pyannote segmentation and 192-dimensional multilingual CAM++ voice embeddings through sherpa-onnx 1.13.8 on CPU. Session-only matching links voices across bounded chunks, independently per track. An embedding requires at least 1.5 seconds of clean speech. Ambiguous or overlapping speech stays unassigned.

Word alignment preserves transcript text. Names are user-confirmed, not verified identities.

Meeting notes#

Gemma uses MLX-LM at four-bit weight precision. It summarizes bounded transcript batches and returns notes with validated references to the original segments. Citation aliases are withheld from owner fields, and optional owner and due text must occur verbatim in the cited evidence.

Source links still require human review for semantic accuracy. A grounded-answer runtime operation exists, but no written-answer UI is shipped.

Text and image retrieval#

One EmbeddingGemma 2 serves every vector in Lumen: the Q8_0 GGUF model and projector run by the native search engine (llama.cpp) embed files, images, meeting segments, memory, and Ask questions in a shared text/image space with explicit query and document roles. Each index stores the embedding fingerprint it was built with; when the model or input recipe changes, Lumen rebuilds the vectors automatically and keeps keyword text searchable meanwhile. The earlier Sentence Transformers copy of the model (1.4 GB of safetensors plus PyTorch) and its reference adapter were removed.

The separately prepared search package adds direct text, image, and audio embeddings, video scenes, and Apple Vision OCR. To add it, run ./scripts/prepare-search.sh and then rebuild. For details, see Native multimodal search. Visual embeddings alone don't provide OCR. Native image previews and a model runtime can support different codecs.

Runtime isolation#

All runtime loads use explicit local paths, offline flags, and no remote model code. The signed app, worker, and sandbox-inheriting Python have no network entitlement. Models unload between bounded requests.

Long corpora and CPU fallback can take time. Capture and meeting work take priority over background search preparation.

Run real offline checks#

Run the standalone harness#

The standalone harness synthesizes a short English audio file without recording or playing room audio. It then checks actual speech, source-linked notes, digital silence, steady noise, and two synthetic image descriptions.

build/local-ai/bundle/Python/bin/python3.11 -B Runtimes/LocalModels/evaluate.py \
  --bundle build/local-ai/bundle --output Evaluation/results/local-models

The harness runs child inference under an OS network-denying sandbox and saves synthetic-only results in the ignored Evaluation/results/local-models directory.

Run the Swift integration check#

The Swift integration check exercises verified adapters, actual inference, and SQLite file and image retrieval. Compile the test outside sandbox-exec, then run the compiled test with networking denied:

swift test --filter LocalModelIntegrationTests
LUMEN_TEST_MODELS_DIR="$PWD/build/local-ai/bundle" \
  /usr/bin/sandbox-exec -p '(version 1) (allow default) (deny network*)' \
  xcrun xctest -XCTest LumenCaptureTests.LocalModelIntegrationTests/testRealSpeechNotesAndImageRetrievalThroughVerifiedAdapters \
  .build/debug/LumenCaptureTests.xctest

The normal check script skips this opt-in test when models aren't provided.

Results and limitations#

The executed synthetic checks passed on an M1 Pro with 16 GB RAM. They establish working integration on that setup. They don't establish broad language or hardware quality, live call routing, speaker diarization, or battery performance. For details, see the validation record.

Speaker package and evaluation#

The speaker package adds approximately 34 MiB of weights. The models are pinned to the sherpa-onnx maintainer’s conversion repositories at 9403a6902bb58e3d5ae8c7e77c3422de279db2e0 (segmentation) and 0743f301363dec56491a490f6d6cbc9d67f9a3bf (CAM++ zh/en advanced). The catalog records artifact SHA-256 values and license notices.

The app never asks for a model account, and it downloads weights only when the person chooses Get started or Download (from these pinned Hugging Face revisions). Developer preparation downloads the published ungated ONNX conversions and bundles them. The core contracts and CPU ONNX model choice permit future portable backends; the current app and runtime package still target Apple Silicon macOS.

To run the speaker evaluation:

1. Generate labeled synthetic voices:

   sh

   build/local-ai/bundle/Python/bin/python3.11 -B Evaluation/prepare-speaker-fixtures.py

2. Compile the test:

   sh

   swift test --filter SpeakerIntegrationTests

3. Run the compiled test with LUMEN_TEST_MODELS_DIR and the same deny-network xcrun xctest pattern as the Swift integration check, selecting LumenCaptureTests.SpeakerIntegrationTests/testRealLocalSpeakerPipelineOnLabeledAudio. For the full command, see Local speaker evaluation.

The test verifies actual speech and word timings, automatic two-voice grouping, repeat-voice matching across chunks, retained text, unchanged input, silence and noise, editable names, export, and cited notes. Voice embeddings and centroids are never saved in session metadata or the search index.

Note: A small synthetic test isn't a speaker-quality benchmark.

Platform direction#

Request and response contracts and model provenance are independent of the UI. This bundled MLX runtime currently targets Apple Silicon macOS. Windows and Linux need corresponding speech and generation backends, packaging, native capture and extraction adapters, and their own tests. The text/image adapter can use a compatible local Torch environment on those platforms, but that alone isn't a supported application port.