Evaluation

This page explains how to reproduce Lumen's retrieval, transcription, multimodal search, and speaker evaluations. It's for developers who measure local runtimes or check a prepared bundle.

Correctness tests and model quality measurements are separate. All files in Fixtures are clearly synthetic. Their excerpts appear only if you explicitly select that folder for indexing.

Evaluate fixture retrieval#

To generate the fixtures, build the evaluator, and run it with networking denied:

swift Evaluation/make-fixtures.swift
swift build
BIN_DIR="$(swift build --show-bin-path)"
sandbox-exec -p '(version 1) (allow default) (deny network*)' \
  "$BIN_DIR/LumenEvaluate" --fixtures "$PWD/Evaluation/Fixtures" \
  --worker "$BIN_DIR/LumenWorker"

The executable does the following:

• Indexes real fixture files through the extraction helper into temporary SQLite.
• Tests exact invoice numbers, a quoted phrase, a name, a common word, and absent evidence.
• Prints source and page references plus pass counts.
• Decodes and indexes the two generated synthetic image fixtures and reports their count. This step doesn't test semantic model quality.
• Deletes only its own temporary evaluation directory.

Note: Timing is fixture-specific and isn't a model speed or hardware support claim.

Evaluate visual retrieval with prepared artifacts#

1. Prepare the Search package (EmbeddingGemma 2, GGUF) with ./scripts/prepare-search.sh; see Native multimodal search.
2. With networking denied, run LumenEvaluate --embedding-tools <package> --query <description> --images-only.

The command indexes actual sources, creates real vectors through that adapter, and prints ranked sources with elapsed time and artifact identity. These are inspection results; they don't set a quality pass/fail threshold. The prepared local bundle has also passed the real-model synthetic check described in Executed bundled-model check.

Evaluate transcription#

Before you begin#

• A checksum-verified local package
• A consented audio clip of at most 25 seconds
• A UTF-8 reference transcript
• BIN_DIR set as in Evaluate fixture retrieval

Run the evaluation#

sandbox-exec -p '(version 1) (allow default) (deny network*)' \
  "$BIN_DIR/LumenEvaluate" --worker "$BIN_DIR/LumenWorker" \
  --tools /absolute/path/to/local-package \
  --audio /absolute/path/to/clip.caf --duration 25 \
  --reference /absolute/path/to/reference.txt

The process records the pinned model and revision, runtime elapsed seconds, the actual transcript, and the word error rate with lowercased letter/number tokenization. For comparable measurements, use the same normalization, quantization, and pinned runtime.

This small harness doesn't compare diarization, streaming latency, partial results, or summaries. For the executed synthetic speech and retrieval checks, see Executed bundled-model check.

Required target matrix#

Broader evaluation needs to cover these targets:

Note: These are test targets, not established supported configurations. The design kit's model memo remains a research baseline rather than a guarantee.

Before you enable automatic model selection, evaluate the following on representative consented content:

• Parakeet v3 or Qwen3-ASR
• Whisper turbo, as a comparison
• Gemma generation
• EmbeddingGemma retrieval

Record exact artifact licenses and provenance rather than copying unverified research license claims.

Executed bundled-model check#

The local-model harness runs actual Whisper speech, Gemma source-linked notes, and EmbeddingGemma 2 text/image retrieval with networking denied. It uses a short synthesized English recording and generated images, without recording or playing room audio. The separate Swift integration case also checks semantic file retrieval through verified adapters and SQLite.

Both passed on an M1 Pro with 16 GB RAM. Results are kept in the ignored Evaluation/results/local-models directory.

Note: This small check doesn't establish broad language, speaker, hardware, or live call quality.

Native multimodal search (DigUp adaptation)#

To prepare the search package explicitly and run a labeled synthetic end-to-end check on macOS:

./scripts/prepare-search.sh
python3 Evaluation/prepare-search-fixtures.py
python3 Evaluation/evaluate-intelligent-search.py --package build/local-ai/bundle/Tools/Search

Fixtures#

Fixture preparation uses the installed Samantha system voice and afconvert. It creates the following sources:

• A scanned PDF
• An OCR receipt image
• Rich and text documents
• Two spoken topics
• A two-scene video
• A code definition and notebook
• Negative and excluded sources

It writes only under build/search-evaluation and never uses private meeting recordings.

Evaluator checks#

The evaluator launches every worker with networking denied. It verifies actual vectors and keyword results, plus page, time, line, and cell locations, and writes build/search-evaluation/report.json. It also checks the following:

• A one-image collection and an unrelated query
• Absent exact evidence
• Folder and date filters
• Code opt-in revocation
• Git and vendor exclusions
• Notebook outputs
• Source changes
• Revoked folder scope

Note: The signed helper requires Lumen as its sandbox parent, and macOS rejects a direct standalone launch of that inheriting executable. Exercise the shipped package through the app UI, and use the explicitly prepared package for this CLI harness.

When the explicitly prepared native archive exists, the ordinary scripts/check.sh runs the adapted engine's extraction, ranking, and regression tests. It never downloads weights. Actual inference remains opt-in.

Results#

The initial 35-file trial returned 25/25 checks after two fixes: grayscale-only video deduplication, and checking index recipe changes before the first batch. An expanded run also validates precise notebook cell mapping and a one-image index.

On the development M1 Pro, steady semantic queries took about 1–2 seconds and exact identifiers about 0.1 seconds. First Metal/OCR initialization was substantially slower.

Note: These are small synthetic checks, not representative relevance, codec, battery, language, or hardware benchmarks.

Password-protected PDF evaluation#

This evaluation uses a prepared search package and the unsigned developer helper under an OS deny-network profile. To run it:

swift build --product LumenSearchWorker
python3 Evaluation/evaluate-protected-search.py --package build/local-ai/bundle/Tools/Search --runtime .build/debug/LumenSearchWorker

The evaluation generates a labeled two-page encrypted PDF with the public test password lumen-demo-only. It checks the following:

• Separate discovery
• Wrong and correct passwords
• Actual local model vectors
• Page-specific retrieval
• Persisted indexed status
• Revoked, excluded, and stale sources
• Removal
• Unchanged source bytes
• Password absence from index and WAL files

Results are in build/protected-evaluation/report.json. The app’s sandbox-inheriting signed helper must run under its parent, not from this CLI harness.

Engine tests separately cover interrupted multi-step PDF indexing without losing completed pages, and detection of a synthetic DOCX encryption-container signature. Office decryption isn't implemented.

Local speaker evaluation#

To prepare the fixtures and run the speaker test with networking denied:

build/local-ai/bundle/Python/bin/python3.11 -B Evaluation/prepare-speaker-fixtures.py
swift test --filter SpeakerIntegrationTests
LUMEN_TEST_MODELS_DIR="$PWD/build/local-ai/bundle" \
  /usr/bin/sandbox-exec -p '(version 1) (allow default) (deny network*)' \
  xcrun xctest -XCTest LumenCaptureTests.SpeakerIntegrationTests/testRealLocalSpeakerPipelineOnLabeledAudio \
  .build/debug/LumenCaptureTests.xctest

The fixture script prepares two labeled synthetic voices without recording room audio. In the test, actual verified adapters do the following:

• Transcribe bounded overlapping chunks with word timings.
• Separate and match voices automatically.
• Retain every reconciled word.
• Leave uncertain speech unassigned.
• Produce cited notes.
• Preserve the source file.

The test also checks silence, noise, and name and export behavior. Results are saved to build/speaker-evaluation/report.json. Synthetic session output has no reusable voice features. Core tests separately cover overlapping speech, legacy transcripts, malformed model results, and keeping tracks separate.

Note: This evaluation doesn't certify real meetings, accents, overlap handling in the wild, or routing through headphones or speakers.