Architecture

This page describes how Lumen's modules, helper processes, and local storage fit together. It's for contributors who change module boundaries, search, meeting capture, or runtime integration.

Lumen is a Swift Package Manager workspace with a native macOS app, a UI-independent core, a shared UI package, and an inherited helper process. The base build uses local Swift packages and system frameworks. The local AI bundle adds a pinned standalone Python runtime and local models.

Module boundaries#

Arrows show dependency and workflow relationships. No cloud service is part of this architecture. Before you substitute an unsigned launch or an external runtime, read The signed app boundary.

Native search package#

The parent app keeps the same bookmarks and active extensions. The separately signed Tools/Search/runtime (LumenSearchWorker) inherits those extensions and receives a bounded IntelligentSearchRequest from the UI-independent core. The worker imports the adapted DigUp search engine and statically linked llama.cpp, without the upstream downloader or app.

Indexing and source access#

• The helper reads only live chosen roots and applies exclusions. It writes the derived IntelligentSearch/index.sqlite and an optional code.sqlite.
• SQLite transactions persist segments, FTS text, and fp16 vectors together.
• For each request, an authorization and freshness predicate removes unavailable sources from keyword results and vector scaling.
• The parent checks each returned path again before preview or open.
• Removed scopes are physically pruned from the derived indexes.

Scheduling and resource use#

Index requests run in bounded batches and long document steps. Queries and meetings interrupt and restart them. A small progress file reports the source that's currently being read, including initial OCR setup. FSEvents coalesces chosen-folder changes.

Low Power Mode, optional battery pause, memory pressure, and meeting work prevent background inference. Queries load one encoder, the text GGUF only; indexing also uses the image and audio encoders. Every request releases its process and model afterward. Exact identifiers and fully quoted phrases don't load a model.

Rich source metadata carries a page, media timestamp, code line, or notebook cell into a native read-only preview or player.

Preparation and fallback#

Explicit preparation builds a pinned, patched, network-disabled llama.cpp archive and downloads checksum-pinned GGUF weights. The app and the ordinary build never download them. The base build works without that archive or package. For the preparation steps, see Native multimodal search.

The Python text/image adapter remains available as a fallback and for explicit development evaluation. Lumen never mixes incompatible vector spaces.

Fallback selected-folder search#

The fallback search path works as follows:

1. The app obtains a folder through the native picker and saves an app-scoped bookmark.
2. The app restores and holds the security scope. The helper inherits that active extension. The helper must not restore the parent's bookmark under its own identity.
3. FolderIndexer walks eligible local files within scope and requests helper extraction. Limits reject unsafe paths, cloud-only files, oversized content, and unsupported formats.
4. TextExtractor and PassageSplitter produce bounded passages with relative paths and PDF page metadata. Still images produce a validated image reference with empty text. NativeImageInspector isolates Apple ImageIO decoding.
5. SearchStore serializes SQLite operations. Atomic per-file replacement updates passages, FTS5 text, and derived vectors. A successful refresh prunes deleted files. A cancelled run preserves the prior index without final pruning.
6. Queries rank keyword passages and optionally combine them with local cosine results. The UI filters unavailable folders and opens the real source and page.

Each vector carries its model, revision, dimension, and normalization identity, plus the artifact and input-recipe digest. Incompatible vectors are invalidated without deleting keyword passages.

Note: The initial cosine scan is capped at 50,000 passages. A larger corpus needs measured retrieval work.

Capture, processing, and retention#

Capture#

AppModel orchestrates manual sessions and explicitly opted-in automatic sessions. MeetingCapture uses AVAudioEngine and optional ScreenCaptureKit audio, with the following properties:

• A bounded audio queue
• Separate tracks
• Timestamped CAF chunks of up to 25 seconds, with at most one second of overlap
• Pause boundaries

A finalized-chunk journal supports interrupted-session recovery.

Transcription and notes#

Closed chunks can be transcribed during capture. When you choose Stop recording, processing reuses cached chunk transcripts and completes the remaining chunks before it generates notes.

To remove duplicate coverage at chunk boundaries, the reconciler matches word text and overlapping timestamps on the same track. The later window keeps context past the boundary. Distinct repetitions and different tracks remain. Legacy adapters without word timing use exact overlapping-segment reconciliation. Boundary-word recognition quality remains a model evaluation concern.

Generated note source IDs are validated against actual transcript segments.

Speaker labels#

After transcription, a verified CPU speaker adapter analyzes bounded chunks. Session-local clustering links voices independently per track, and Whisper word timings allow conservative, text-preserving attribution. Ambiguous or overlapping speech stays unassigned.

Voice vectors exist only in memory for the lifetime of a request or session, and are never persisted. Speaker labels and optional user-confirmed names are saved before notes and retention. Lumen has no live diarization or cross-meeting identity recognition.

Retention#

The meeting repository writes the completed transcript, notes, and metadata before it applies retention. Original audio is eligible for removal only when the session has completed the required processing and the session's retention policy allows it. Failed, cancelled, unprepared, and interrupted input remains available for retry.

User edits are saved locally. The editable surfaces are rough notes and action items.

Optional runtime boundary#

LocalProcessAdapter verifies declared artifacts and sends one bounded JSON request to a directly managed process. It checks response identity, schema, timestamps, dimensions, finite values, and source references as appropriate. The runtime exits after each request, and cancellation terminates its managed process group.

The core exposes transcription, diarization, embedding, and generation protocols. Not every protocol has a shipped implementation or UI. The local AI bundle ships a speaker-analysis adapter; the written-answer UI isn't implemented. For the exact integration boundary, see the local tool contract.

Resource priorities and evidence#

• Meeting capture and note processing pause indexing.
• Memory-pressure events pause indexing and cancel embedding work, with bounded batches and working contexts.
• Long-meeting notes use serial batches of at most 24 KB of transcript text and merge their source-linked results.
• Each inference process exits before the next request.

Note: These design choices don't establish an 8 GB hardware guarantee or model quality. The validation record tracks correctness checks separately from the model evaluation work that's still needed.

Platform direction#

The text/image input contract in MediaInputs.swift represents modalities, query/document purpose, local paths, and vectors without AppKit or SwiftUI types. MultimodalEmbeddingService preserves the text service boundary. The reference Python adapter uses MPS/float32 when available, CPU/float32 otherwise, and explicit local files. Future platform launchers can integrate it.

The Swift core isn't fully portable yet. These components need native equivalents or portability work for Windows and Linux:

• PDFKit extraction
• ImageIO decoding and previews
• CryptoKit verification
• Darwin process control
• Scoped bookmarks
• Capture

The macOS UI remains SwiftUI. The search package uses Apple Vision for OCR on macOS, so future platforms need a local OCR adapter. Image-vector search doesn't require an OCR dependency.