Local AI Tool Integration — Lumen Developer Docs

This page defines the contract between Lumen and a local tool package: the manifest, the JSON requests and responses, and the checks Lumen applies to them. It's for developers who package a local runtime for transcription, speaker analysis, notes, or embeddings.

How Lumen runs a tool#

The contract is local only. It has no remote endpoints, automatic downloads, telemetry, or accounts.

LocalProcessAdapter verifies the package artifacts in the worker.
It invokes the matching executable as a directly managed, sandbox-inheriting child process.
It sends one bounded JSON request on stdin and reads one bounded JSON response from stdout.
The runtime exits.

Direct ownership lets cancellation terminate the runtime's process group without orphaning a nested inference process.

Imported runtimes must work inside the sandbox, with their models at explicit paths. A script or executable that requires network access or undeclared filesystem access should fail, not fall back online.

Import a tool package#

To import a package, in Settings → Local tools → Support details, choose a folder that contains lumen-tools.json.

Several packages can provide different tasks. Lumen verifies bundled packages automatically. An imported package takes precedence for the operations it supports. Onboarding doesn't show model details.

Lumen chooses an explicitly imported adapter by supported operation. Automatic selection based on measured language or hardware quality isn't implemented yet.

Manifest#

The manifest uses schema version 1:

{
  "schemaVersion": 1,
  "runtime": { "path": "bin/local-runtime", "sha256": "<actual 64-character SHA-256>" },
  "modelFiles": [{ "path": "models/model.gguf", "sha256": "<actual 64-character SHA-256>" }],
  "modelID": "<exact upstream model/conversion ID>",
  "revision": "<pinned revision>",
  "license": "<verified license and notices>",
  "provenance": "<origin and conversion details>",
  "operations": ["transcribe"],
  "languages": ["en"]
}

All paths are relative to the chosen package. Path traversal and escaping symlinks are rejected.
Files are streamed through SHA-256 verification.
An embedding package also declares embedding: { model, revision, dimensions, normalization }, matching its identity.
The fingerprint includes the revision, dimensions, normalization, input recipe, and a digest of the runtime/model inventory. Incompatible vectors are removed and regenerated without deleting keyword passages.

Requests#

Every request contains operation, modelID, revision, and absolute modelPaths, which the worker populates. Depending on the operation, a request also includes one of the following:

audio: { id, path, track, start, duration }
texts
transcript
question and evidence

Important: Treat all source text as untrusted quoted data. A runtime must not interpret document instructions as tool authorization.

Responses#

Every response contains schemaVersion: 1, the matching modelID and revision, and one of segments, summary, vectors, speakerAnalysis, answer, or error. For the exact schema, see the Codable types in LocalServices.swift and Meetings.swift.

Transcription#

Segment times are relative to the supplied chunk, finite, nonnegative, ordered, and within its duration. The adapter applies the track's absolute offset and removes any unverified speaker name. Chunks are at most 30 seconds. Lumen's capture chunks are at most 25 seconds, with a bounded overlap.

Each segment can include optional words: [{ start, end, text }], also relative to the chunk. The adapter validates and offsets them. Old adapters without words remain compatible. Their mixed-speaker passages stay unassigned rather than being split by guessed character offsets. Speaker labels are assigned before note generation, so citations refer to the final durable transcript segments.

Summaries#

A summary contains summary, decisions, actions, and questions.
Cited notes have id, text, and sourceIDs. Action items also have owner, due, and completed.
Source IDs must refer to actual transcript segments.
Empty or unsupported notes fail validation.
Names must come from evidence or explicit user edits.

Vectors#

Vectors must match the declared dimensions, contain finite numbers, and have a nonzero norm.
Batches contain at most 16 passages. The app requests eight.
Search combines exact keyword hits and local cosine matches.

Note: The initial vector scan is capped at 50,000 passages. Larger-scale indexing requires a measured ANN implementation.

Answers#

Answers require saved evidence and must return valid evidenceIDs. The core implements this boundary, but this release has no answer-generation UI.

Speaker analysis#

The bundled Speakers adapter implements the diarize operation on one bounded audio chunk. diarize accepts the same bounded audio input as transcription. It returns speaker regions and transient normalized voice vectors in speakerAnalysis:

regions: [{ start, end, localSpeaker }]
vectors: [{ localSpeaker, values, seconds }]

Times are relative to the supplied chunk, finite, and bounded. The bundled CAM++ vector has 192 finite values. Each chunk is bounded to 512 regions and 32 vectors. Clean speech under 1.5 seconds isn't used to identify a voice.

The core validates and matches regions and vectors within one meeting, keeping tracks separate. It converts times to absolute track positions, conservatively matches clean voice vectors across chunks, and uses timestamped transcript words to assign labels. Voice vectors are transient and never added to the meeting search index.

Names require confirmation. Attendee-list inference and a reusable voice-profile store aren't implemented.

Text and image embedding inputs#

Legacy text tools continue to receive texts. A multimodal package also declares these manifest fields:

{
  "embeddingInputVersion": 2,
  "embeddingModalities": ["text", "image"],
  "modelRoot": "models"
}

These fields extend the manifest; they aren't a complete manifest.

Images require input version 2 and text support, so the same model can embed a search query.
modelRoot is optional for generic runtimes. When supplied, it's relative to the package, and every file under it must be inventoried. Escaping symlinks and uninventoried tokenizer or config files aren't allowed.
The verified full model root and declared dimensions are passed to the runtime.

Version 2 embedding requests contain embeddingInputVersion: 2, purpose: "query" or "document", and inputs. For example:

{
  "operation": "embed",
  "modelID": "google/embeddinggemma-2",
  "revision": "<actual pinned revision>",
  "embeddingInputVersion": 2,
  "purpose": "document",
  "inputs": [{ "modality": "image", "path": "/absolute/chosen-folder/photo.png" }],
  "modelRoot": "/absolute/verified-package/models",
  "modelPaths": ["<every verified artifact path>"],
  "dimensions": 768
}

Text input is { "modality": "text", "text": "..." }.
Media paths must be local files. URL inputs are rejected.
The app checks chosen-folder scope and modification metadata before embedding and before saving.
Images are processed one per batch. Each image contributes one vector, not fabricated searchable text.
A late result can't attach to a replaced file's passage row.
Search queries and image or document embeddings must use the same artifact fingerprint and dimensionality. Model, runtime, or recipe changes rebuild vectors and preserve keyword text.

The native search runtime implements this contract with EmbeddingGemma 2 (GGUF) and already-prepared local files. OCR is a separate future extraction adapter, and visual embeddings don't require it.

Verification limits#

Hash verification proves artifact identity, not runtime quality or its internal intent. Sandboxed processes have no network entitlement.

Before you claim supported languages or performance, verify a prepared package under a network-denying sandbox and on the target hardware. The bundled-model integration has passed a small real-inference synthetic evaluation with networking denied; see Bundled local models. That evaluation doesn't establish broad language or hardware quality.

Local tool contract