Cortex Embedding Providers

Cortex stores embeddings for three LanceDB tables — page_history (chunks), bookmarks, and scripts — and uses them for recall_memory, list_scripts (when a query is supplied), and bookmark relevance ranking.

Embedding is provided through a small pluggable interface so a real semantic model can be adopted without rewriting the storage layer.

The interface

export interface EmbeddingProvider {
  readonly id: string;    // stable, persisted alongside the vectors, e.g. "hash-128"
  readonly dim: number;   // vector dimension this provider produces
  embed(text: string): Promise<number[]>;
}

Selection helpers live in src/main/cortex/embedder.ts:

registerEmbeddingProvider(provider) — make a provider selectable by id.
setEmbeddingProvider(id) / getEmbeddingProvider() — choose the active one.
listEmbeddingProviders() — ids currently registered.
embedText(text) — embed with the active provider (what Cortex uses).

Default: `hash-128`

The default provider is a deterministic 128-dimensional bag-of-words/trigram hash embedder with no dependencies and no model download. It captures lexical similarity (shared words/subwords), not learned semantics. It is intentionally cheap so Cortex works offline and out of the box.

generateEmbedding() remains as a synchronous helper bound to the hash provider, but Cortex itself always calls the async embedText() so an alternative provider takes effect everywhere (history, bookmarks, scripts, and the query side alike).

Selecting a provider

Set KUANG_EMBEDDING_PROVIDER to a registered provider id:

# PowerShell
$env:KUANG_EMBEDDING_PROVIDER = "embeddinggemma2-768"

Unknown ids are ignored with a warning; the app never fails to start.

The model-identity guard

Changing the provider changes the vector space (dimension and semantics). Vectors from different models are not comparable, so Cortex records the active provider in <storageDir>/embedding-provider.json:

{ "id": "hash-128", "dim": 128, "updatedAt": "2026-10-19T00:00:00.000Z" }

When the active provider differs from the recorded one, CortexDB logs a warning on startup. It does not silently mix spaces. Re-embedding existing rows is a follow-up migration step (see below).

Adding a real (ONNX) provider

EmbeddingGemma 2 (Gemma 4 decoder, 740M params) and the earlier embeddinggemma-300m both emit 768-dimensional vectors, and ONNX builds exist (e.g. onnx-community/embeddinggemma-2-ONNX). Sketch:

import { registerEmbeddingProvider, type EmbeddingProvider } from './embedder.js';

// Lazily imported so the dependency/model is only paid for when selected.
class GemmaOnnxProvider implements EmbeddingProvider {
  readonly id = 'embeddinggemma2-768';
  readonly dim = 768;
  private session: any;

  async embed(text: string): Promise<number[]> {
    if (!this.session) {
      const ort = await import('onnxruntime-node');
      const modelPath = process.env.KUANG_EMBEDDING_MODEL; // path to model.onnx
      this.session = await ort.InferenceSession.create(modelPath!);
    }
    // Tokenize -> run session -> mean-pool -> L2-normalize -> number[768]
    // (tokenizer + pooling depend on the exported model; use the model card.)
    throw new Error('wire tokenizer + session.run here');
  }
}

registerEmbeddingProvider(new GemmaOnnxProvider());

Adopting this means, at minimum:

Add onnxruntime-node (native dependency) and ship/point at a ~200–300 MB model artifact.
Accept per-call inference latency (CPU) where the hash provider is ~free.
Migrate the three Cortex tables from 128-d to the new dimension: drop/createTable them and re-embed the source text (page_history.content, bookmark title/notes/tags, script name/description/code). LanceDB infers the vector column from the rows, so a dimension change cannot be applied in place.