Which embedding should an AI agent's memory use?
covey's built-in embedding measures word overlap. What a self-hosted model, a third-party service and a later switch each mean for agent memory.
Two weeks ago an agent wrote a wiki page on what to do when the pipeline is red. Today it gets a ticket saying "The CI build is failing", which means the same thing and shares only "the" and "is" with the sentence on the page. Whether the agent finds the old page in its memory again, or works out the fix a second time and files it on a second page, depends in large part on the embedding.
Anyone running agents on their own hardware hits a version of this question. On Hacker News in March 2026, a user posting as asim described the problem that their project's embeddings depended on the OpenAI API. Their attempt with nomic-embed-text under Ollama was, in their words, quite slow on the cheap VM without a GPU they were running. Covey offers four values for this choice, set through COVEY_EMBEDDING_PROVIDER in the installation's .env: builtin, ollama, voyage and openai.
The built-in embedding measures word overlap
The default, builtin, needs no key and no extra container. The HashEmbedder spreads words and word pairs over a 256-dimension vector by feature hashing, deterministically and offline. What it measures is which words two texts share: it lowercases the text, splits it at anything that is not a letter or a digit, and counts filler words like any other word. For the pair in this example the only overlap is "the" and "is", so the similarity is low and above zero only because of those two words.
Three parts of the memory depend on that at once. When an agent picks up a task, the search hands it the five closest pages of the active model with no minimum score, so the old page is among them as long as fewer than five others score at least as high. Next to those hits the agent sees its wiki's index, which is independent of the embedding. Assigning a new insight to an existing page only happens from a similarity of 0.80, and the consolidation pass only merges pages from 0.93. So covey writes a warning to the log at startup while the built-in embedding is active, and the docs say plainly that for real operation it is not enough.
Self-hosted, the wiki content stays in the installation
The docs recommend the self-hosted route because the wiki contents stay in the house. It takes two lines in .env, COMPOSE_PROFILES=embeddings and COVEY_EMBEDDING_PROVIDER=ollama, and after docker compose up -d two more containers from the ollama/ollama image are running. The embeddings service answers requests. A one-off job, embeddings-init, pulls the model, because Ollama does not fetch a missing model by itself on an embedding call.
The default model is EmbeddingGemma: 308M parameters, multilingual, runs on the CPU. COVEY_EMBEDDING_MODEL selects another one, as long as it delivers at least the 256 dimensions the database schema expects. If it returns more, covey keeps the first 256 values and re-normalises the vector. For Matryoshka-trained models that is the intended way to shorten them, and covey logs it once per process. If it returns fewer, the call fails with an error, since the code treats a vector that is too short as beyond repair by padding.
The docs are silent on how fast EmbeddingGemma runs on a given machine. asim's question stays a measurement each installation has to make for itself.
A third-party service receives the title and text of every page
The alternative is an outside embedding service: voyage with the default model voyage-3.5-lite, or openai with text-embedding-3-small. Both require COVEY_EMBEDDING_API_KEY, and if the key is missing the control plane aborts its startup with an error. COVEY_EMBEDDING_URL replaces the endpoint, for a proxy, an Azure deployment or a compatible service.
The price of this route is what goes into the requests. When a page is written, its title and text are sent to the service, cut to 8,000 characters. Every search sends the query itself. The memory chapter of the docs states it directly: the wiki leaves the installation only if someone deliberately configures an external embedding service.
Switching re-embeds the existing pages
Vectors from two different models cannot be compared, so every page carries its model's fingerprint in the embed_model column, for example ollama:embeddinggemma:256. Search, assignment and the consolidation pass only consider pages of the active model. After a switch, the control plane re-embeds the existing pages in the background at the next start, in batches of 50. Until that has run, search does not find the affected pages, though they are still there.
If the self-hosted service is still loading its model on first start, covey retries every minute, up to 30 times. A silent fallback to the built-in embedding is ruled out on purpose, so a broken configuration stops the startup. The spec gives the reason: a mixed index would be worse than a uniformly weak one.
For the agent with the red pipeline, this means the choice is made once for the whole installation and is the same for every agent. Switching later costs a re-embedding of everything, and with an outside service that includes every page that had never left the house until then.