how we fixed voice memory so rewrites learn from you
you spent weeks teaching hold your voice which flags to keep and which to kill. the rewrite call still opened the same static voice block. we fixed the read path, not the logging.
the product had a credibility problem that did not show up in demos.
on paper, hold your voice already learned from you. every accept, reject, and dismiss logged a suggestion event. drift notes merged into your profile. lessons accumulated from the cli and mcp. from the outside, the loop looked closed. inside a rewrite call, it was not.
what voice memory was supposed to do
the promise is simple: the more you edit with hold your voice, the less it should argue with you about the wrong things.
if you accept em-dash fixes most of the time but dismiss "rule of one" flags on short paragraphs, the rewriter should know that as live context in the next prompt, weighted toward the rule you are fixing and trimmed to fit the budget. voice memory is a running record of your decisions, turned into instructions the model can act on, not a personality quiz you fill out once.
what it was actually doing before
we collected the signal. events, lessons, drift notes, accepted and rejected pairs, all real, all in sqlite. rewrites got a fixed block. keywords, anchors, a few lessons, maybe a rule rollup. same payload whether you were fixing an em-dash or a headline. no query conditioning.
half the learning never reached the prompt. drift notes and cli lessons were inconsistent with web rewrites. team members editing on a shared profile did not move stats for the owner because aggregation keyed on user_id first, not profile_id.
early compose v2 duplicated context. we prepended a composed block and still appended the full legacy voice context, which added more tokens, not fewer. worse: a javascript ordering bug referenced highlighted text before it was defined. the error was swallowed. compose could be on for your account and silently never run.
we were measuring voice memory. we were not always using it.
phase 0: wire what we already had
first fix was plumbing. normalize lessons across cli, mcp, and web. pipe drift notes into analyze and rewrite prompts. persist per-rule accept rates on learned_patterns. still a static dump, but at least the dump included decisions you had already made.
the composer: memory that answers the question in front of you
static context does not scale. a heavy user accumulates hundreds of events across dozens of rules. you cannot paste all of that into every rewrite and expect correct prioritization.
compose_voice_memory() takes a role, optional rule_id, an optional draft excerpt, and a token budget. it returns one ranked block. each rule gets a utility score from accept rate, recency, and engagement. the rule you are fixing now gets emphasized. factual profile context stays; experiential sections compete for space and drop by utility when the budget is tight.
when compose succeeds, it replaces the legacy voice context instead of stacking on top. one source per call. editor, api, and mcp all use the same path.
overnight consolidation and optional retrieval
raw percentages are weak guidance. a nightly job inside the production container (not github actions against prod sqlite) distills strategy summaries per rule into memory_strategies, capped at fifty llm calls per run. the composer prefers those summaries over bare accept rates.
optional embedding retrieval, behind flags, surfaces similar passages you already accepted or rejected when the current excerpt is close. cold-start profiles need a few stored examples first; we lowered the minimum from ten embeddings to three after review.
what we fixed before calling it done
code review caught blockers after phases zero through five landed: the editor tdz bug, sqlite row access for embed flags, team profile aggregation by profile_id, cron enqueue for consolidation, compose rule_id validation, and server-side excerpt caps. none of that is marketing copy. it is what keeps the feature working on tuesday.
what is live now
compose v2 and embed retrieval are off by default, same rollout pattern as voice profile v2. when your account is in the cohort, the editor calls GET /api/data/voice-memory/compose before rewrite suggestions. the admin dashboard tracks compose adoption and last consolidation time while we roll out.
why this matters
most ai writing tools treat personalization as setup: upload samples, pick adjectives, done. real voice is correction over time, thousands of small accept and dismiss decisions. if your tool logs them but does not feed them into the next rewrite, you do the training labor and the product does not learn. you feel it as the same flags over and over, or rewrites that ignore patterns you thought you had already taught it.
voice memory composer is our step from "we store your history" to "we use your history": query-conditioned, budget-aware, consolidated overnight. not latent kv cache yet. but if you have been using hold your voice for a while, your events are already in the database. the work was making sure the next rewrite can finally read them.
get started for $1. create your account and scan your first draft in minutes.
get started for $1 and create your account →
writes about brand voice, ai writing patterns, and the craft of sounding like yourself. built hold your voice after watching his own voice flatten across six months of heavy ai drafts.
co-written with ai as sidekick. shashank drafted the observations; the ai pressure-tested the structural claims. if something reads too smooth, that's the ai's fault.