How I build with AI: a 9-step loop, from first question to weekly review
Yu Han Tey · 18 October 2026 · 6 min read
I build memory for an AI assistant: what it remembers about each user, and how that memory gets updated. I build it mostly with AI.
AI can read twenty papers in the time it takes me to read one. So speed is no longer the bottleneck. My understanding is.
Short answer
I write what the system should do. AI reads, drafts and tests. I keep the decisions.
To make decisions fast, I make AI explain things the way a good textbook does: example first, plain words, a diagram instead of a paragraph.
Before and after the loop
Before: vibe coding
After: the loop
Who decides
AI wrote it, I trusted it
AI drafts, I decide, with a number
Starting point
A vague ask
Seven questions and expected behaviour, written first
Bugs
103 in five weeks; 70 were repeats
Each old bug is a test; every change runs all of them
Fixing
One problem took 11 pull requests
Second fix on the same thing → find the root cause first
Testing
Each fix passed its own tests, and broke a neighbour
Neighbour cases, held-out cases, real browser runs
Reading research
Long AI summaries I couldn't follow
One plain page with diagrams; a rule for when to stop. I understand a new topic in one sitting instead of several.
01AI often explains things in jargon
Ask AI to summarise ten papers on agent memory, and you get sentences like the one on the left. Every word is correct. I still can't tell what happens.
What AI writes
The consolidation stage merges admitted, source-linked clusters into atomic statements, with lifecycle-scoped deduplication.
What I need
Tonight, three new facts passed review: "avoids nuts", "doesn't eat nuts", "is planning a trip to Tokyo".
The first two say the same thing, so they become one memory. The third stays on its own.
This step is called consolidation.
One day I asked my AI assistant, in frustration: you were trained on more books than any person could read, so why can't you explain things as clearly as a good book does?
I'm not the only one. One developer had to ask three times before their assistant spelled out an abbreviation. And one write-up points out that a fixed bug can end up harder to understand than the bug itself, because the final answer retells the whole investigation instead of the result.
This isn't one tool's fault. Most models write like this by default, and it's easy to fix once you tell them how you want to read.
The problem isn't too much information. It's steps that are too big.
AI jumps from a fact to its conclusion. A person needs the steps in between. Often the idea itself wasn't hard. I was stuck on understanding what the AI had just told me.
02My rules for anything AI writes for me
These live in the instructions my coding assistant reads at the start of every session.
Example first, name secondFirst: "a student memorises last year's exam, scores 100 on it, and fails a new one." Then: "this is called overfitting."
No abbreviationsWrite the full name, and explain it in one plain sentence right there.
One idea per sentenceIf a sentence needs to be read twice, split it.
Sources at the endNo "(Author, 2024)" in the middle of a sentence.
A diagram beats a paragraphAnything with steps, order or structure gets a picture. In a chat, even a quick ASCII diagram:
chat → pull out facts → review → merge → memory
Show the originalWhen AI tells me what a paper or file says, it quotes the actual line first. Without this, AI sometimes explains its own reading of the text, not what the text says.
Three reasons it could be wrongEvery recommendation ends with three real ways it might fail, because the first obvious answer is not always the best one.
Hard terms still get used. If I'll meet a word like "held-out set" elsewhere, I want to learn it. I just want the example first.
03My loop
1
Seven questions AI asks · I answer
Most people ask AI questions. Here I make AI ask me. On every new project, my assistant won't start until I answer seven questions, and it pushes back on vague answers:
Who has the problem?
How painful is it?
How do they handle it today?
Which number should get better?
What is the smallest version that is still useful?
How will we test it?
What has to be true before it goes live?
It's practice. I want to think this way without being asked.
2
Expected behaviour Me
Before any research, I write what the system should do. For memory, my goal is one line: remember what is worth remembering about the user, and nothing else.
Memory gets written in two ways:
Explicit
The user asks directly: "Remember that I'm vegetarian." It is saved right away.
Nightly
The assistant notices on its own. The user asked for nut-free recipes in three different chats, so overnight it saves "avoids nuts". This is sometimes called dreaming.
Then I turn the goal into small examples, with the answer I expect:
User says
Expected
"I'm vegetarian."
Remember it.
"Make this one spicy."
Don't remember. It's for this recipe only.
"Hi."
Nothing to remember.
"I eat fish now." (memory says "is vegetarian")
Change the memory to "eats fish, but no other meat". Don't just delete it, or the assistant may start suggesting steak.
Each example later becomes a test case.
3
Research AI
AI finds papers and engineering blogs, and turns them into one HTML page: each approach as a diagram, its trade-offs in plain words. I read the page, not the twenty papers.
4
Decide Me
AI lists the decisions that need a human. I make them, and write down what the system must never do.
5
Hypothesis Me
Not one guess, but several competing ones: which option should win, and why. All written down before any test runs, so the test can prove me wrong.
6
Eval AI drafts · I judge
I don't write every test case myself. AI writes them from my examples in step 2, with the answer it expects, and marks the ones it isn't sure about. I answer those first, then check the rest.
Last time, AI wrote 54 cases and flagged 13. I answered the 13 and checked the other 41. Answers a person has checked are called gold; answers only AI has written are silver.
Then AI compares the options. Say there are three ways to merge memories. All three take the same test, and we compare scores.
Then it takes parts away. If an option has two parts, run it again without one of them. If the score doesn't drop, that part wasn't helping.
One rule: the answers are set before the test, like a teacher's answer key. Nobody changes them after seeing the results, even when the model gets one wrong.
7
Build and test AI
AI writes the code, runs end-to-end tests from my expected behaviour in step 2, and tries many odd inputs fast.
8
Ship Me
The call is mine, but numbers make it. I ship when three things hold:
Eval
The score on the held-out cases doesn't drop, and there are no harmful mistakes, like deleting or inventing a memory.
End-to-end
Every example from step 2 behaves as expected in the test environment. AI runs them in a real browser and saves a screenshot or a short video of each, so I can check with my own eyes, not just read "passed".
After launch
One number to watch. For memory: how often users delete a memory the assistant saved.
9
Keep checking, keep exploring AI
AI changes every month. New models, new papers, new tricks. A design that was best in spring may not be best in autumn.
So a few jobs run on their own:
Check
Re-run the tests and evals. I look at what broke.
Explore
Collect new memory papers and engineering blogs, pick the most promising, try it against our evals, and report whether it beats what we have. I decide whether to adopt it.
Reflect
AI summarises everything I merged that week into my notes app: what broke, why, how it was fixed. I read it and look for fixes that keep coming back. Each lesson goes to one of two places: my coding assistant's memory and skills, so it doesn't repeat the mistake, or my own checklist.
Prune
Once a month, AI reviews my assistant's own rules and memories. It merges duplicates and removes outdated ones, after taking a backup, and asks me about the rest. One question for every new rule: did it come from a one-off mistake, or one that keeps happening? One-offs don't earn a rule. Too many rules confuse the assistant.
Held-out set: test cases I don't look at while improving the prompt. If a fix only works on the cases I looked at, the held-out set catches it.
Removing one part at a time: if a design has three parts, run it without each one. If the score doesn't drop, that part isn't helping. This is called an ablation.
04The ideas behind the loop
Never ship the first answerThe first answer is just the first one. Step 6 compares several options, and removes one part at a time to see what actually helps. That second part is called an ablation. AI makes this cheap: it can write three versions in the time I used to write one.
Don't defend an idea, test itI'm not attached to my first idea. I start with several competing guesses, look for evidence against my favourite, and drop it when the test says so. Every AI recommendation ends with three reasons it could fail, for the same reason.
Tests should look like real useA test that is easier than real life will pass and prove nothing. That's why end-to-end checks run in a real browser, with screenshots I look at myself.
Check the question before the answerA good answer to the wrong question is wasted work. That's what the seven questions in step 1 are for.
A number before any changeOtherwise "it's better now" is a feeling.
Explain it backBefore I accept a design, I explain it in my own words. If I can't, I ask for an example until I can.
A rule for when to stop researchingAI can always find one more paper, so research never stops by itself. My rule: stop when the next paper can't change my choice. Say I'm choosing between design A and design B, and A is ahead. If nothing I could still read would make me pick B, I stop and build A. Often the fastest way out is to stop reading and test: write down a hypothesis, build both versions with AI, and run an ablation. The result decides, not the next paper.
05What went wrong before the loop
These happened before I worked this way. They are a big part of why the loop looks the way it does.
Every fix passed its tests. Together, one broke another.
AI makes fixes fast. In one week I merged dozens of fixes to the memory system, 15 of them on a single day. Each one passed its own tests.
One fix taught the system a new rule: if a new preference contradicts an old one, remove the old one. Its tests were all about contradictions, and they passed.
The next day, a user changed one memory from "suggest chicken recipes" to "suggest fish recipes". A different memory, "cooks fish every Friday", disappeared.
Before After the new rule
"suggest chicken recipes" → "suggest fish recipes"
"cooks fish every Friday" → (gone)
Why: both mention "fish", so the rule read it as a replacement.
The new rule had compared the whole list. It saw two memories about fish and decided one replaced the other. They didn't contradict at all.
The fix was right for the problem it was built for. Nobody tested it against its neighbours.
Related changes have to be tested together. Fast merging hides this, because every single change looks green.
I let AI do the reading, and almost the thinking
AI found how other memory systems solve a problem, and the answer looked clear. But when I closed the page, could I explain why they do it that way?
Not always.
That is the line between using AI and handing over the decision.
06Why the same bugs kept coming back when I vibe coded
Back then I was vibe coding: AI wrote the code, and I trusted it fully without checking much myself. The weekly notes showed what that cost. In five weeks I fixed 103 bugs. 70 of them were repeats: the same root cause, back in a new form. Every repeat meant another pull request.
Other developers describe the same loop, and it shows up across many popular coding tools. In one team, a bug had been "fixed" three times, each time with green tests, while the real cause stayed untouched. The agent was right that the symptom would go away. It was wrong about why.
So I went back and sorted every bug by its root cause. The repeats came from five habits:
I patched the symptom, not the causeOne problem took 11 pull requests before I found the real cause: a single setting.
A fix was tested only on its own bugSo it broke something next to it, and that became the next bug.
I fixed one path and missed the othersThe same data could come in through a second door I hadn't checked.
Tests didn't take the path real use takesSo a fix passed its test while the bug was still there.
Failures were silentProblems showed up late and one at a time, so each one became its own fix.
Only about 1 in 7 repeats was a badly worded prompt.
Most came from not thinking a case through. So I changed where I spend my time: less on rewording prompts, more on listing what could go wrong before AI writes any code.
07How I stop the repeats
I patched the symptom
→
Find the cause firstIf the same thing needs a second fix, I stop asking AI for another patch. It finds the root cause and shows me the evidence.
Tested a fix only on its own bug
→
Test the neighboursA rule that removes things also gets cases that look similar but must stay.
Fixed one path, missed the others
→
List every path firstEvery place the data can get in, not just the one that broke.
Tests didn't follow real use
→
Test the way it really runsSame path as real use. End-to-end in a real browser, with screenshots.
Failures were silent
→
Make every failure loudMarked as an error, with the reason. All errors at once.
And one rule over all of them: every old bug becomes a test, and every change runs the whole set.
08What's next
The "explore" job in step 9 is next. First, a one-time map of where our design is behind recent research. Then the weekly job: up to three new ideas, each tried against our evals.
The goal is the same as everything above. AI does more of the work, and I still understand every decision.
09Where I could be wrong
"Prompt" or "thinking" is my callSome fixes changed what the model is given, not how the prompt is worded. Count those as prompt problems, and prompts become about 1 in 3 repeats. Still not the biggest group.
The loop is heavyNine steps make sense for something real users depend on. For a quick script or a throwaway prototype, plain vibe coding is probably fine.
10FAQ
How do you stop AI from writing in jargon?
Give it fixed writing rules: give an example before the name of a thing, spell out every abbreviation, one idea per sentence, and sources at the end. Ask for a diagram whenever there is a structure or a sequence, even a quick ASCII diagram.
What should an AI engineer let AI do, and what should they keep?
Let AI do the reading, the first drafts of code, and the repetitive testing. Keep the decisions and the product taste: which trade-off to accept, what the system must never do, what will feel right to the user, and whether a result is good enough to ship.
Why do systems built with AI keep getting the same bugs?
In my case, most repeat bugs were not badly worded prompts. Only about 1 in 7 was. The rest came from patching symptoms instead of causes, testing a fix only on its own bug, missing a second path for the same data, tests that don't follow real use, and failures that stay silent.
How do you know when to stop researching and start building?
Stop when the next paper can't change your choice. If you're still unsure between two designs, stop reading and test: write a hypothesis, build both versions with AI, and let an ablation decide.