Question
What happens when AI guesses your preferences?
It learns your average behaviour rather than your standard. Your average behaviour includes every compromise you made under time pressure, so it optimises you toward the version of you that was tired.
Inference runs on frequency, and frequency is not preference
To infer what you like, a system counts. What did you accept, what did you choose, what did you not change. The things that happen most often become the model of what you want.
That step is reasonable and it is where the damage enters.
What you do most often is not what you prefer. It is what you preferred, filtered through how much time you had, how tired you were, how much the thing mattered, and whether it was Friday. Half of your observable behaviour is circumstance wearing the costume of taste.
So the system does not fail to model you. It models you accurately, including the parts you would not have submitted for the record. Every shipped-good-enough, every corner cut for a reason that was fine that day, every draft you let go because the alternative was missing a deadline. All of it counts as evidence, and none of it was a preference.
Then it optimises toward that
The second step is what makes it compound. Having inferred a preference, the system produces work that matches it, and then observes that you accepted the work, which is more evidence for the same inference.
You did not accept it because it was right. You accepted it because it was close enough and you were busy, which is exactly the state that generated the bad evidence in the first place.
Left running, this pulls in one direction: toward the middle of your own history. Not toward the average of everybody, which is the failure mode of an empty system, but toward your personal average, which contains your compromises at full weight and your best work as a rounding error.
You cannot audit it by reading it
This is the property that makes it hard to manage even once you know about it.
An inferred preference is presented in the same tone as a fact you supplied. There is no marking, no confidence level, nothing that says you told me this against I worked this out. Read a profile a system holds about you and every line looks equally authoritative.
So auditing means remembering what you actually said, months back, across hundreds of sessions. Nobody can do that, which means in practice nobody audits, which means the inferred entries live in the record permanently with the same standing as the real ones.
| An entry you supplied | An entry it inferred | |
|---|---|---|
| How it reads | Confident | Confident, identically |
| Traceable to a moment | Yes | No |
| Right about the edge cases | Usually | Rarely, and that is where it matters |
| Correctable | Yes, if you notice | Only the ones you notice |
Why a wrong guess beats no guess, and why that is the trap
An empty system produces an obvious average. It reads as generic, you notice immediately, and you correct it. The error is loud, which makes it cheap.
A system running on a wrong guess produces something specific and plausible and mildly off. It looks like it understood you. The error is quiet, which makes it expensive, and it survives review in a way the loud version never would.
Which is the uncomfortable conclusion: the better the inference gets, the harder the remaining errors are to catch. Improvement in this dimension moves the failures further underground rather than removing them.
What the alternative actually requires
Not more accurate inference. A different admission standard.
An entry qualifies if it traces to a decision you actually made, at a moment that can be located, with the reason attached. One question tests it: what happened, and when. An entry that cannot answer was inferred.
Inferred material can still be useful. It just cannot sit in the same list without a mark on it, because the moment it does, nobody downstream can tell which half of the record is you.
Frequently asked
What happens when AI guesses your preferences?
It infers from what you do most often, and what you do most often includes every compromise you made under time pressure. So it does not learn your standard, it learns your average behaviour, which contains your worst days at full weight. Then it optimises toward that, and it does so confidently, because an inferred preference is stated in exactly the same tone as one you gave it.
Why is frequency the wrong signal?
Because your standard is what you do when you have time to do it properly, and that is not your most common behaviour. Frequency measures circumstance as much as preference. The version of you that ships something adequate at six on a Friday is well represented in the data and is not the version you would want represented anywhere.
Can I just correct the wrong guesses?
Only the ones you can see, and the visible ones are the minority. A guess is stated with the same confidence as a fact you supplied, so nothing marks which is which. You would have to audit by remembering what you told it, which people cannot do past the first few weeks.
Is a wrong guess worse than no information?
Usually, yes. With no information the system produces an obvious average and you know to correct it. With a wrong guess it produces something specific, plausible and mildly off, which is much harder to catch and reads as though it understood you.
What is the alternative?
Entries that trace to a decision you actually made, at a locatable moment, with the reason attached. The test is one question per entry: what happened, and when. An entry that cannot answer it was inferred, and it should be marked as such rather than sitting in the same list as the rest.
Where this sits
Why one admitted guess is not a small amount of damage is at why capture beats inference. The one-question test is at captured, not guessed. The category is defined at what Imprinted AI is.
Related
- Why capture beats inference. The mechanism, and why one guess is not a small amount of damage.
- Captured, not guessed. The one-question audit.
- Why doesn't ChatGPT remember how I think? Why nothing was captured to begin with.
- Digital DNA. Why who you are now and who you have chosen to become are stored apart.
Keep going
Regression to your own mean is the quiet version of this. The loud version is what happens to a company that automated a function before capturing what made it good.
The full case is The A.I. Business Manifesto. About 23,000 words, free to read on the page, no gate in front of it. If you would rather have the short version, the same page will send you the three fixes and the PDF.
Read The A.I. Business Manifesto
Free either way. Reading it costs nothing and asks nothing.
Last updated: 28 July 2026