The Risks of Over-Documenting AI Prompts & Knowledge
By Izzy Aly, Co-Founder, Systems and Architecture
What your company deliberately doesn't write down may matter more than what it does. Three tests and five categories for what should never touch a system an AI can read.
By Scribe's estimate, 71% of company know-how was never documented. It lives in conversations, in intuition, in the stuff nobody bothered to write down (Scribe, 2025). Most leaders look at that 71% and see a problem to fix. They shouldn't.
The uncomfortable truth is this: in the AI era, what your company deliberately chooses not to document may matter more than what it does. Anything committed to a system accessible to an AI is no longer just internal documentation. It's training data. It's a competitor's cheat sheet. It's the blueprint someone uses to rebuild your business.
This post gives you a framework for the reverse decision most teams never make: what should never touch a formal system.
Key takeaways
- Most company know-how is already undocumented. That's not a bug; it's your last real competitive moat (Scribe, 2025).
- AI tools ingest everything that IS documented. 43% of AI users admit to sharing sensitive workplace information with AI tools without their employer's knowledge (National Cybersecurity Alliance and CybSafe, 2025).
- Three tests tell you what to keep informal: the Vendor Test, the Departure Test, and the Replication Test.
- Five categories should never be written down: founder mental models, taste judgments, strategic optionality, negotiating intuition, and informal power dynamics.
Why do we treat documentation as an unquestionable good?
Most companies treat documentation as an unqualified good. More docs = more mature. Write everything down. "If it isn't in the wiki, it didn't happen." This assumption is dangerously outdated.
The instinct comes from legitimate places. Remote work made async documentation essential. Onboarding without docs is chaos. The "bus factor" anxiety, what if the one person who knows the payment system gets hit by a bus, is real. So teams build elaborate Notion hierarchies, Confluence taxonomies, and internal wikis. The goal is to externalise all knowledge from everyone's head into a shared system.
Here's what nobody talks about: most of that knowledge never made it into the shared system in the first place, and 39% of documentation teams don't track metrics or measure the success of their docs at all (State of Docs Report, 2025).
So you're spending enormous organisational energy on a system where most content is either incomplete or unmeasured. But the real problem isn't the waste. The real problem is that documentation was never neutral, and AI just made it actively dangerous.
The assumption that documentation is inherently good misses the point. Every piece of knowledge you write down becomes a fixed artefact. It can be copied. It can be searched. It can be fed into a model. The question isn't "should we document this?" It's "what happens when this documentation leaves the building?"
What changes when AI reads everything you write
AI fundamentally changes the documentation calculus. Anything written down is now machine-readable, machine-searchable, and potentially machine-trainable at zero marginal cost. There's no friction between "document exists" and "AI has absorbed it."
Samsung learned this the hard way. In 2023, engineers pasted confidential source code and a transcript of an internal meeting recording into ChatGPT on three separate occasions (Economist Korea, 2023). Within weeks, the company restricted generative AI tools on company devices (TechCrunch, 2023). That was three years ago. Today's AI agents don't just wait for you to paste things in; they "click through screens like a human" and pull data themselves (DataCamp, 2026).
And once data is in, it's permanent. As Tom Gillis at Cisco put it, once an AI model learns sensitive data, you cannot delete it. There is no undo (Built for Trust Podcast, 2025). You can't call support and ask them to remove your pricing strategy from the weights. It's not a database. It's a model.
Video: AI Is Quietly Exposing Company Secrets by DataCamp. Jeremy Epling, CPO at Vanta, discusses how AI agents create new attack surfaces for trade secret leakage.
The scale of exposure is already massive. Truffle Security found 11,908 live API keys and passwords in the December 2024 Common Crawl archive, a dataset popular LLMs, including DeepSeek, are trained on. 63% of those secrets were repeated across multiple web pages (Truffle Security, 2025).
Meanwhile, 13% of organisations reported breaches of AI models or applications, and 97% of those breached lacked proper AI access controls. 63% of breached organisations had no AI governance policy or were still developing one. Organisations with high levels of shadow AI saw an average of $670,000 in higher breach costs (IBM/Ponemon, 2025).
This isn't a theoretical risk. Your documentation strategy now has direct financial consequences. If your internal docs land in a training corpus, the cost isn't embarrassment. It's competitive erosion you can't reverse.
The three tests for what NOT to write down
Here's the framework. Before you commit anything to a system an AI can access, run it through three gates.
The Vendor Test. If a vendor's AI agent has this knowledge, it would weaken your negotiating position; keep it informal. This covers your actual price floors, your deal patterns, your concession behaviours, and the things that make your commercial relationships profitable. The moment your vendor knows your real walk-away number, every negotiation becomes a one-sided game.
The Departure Test. If a key employee leaving with this knowledge would take six months to recover from, document just enough for business continuity, but not enough to expose the full playbook. The goal is continuity, not replication. Write the runbook, not the strategy behind it.
The Replication Test. If documenting this lets a smart competitor rebuild your capabilities, keep it hidden. Job postings alone leak strategy. When OpenAI's share of go-to-market job openings grew from 18% to 28% in a year, competitors could read the commercial pivot straight off its careers page (Epoch AI, 2026). That was just from job listings. Full documentation provides them with the blueprint, including page numbers.
Most teams default to documenting everything because the cost of NOT documenting is visible (someone asks a question, nobody knows the answer). In contrast, the cost of documentation is invisible (your strategy ends up training your competitor's next model). These three tests flip that default. Documentation becomes an active decision with a burden of proof, not an automatic reflex.
Five things that should never touch a formal system
Beyond the three tests, five categories of knowledge are so strategic that they should remain deliberately illegible.
Founder mental models. The way your founder thinks about the market isn't replicable and shouldn't be. Writing it down ossifies it. It turns a living, evolving mental framework into a fixed artefact anyone can study. Worse, it hands your strategic lens to competitors. They don't need to figure out how you see the market. You gave them the map.
Taste judgments. An explicit written definition of what "good" looks like in your product is a gift to every competitor. Once documented, taste becomes a checklist anyone can replicate. But taste is what separates your product from a competitor with the same feature set. It's the difference between "the search results are fast" and "the search results feel right." Checklists produce the first one.
Strategic optionality. Potential moves you haven't decided on yet. M&A targets you're watching. Adjacent markets you might enter. Once these are written down, they become discoverable by AI, by departing employees, and by anyone with access. 43% of AI users already admit to sharing sensitive workplace information with AI tools without their employer's knowledge (National Cybersecurity Alliance and CybSafe, 2025). Your strategy doc is one prompt away from becoming public.
Negotiating intuition. Your actual price floors. Deal patterns that worked. Concession behaviours that closed deals. Document these, and every future counterparty can reverse-engineer your approach before the first call. This isn't about hiding from regulators; it's about not arming the other side of the table.
Informal power dynamics. Who actually makes decisions in your organisation? Formal org charts lie, and everyone knows it. But documenting the real influence map creates internal political risk and gives external parties a guide to manipulating your decision-making. Some things belong in conversations, not in Notion.
Notice a pattern? Every one of these categories is something that gives you an edge precisely because it's hard to replicate. Taste isn't valuable if it's a rubric. Strategy isn't valuable if it's a document anyone can read. Power isn't influential if it's mapped. Illegibility isn't a failure mode. It's the feature.
What you SHOULD document (and how)
This isn't an argument for zero documentation. That would be chaos. Some things must be written down, and those things matter enormously. The distinction is simple: document what helps someone do their job today, not what reveals how you'll win tomorrow.
Write onboarding docs, runbooks, incident postmortems, and compliance requirements. Document processes, not judgment. Document decisions and their rationale after they're made, not your menu of strategic options before you choose. Document your API, not your architecture philosophy.
Here's what that looks like for Acme Corp, a hypothetical software company. Acme writes extensive docs on how its system works: setup, configuration, and troubleshooting. It deliberately never writes down why it chose certain architectures over others, where it thinks the market is heading, or which features it is considering next. New team members can operate the product on day one. Competitors can't read its roadmap.
The rule of thumb: if a document helps your team ship faster, write it. If it helps a competitor think like you, don't.
The competitive advantage of being illegible
In a world where AI makes all written knowledge instantly retrievable and analysable, illegibility is the new defensibility. Not secrecy: illegibility. The difference matters. Secrecy implies hiding things that exist in written form. Illegibility means they were never written down in the first place.
Look at what's happening right now. AI Overviews cite published content. Competitors train models on public docs. Recruiters reverse-engineer team structures from your documentation. Most teams try to fix this by implementing better access controls. They should be solving it by writing less.
The companies that win won't be the ones with the best wiki. They'll be the ones where critical strategic knowledge lives in conversations, in relationships, and in hard-won intuition that was never committed to any system an AI can touch. The most well-documented company in your industry isn't the one you should fear. It's the one where nobody can figure out how they actually operate.
Video: Episode 80: AI Remembers Everything: The Sovereignty Dilemma by Built for Trust Podcast. Tom Gillis of Cisco explains why AI models can never forget sensitive data once ingested.
This is the strategic paradox of 2026: the tools that promise to make your organisation smarter by capturing everything are the same tools that make you easier to copy. The most defensible knowledge isn't the knowledge you protect with permissions. It's the knowledge you protected by never writing it down.
When shouldn't you use this framework?
This framework doesn't apply everywhere. If you're in a regulated industry where documentation is a legal requirement, the compliance floor is non-negotiable. Write what the law demands. This framework covers the discretionary layer above that: strategy, not audit trails.
If you're a five-person startup, you have bigger problems than documentation strategy. Come back to this when you have something worth protecting. And if your culture already has a knowledge-hoarding problem, people hiding information out of fear or territorial behaviour, this advice will make things worse. Fix the sharing muscle first, then get selective.
The real tension: deliberate non-documentation creates key-person risk. Every piece of knowledge keeps walking out the door when someone leaves. The Departure Test is meant to mitigate that, but the tension is real, and you'll feel it. The question isn't whether there's a tradeoff. It's whether you're making it consciously or accidentally.
Frequently asked questions
Isn't this just encouraging knowledge hoarding?
No. Knowledge hoarding is accidental, fear-driven, and unstrategic, as people hide things because they feel insecure or territorial. This framework is deliberate, criteria-based, and designed to protect competitive advantage. The difference is intent. Hoarders hide everything. Smart teams hide specific things for specific reasons.
What if we get sued and need documentation for discovery?
Regulatory and legal requirements override this framework. If you're in a regulated industry, document what compliance demands. The five categories above sit above the compliance floor. Your audit trail stays. Your strategic optionality stays informal. These things don't conflict unless you've been writing down things you shouldn't have.
How do I explain this to investors who want to see "process"?
Investors don't want your strategic playbook; they want evidence that you have one. Show outputs, not internals. Share results, metrics, and customer evidence. If an investor demands your pricing decision framework as a condition of investment, that's a conversation worth having in a room, not a document worth sending.
Doesn't remote work make this impossible?
Remote work makes deliberate communication more important, not more documented. Use synchronous conversations for strategy, async docs for execution. The distinction isn't between remote and in-office; it's between strategic and operational knowledge. You can run a fully remote team and still keep your competitive intuition verbal.
What's the first thing I should pull out of our docs?
Your pricing rationale. Not your pricing page; your customers need that. The internal doc that explains why you priced things that way, what your floor is, and which deals you walked from. That doc is training your competitors' AI for negotiation. Delete it today.
Conclusion
In the AI era, strategic illegibility is a competitive advantage; what you don't write down matters as much as what you do.
Three things to do this week:
- Run your existing docs through the three tests. Open your most sensitive internal docs and ask: Does this pass the Vendor Test? The Departure Test? The Replication Test?
- Pull anything that fails the Replication Test. If a competitor could rebuild a capability from this document, it doesn't belong in a shared system.
- Make non-documentation a deliberate leadership decision. The default shouldn't be "write it down." The default should be "prove this is safe to write down."
The best-protected trade secret is the one that was never committed to paper in the first place. AI didn't change that principle. It just raised the stakes.
- Topics
- Share