Inside Modern AIFront matter

How to Read Vendor Claims

Frontier AI attracts confident explanations. A diagram appears online showing an exact hidden architecture. A thread states a parameter count. Someone explains precisely how a model's private memory system works. Much of that information has never been publicly disclosed, and a technically serious book has to know when not to pretend. This section sets the rules the rest of the book follows, so that when a later chapter says "Anthropic does not disclose this" or "OpenAI has not documented that mechanism," you read it as a boundary of evidence rather than a gap in research.

Four evidence levels

Every model-specific claim in this book fits one of four categories.

Documented. The vendor or another authoritative primary source directly states the fact. Example: OpenAI documents Astra's 1,050,000-token context window.

Observed. A behavior can be reproduced or measured from the public system, even if the internal reason is unknown. This means a reproducible observation, either our own or a published one, not an online anecdote. Example: a model repeatedly succeeds at a particular tool workflow under controlled testing.

Reasonable inference. Available evidence supports a likely explanation, but the mechanism is not directly confirmed. Inference is labeled as inference, never rewritten as fact.

Unknown. The information is not publicly available, or the evidence is too weak. Unknown is a valid technical answer, and some of the most important questions about these systems currently sit here.

The five-layer lens

The word "model" is used loosely in most discussion of AI. In this book it is used precisely, because the thing you interact with is a system of five layers, and a claim about one layer is often mistaken for a claim about another.

1. Underlying model capability — the capability provided by the trained model itself, before surrounding product layers are considered.

2. Post-training behavior — how further training has shaped what it tends to do.

3. Tools and harness — the software around it that retrieves, executes, checks and stores.

4. Safety and policy controls — the rules and monitors that limit or route what it may do.

5. Product experience — the interface, defaults and pricing through which a person meets all of the above.

Figure 1. The underlying model is only one layer of the complete AI system.
Figure 1. The underlying model is only one layer of the complete AI system.

When a benchmark improves, a feature ships or a request is refused, the first question is which layer changed. Chapter 5 develops this idea in full; every model-specific chapter after it uses the lens.

Why the obvious numbers mislead

People ask how many parameters a frontier model has because it feels like a simple measure of size. Even when the number is available, it explains little on its own. Practical capability also depends on architecture, training data, training and inference compute, post-training, tool access, context management, serving optimizations and the safety and routing layers. A single number can create the appearance of understanding without explaining behavior.

Benchmarks have a similar limit. A result tells you how a system performed under one defined setup. It does not by itself tell you why it succeeded, whether the result carries to a different environment, whether a tool or harness did much of the work, or how the system will behave after the next product update. This is why the technical references in this book carry a date and a source.

Behavior does not uniquely reveal mechanism either. If a model recalls a detail after hours of work, that could be because the detail stayed in active context, was retrieved again, was kept in a state store, survived a compacted summary, or was rebuilt by the product from earlier records. Seeing the right answer does not tell you which of those happened unless the system exposes it.

Diagrams carry labels too

This book still draws diagrams. Each technical diagram is labeled according to the evidence behind it: documented architecture, directly supported by vendor sources; system-level conceptual architecture, which explains the observable components without claiming hidden neural detail; or reasonable inference, clearly marked. Most diagrams here are the second kind. The rule is to explain the system the evidence supports and not to decorate uncertainty until it looks like architecture.

Why vendors do not disclose everything

There are legitimate reasons for limited disclosure: competitive advantage, security risk, misuse prevention, systems that change too fast to document, and complexity that does not reduce to a public diagram. At the same time, limited disclosure makes independent evaluation harder. Both are true, and the book does not pretend otherwise.

Seven questions for any frontier-model claim

1. Who is making the claim?

2. Is it about the model, the product or the complete system?

3. Is the mechanism directly documented or merely inferred?

4. Can the behavior be reproduced?

5. What tool, harness and safeguard conditions were present?

6. What has changed since the claim was measured?

7. What would falsify the explanation?

Those seven questions are worth more than any memorized architecture diagram. They are also the questions this book asks of its own case studies.

Documented behavior is not the same as documented mechanism. Where the evidence ends, this book says so.

Report a correction

Corrections go to the editor and are never published automatically. No account needed.