Architecture tests read structure. Architecture lives in intent

Thu, 17 Sep 2026 00:00:00 +0000

Two endpoints on one controller. They call the same service method, reach the same repository, and finish at the same write to the same table. Both contain a validation call. One does what the team decided its architecture should do, and the other does not.

Run every architecture test you have against that pair and they come back identical, because structurally they are identical: same components, same package boundaries, same dependency directions, same operations against the same infrastructure. The difference is a legacy branch behind a flag nobody removed, where the write happens without the request ever passing through validation.

The decision was requests are validated before they change state. What the team can assert is that a class in one package does not reference a class in another. Passing that assertion is not evidence for that decision.

I have argued before that an agent’s confidence has to come from evidence outside it — tests, compilation, static analysis, contracts — and then that those checks do not automatically express everything a team values about architecture, operability or maintainability. This picks up that unfinished second thread at the architecture layer, where it matters most when an agent is implementing. The agent sees green tests, a green dependency rule and a plausible review, and concludes the work is done. Nothing in that loop could notice the branch. The checks described the implementation accurately. They just had no way to describe the architecture.

The tools are not the problem

Architecture tests understand far more than dependency direction, and an argument built on the opposite claim is a strawman. ArchUnit exposes packages, classes, dependencies, inheritance, annotations, field access and method calls, and a custom rule is ordinary Java, so you can assert anything you can compute.¹ jQAssistant scans a project into a graph and evaluates Cypher constraints; its own introductory example finds test methods containing no assertion, not a package edge.² Semgrep’s taint mode expresses sources, propagators, sanitizers and sinks.³

CodeQL will do whole-program data flow, and is candid about the price: global flow costs materially more time and memory, is less precise than local flow, and wants a purpose-built model of sources, sinks, barriers and framework edges first.⁴ It also, to concede the point up front, goes furthest towards what I am about to argue for: data extension files pull framework knowledge out of the query into declarative, shareable tuples.⁵ The seam already exists; its vocabulary is the limit. Those relations describe data moving towards danger, because that is the question they were built for. None of them can say this is a state change, this is an event publication, or this happens inside a transaction.

Which points at what they share. Every one is a rules engine sitting on a model of your code: bytecode for ArchUnit, a graph for jQAssistant, value flow for CodeQL, an abstract syntax tree for the linters most engineers have actually touched.⁶ The models differ; the ceiling does not. Each records what the code is, and none of them records what the code means. To a syntax tree, new PutItemCommand({...}) is a NewExpression whose callee is an Identifier. Perfectly accurate. Architecturally useless. Nothing in that description says a row is about to be written, and no rule, however cleverly authored, recovers what the model never carried.

That ceiling explains why dependency rules work and everything else is a fight. Dependency direction genuinely is structural, right there in the imports, so the model holds it and a rule can ask about it. Requests are validated before they change state is not. So teams assert the nearest structural proxy instead, and quietly start treating the proxy as the decision. Testing has been here already: control for the number of tests in a suite and coverage turns out only weakly correlated with its ability to catch faults, with stronger coverage criteria buying no more insight than weaker ones.⁷ Coverage was never the property anyone wanted. It was the part that had a number. You believe you have protected state changes are validated first; what has a build status is the class named OrdersController does not call the package named infrastructure.

Has the field already solved this?

You might reasonably suspect I am rediscovering something with a thirty-year literature behind it. Partly, I am.

Perry and Wolf, in 1992, proposed describing an architecture in three parts — elements, form, and rationale — the last of them “an underlying, but integral, part of an architecture”.⁸ Elements and form are precisely what every tool above extracts. Rationale is where requests are validated before they change state lives, and nothing has ever read it. The same paper names the two ways architecture decays. Erosion is violation: something contradicts the architecture. Drift is insensitivity to it, and it “leads more to inadaptability than to disasters”. Every architecture test you run hunts erosion. The two endpoints I opened with are drift, and drift has never had a build status.

Software reflexion models then made the move that matters most here: state a high-level model separately, map extracted source relations onto it, and report not only where the two converge and diverge, but where the model expects something the code does not contain.⁹ Absence got its own verdict in 1995, and nothing since has moved the boundary: surveys of erosion control keep concluding that no single strategy suffices,¹⁰ and set side by side, the established conformance techniques are structural by design.¹¹

So I part company in a narrow place: the field never made the extracted relations mean anything beyond structure. A reflexion model can tell you the API layer talks to the repository layer, not that one path through that conversation writes to the database without validating first; “writes to the database” is not a relation it knows how to extract. The representation half was solved; the extraction half was left at structure, and that is now the half that matters.

Separate the concern from the evidence

The alternative is to write the architecture down in the team’s own words first, and only then go looking for something that can check it.

Write requests are validated before they change state down as the thing being protected, in exactly those words: no class, no package, no framework, and still true after the team moves off DynamoDB. Then ask a separate question: what would have to be true for us to believe it? Validation has to apply to every endpoint that writes, and no path from a request to a write may skip it. Those are the obligations.

Only then do you go looking for checks. Evolutionary architecture calls these fitness functions and deliberately refuses to tie them to one mechanism: a test, a metric, a monitor or a manual process.¹² The framing is right. What it needs is a harder requirement on each one: that it states what it proves, what it is blind to, what it costs, and how often it should run.

That sounds like bureaucracy until you try it on something harder than validation. Take the transactional outbox, which exists to stop a service’s internal state and the events other services consume from drifting apart.¹³ What would it take to believe a service really implements it?

A dependency rule proves application code cannot bypass the outbox and publish directly, not that an outbox event is ever created.
A code-graph rule can require transactional command methods reaching a known mutation API to also reach the event API, but only for the mutations and transaction semantics its model represents. Dirty checking, indirect calls, branches and reflection walk straight past it.
A whole-program query can ask whether every relevant path reaches both operations inside the modelled transaction boundary, at the cost of framework modelling, analysis time and both kinds of false result.
An integration test with rollback and failure injection shows state and outbox rows committing or rolling back together for the cases it exercises, and enumerates no path it did not run.
Relay and consumer checks exercise duplicate delivery, idempotency and ordering, and say nothing about whether the originating write produced the right event.

Five checks, five models of the world, five different blind spots. Not one of them is the architecture. Each is evidence about it, and what the team knows is all five put together, not the colour of whichever one happens to run.

The reporting has to change too. Today a check is green or red; a team needs three answers. The thing we wrote down is supported by what we ran, contradicted by it, or unassessed: nothing we ran can see it at all. That third is the one almost nothing reports, and the one that matters most. Reflexion models had it in 1995. If nobody ever wrote a check for no path may skip validation, that shows up as silence, and silence looks exactly like green.

None of the three tells you whether the architecture was a good idea in the first place. That judgement stays where it was.

Something has to turn code into intent

All of which is fine until you have to write the check. Somewhere, something has to know that PutItemCommand is a database write.

So yes, this moves the difficulty, and that is most of the point. Right now that knowledge is buried in the conditions of a dozen unrelated rules, owned by nobody. Pull it out and it becomes a thing with a name, a maintainer, and a known list of cases it gets wrong. That means giving the rules a different tree.

Not an abstract syntax tree — an intent tree, and really a graph once you add call edges. Where the syntax tree said NewExpression with an Identifier callee, the intent tree says this is a database write, reachable from POST /orders, and on one of the two paths that get there it happens before validation.

Every linter you have configured already demonstrates the property that matters. Rules match on node types; they never read source text and never name a vendor. The messy knowledge — which SDK call means what, how this framework attaches validation, which branches exist — sits on the far side of the seam, in the thing that builds the tree. Swapping a parser does not rewrite the rules above it; swapping DynamoDB for Postgres should not rewrite yours.

The comparison stops there, and where it stops is the whole difficulty. A syntax tree is a faithful transform of the source: two parsers disagree only if one of them is wrong. An intent tree is a judgement. Whether a call counts as a state change is a decision somebody made, and it can be defensible and still wrong. That is why the thing building the tree needs an owner and a published list of what it gets wrong, and why the rules above it have to be able to answer that they cannot tell.

Building that tree does not remove the difficulty. It concentrates it, and the places it gathers are not the ones I first expected.

Knowing a thing exists is not the same as knowing it applies. Establishing that an application validates requests somewhere is trivial and worth nothing. To know POST /orders is covered you have to work out how the framework attaches validation to that handler, and whether the type it validates carries any rules at all. Take away either half and the endpoint is wide open, however much validation the codebase contains.

Execution flow is not the hard part. Getting to it is. Path reconstruction looks like the place this breaks down. It is not: once you have a call graph, whether one effect precedes another or some path bypasses a required step is reachability and ordering over edges you already hold. The cost sits one step earlier. Checked against the edges programs actually took at runtime, static call graphs for ordinary Java have shown a median recall of 0.884 — roughly one edge in nine missing, with the analysis working exactly as designed.¹⁴ The dominant causes were not the reflective calls everyone worries about. They were objects arriving through native methods, and invocations the runtime initiates with no matching call site in the analysed code: framework-registered handlers, the part of a modern service you are most likely to want a rule about. No rule authoring above pays that down. Recall is the ceiling on what every rule can honestly claim, because each missing edge is a path that will never be examined and never be mentioned.

Declining to judge has to be designed in from the start. If the tooling cannot recognise the operations a rule is about, the rule must say it cannot tell, not that the code complies. Obvious written down; not what tools do by default, and retrofitting it is far harder, because by then a great many passes have been issued on the strength of having looked for nothing. Static analysis settled this better. It has a word for what real tools are. Not sound but soundy: over-approximate about most of a language, deliberately under-approximate about a well-understood subset like reflection, eval and native calls. The demand that comes with the word is that authors say precisely which features they do not handle.¹⁵ Architecture testing has never made that demand of anyone, least of all itself.

The quietest limitation outranks all three. A tool can say “I don’t know” where it notices ambiguity, but noticing is itself a capability, and the gaps are exactly where it does not notice. A call into a library nobody has modelled produces no evidence. So does an unanalysed catch block, a handler registered for later use, a dynamically dispatched call. How often a tool like this says everything is fine when it simply could not see is the measurement that would decide whether any of this is worth building, and it is not one I have made.

Putting it inside the loop

The reason to care now rather than in 1995 is that the loop has changed shape.

SWE-agent is a direct precedent for deterministic tool feedback arriving inside an agent’s own turn: its edit command runs a linter and rejects newly introduced syntax errors rather than deferring them to a later stage.¹⁶ Execution feedback supplied during generation beats no-feedback baselines,¹⁷ and in-execution feedback sidesteps a bias towards self-generated tests that post-execution feedback picks up.¹⁸ Every one of those studies concerns functional correctness, though. I could find no comparative study isolating the effect of architecture-check feedback on autonomous coding performance, and borrowing the syntax-linting result to cover it would be precisely the move this article is arguing against. So this is a reasoned proposal, not a demonstrated result.

The shape is not mysterious. The agent adds that legacy branch, saves the file, and before it moves on something works out which written-down decisions the edit could affect, runs the cheapest checks behind them, and answers in words it can act on: a database write is now reachable without passing through validation, on this path. The agent removes the branch or routes it through validation, and never needed a person. What it must not do is guess when the answer comes back unassessed: if nothing can see whether the rule holds, that is a decision for a human, not a defect for the agent to fix.

Another model reviewing the architecture is not a substitute. It can help author a rule, explain a failure, or notice that an encoded model has gone stale. It is not the repeatable, independent oracle that closes the loop, and treating it as one puts you back where the XP article started.

The representation can drift too

The strongest objection is that this only creates a second description of the architecture, which then rots at its own pace. A year after someone writes the decision down, bulk imports are exempt, the analyser has never been taught about a new database client, and half the endpoints have moved to a framework nobody updated the rules for. Four things drifting apart instead of two, and the dashboard is still green. You have encoded the wrong architecture, more legibly than before.

There is a sharper version, and it is the one that bothers me. The rules that move cleanly between technologies are the easy ones: a forbidden dependency, a write in the wrong place. The outbox needs two effects correlated inside one atomic boundary, and a request related to a background processor that runs later. It is hard to imagine writing that without reaching for the specifics of one system’s transaction mechanism. And if every concern worth checking needs a bespoke rule, the shared vocabulary bought portability only for the cases that were never the problem. The bespoke rule becomes the definition of the architecture, by exactly the route I complained about earlier. It is an analyser wearing a rule’s clothing.

I do not think that is fatal, but I cannot tell you where the line falls, and anyone who claims to at this stage is guessing. The separation is most plausible where evidence is derived from code and the vocabulary is shared, and least plausible where it is sampled from a running system: a probe measuring availability during a broker outage is accurate about the runs it watched and silent about the ones it missed. That is a claim I am making, not a result I am reporting.

What bounds the risk is not the schema. It is ownership, versioning, blind spots published in the way a soundy analysis is supposed to publish them, tests that prove the analysers can still be wrong, and making the things nothing checks impossible to overlook. What a team writes down has to sit above the checks and outlive them. None of it removes the judgement underneath.

What a green check actually establishes

Whether or not anything like this gets built, there is a question worth taking back to the architecture tests you already run on Monday.

A passing architectural check is a claim about a model, not about your system. It establishes that within the relations the tool represents, using classifications somebody chose, across the paths it managed to reconstruct, no counterexample turned up. It is a statement about elements and form; your rationale was never in scope. Everything outside that boundary is not evidence of compliance. It is absence of evidence, rendered in the same colour.

So the question is not whether your architecture tests pass. It is what each one would have to miss for the architecture to be wrong anyway, and whether anybody would be able to tell. The two endpoints I opened with were not caught by a weaker tool than the team had bought. They were caught by no tool at all, because the thing that distinguished them was never something anyone had asked a tool to see.

Erosion was always the easy half. Red builds get fixed; confident silence is the failure mode worth designing against.

References

TNG, ArchUnit, source and user guide, checked at commit be16e6fd68c2155cfaac08cf87c1098025203a0f. https://github.com/TNG/ArchUnit/tree/be16e6fd68c2155cfaac08cf87c1098025203a0f and https://github.com/TNG/ArchUnit/blob/be16e6fd68c2155cfaac08cf87c1098025203a0f/docs/userguide/008_The_Library_API.adoc#freezing-arch-rules ↩︎
jQAssistant, “Overview,” checked at commit 582406a37fa11837e7282a0c415c76812990f579. https://github.com/jQAssistant/jqassistant/blob/582406a37fa11837e7282a0c415c76812990f579/manual/src/main/asciidoc/include/overview.adoc ↩︎
Semgrep, “Taint mode overview” and “Semgrep philosophy,” checked at commit 2fa2aa636c2591a029d5c4e9d2b1ed1c4cd93011. https://github.com/semgrep/semgrep-docs/blob/2fa2aa636c2591a029d5c4e9d2b1ed1c4cd93011/docs/writing-rules/data-flow/taint-mode/overview.mdx and https://github.com/semgrep/semgrep-docs/blob/2fa2aa636c2591a029d5c4e9d2b1ed1c4cd93011/docs/snippets/contributing/semgrep-philosophy.mdx ↩︎
GitHub, “Analyzing data flow in JavaScript and TypeScript,” CodeQL documentation, checked at commit 9567f1a22fceaf9f9124e5b4adef879a8aef058e. https://github.com/github/codeql/blob/9567f1a22fceaf9f9124e5b4adef879a8aef058e/docs/codeql/codeql-language-guides/analyzing-data-flow-in-javascript-and-typescript.rst#global-data-flow ↩︎
GitHub, “Customizing Library Models for JavaScript,” CodeQL documentation, checked at commit 9567f1a22fceaf9f9124e5b4adef879a8aef058e. The exposed extensible predicates are sourceModel, sinkModel, typeModel, summaryModel, barrierModel and barrierGuardModel; the feature is documented as beta. https://github.com/github/codeql/blob/9567f1a22fceaf9f9124e5b4adef879a8aef058e/docs/codeql/codeql-language-guides/customizing-library-models-for-javascript.rst ↩︎
ESLint, “Custom Rules” and “Custom Parsers,” ESLint documentation. https://eslint.org/docs/latest/extend/custom-rules and https://eslint.org/docs/latest/extend/custom-parsers ↩︎
Laura Inozemtseva and Reid Holmes, “Coverage Is Not Strongly Correlated with Test Suite Effectiveness,” Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), pp. 435–445. https://doi.org/10.1145/2568225.2568271 ↩︎
Dewayne E. Perry and Alexander L. Wolf, “Foundations for the Study of Software Architecture,” ACM SIGSOFT Software Engineering Notes 17(4), October 1992, pp. 40–52. Quotations verified against the paper text. https://doi.org/10.1145/141874.141884 ↩︎
Gail C. Murphy, David Notkin and Kevin Sullivan, “Software Reflexion Models: Bridging the Gap between Source and High-Level Models,” Proceedings of the 3rd ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE 1995), pp. 18–28 (https://doi.org/10.1145/222124.222136); extended as “Software Reflexion Models: Bridging the Gap between Design and Implementation,” IEEE Transactions on Software Engineering 27(4), 2001. https://doi.org/10.1109/32.917525 ↩︎
Lakshitha de Silva and Dharini Balasubramaniam, “Controlling software architecture erosion: A survey,” Journal of Systems and Software 85(1), 2012. https://doi.org/10.1016/j.jss.2011.07.036 ↩︎
Leonardo Passos, Ricardo Terra, Marco Tulio Valente, Renato Diniz and Nabor das Chagas Mendonça, “Static Architecture-Conformance Checking: An Illustrative Overview,” IEEE Software 27(5), 2010. https://doi.org/10.1109/MS.2009.117 ↩︎
Neal Ford, Rebecca Parsons, Patrick Kua and Pramod Sadalage, Building Evolutionary Architectures: Automated Software Governance, 2nd edition, O’Reilly Media, 2022. ISBN 978-1-492-09754-9. ↩︎
Debezium, “Outbox Event Router,” checked at commit 1e05a09e98d948485df11a8e9a791d8dabfe18d1. https://github.com/debezium/debezium/blob/1e05a09e98d948485df11a8e9a791d8dabfe18d1/documentation/modules/ROOT/pages/transformations/outbox-event-router.adoc ↩︎
Li Sui, Jens Dietrich, Amjed Tahir and George Fourtounis, “On the Recall of Static Call Graph Construction in Practice,” Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE 2020). Median recall 0.884, rising to a median of 0.935 with state-of-the-art dynamic-feature support at a substantial performance cost. https://doi.org/10.1145/3377811.3380441 ↩︎
Benjamin Livshits, Manu Sridharan, Yannis Smaragdakis, Ondřej Lhoták, J. Nelson Amaral, Bor-Yuh Evan Chang, Samuel Z. Guyer, Uday P. Khedker, Anders Møller and Dimitrios Vardoulakis, “In Defense of Soundiness: A Manifesto,” Communications of the ACM 58(2), February 2015, pp. 44–46. https://doi.org/10.1145/2644805 ↩︎
SWE-agent, “Agent-Computer Interface,” checked at commit 3ea751c087f32b16e039a2233dd6eefecef325d5. https://github.com/SWE-agent/SWE-agent/blob/3ea751c087f32b16e039a2233dd6eefecef325d5/docs/background/aci.md ↩︎
Xinyun Chen, Maxwell Lin, Nathanael Schärli and Denny Zhou, “Teaching Large Language Models to Self-Debug,” arXiv:2304.05128, 2023. https://arxiv.org/abs/2304.05128 ↩︎
“Revisit Self-Debugging with Self-Generated Tests for Code Generation,” arXiv:2501.12793, 2025. https://arxiv.org/abs/2501.12793 ↩︎

Software Architecture on James Sawle