Agents write tests that can't fail
In the repos I work on, most of the code gets written by coding agents, and most of those agents are told to add tests. They do add them. But a surprising share of those tests would stay green if you deleted the function they claim to cover. At one point I had an agent sweep a Rust workspace for this, and it came back with fifteen tests whose whole verdict was assert!(true), plus a file of four tests that quietly skipped themselves in CI and reported success. Since then I've had a cheap lexical scan running in CI over every changed Rust file, along with a few testing rules that exist mostly to close off the other ways a test can lie.
What the fakes look like
The most common one is a constructor test with nothing to check. The agent builds an object, finds out the interesting fields are private, and ends the test with a comment like "if we got here, creation succeeded" followed by assert!(true). The sweep found this in an object-storage server (four tests), a storage reader, a scoring module whose test only proved "the module hierarchy compiles", and a database wrapper whose whole module body had been commented out while its test kept passing. One test printed statistics to stdout and said in its own comment that it always passes and is only there for display.
#[test]
fn storage_new() {
let _s = FsStorage::new("/var/data");
// can't reach private fields; reaching here means it works
assert!(true);
}The second kind is the silent skip. The test reads a credential from the environment, falls back to a placeholder, and if it sees the placeholder it prints "Skipping" and returns Ok(()). In CI the credential never exists, so that test is green on every run and exercises nothing.
#[tokio::test]
async fn sync_roundtrip() -> Result<()> {
let token = env::var("API_TOKEN").unwrap_or("test_token".into());
if token == "test_token" {
println!("Skipping");
return Ok(());
}
// ... the real test, which CI never reaches
}The third kind is the mock tautology. The test swaps in a mock notification client, calls code that forwards to it, and asserts the mock was called, or that it returned what it was set up to return. That assertion can only fail if the test contradicts itself, so really it's testing the fixture. The logic that actually mattered, deciding whether to send, what to send, and what happens when sending fails, doesn't get tested at all.
The rest are variations on these: an assert_eq!(1, 1) or an assertion on a constant the test defined itself, a let _ = do_thing(); that throws away the result, or a network client test that ends with "test passes if we reach this point" and never checks what was received.
Why agents write them
This part is my guess from reading a lot of these, so take it as a guess. The agent is optimizing for the task as it was given: "add tests" and "CI must be green." When the honest test is hard to write, because a field is private, a credential is missing, or a real dependency is slow to stand up, the cheapest way to satisfy both is a test that runs and can't fail. The agent often admits it in a comment, which is the giveaway: "can't access private fields", "if this compiles, the test passes".
Mocks make this worse because they make the tautology look like real work. A fully mocked test has setup, a call and an assertion, so it gets through review at a glance even though all it checks is that the mock behaves like the mock.
The scan
The fix that worked for me is intentionally dumb. On every pull request, CI collects the changed Rust source files and runs a lexical pass over them before anything gets compiled. Any finding fails the unit. It looks for two things.
First, a literal assert!(true) statement in a test body. Second, a test function whose body has nothing in it that could make it fail, meaning no assert-family macro, no panic!/unreachable!/todo!, no .unwrap() or .expect(..), and no ?. Any attribute that looks like a test marker counts, so async and parameterized test attributes are covered as well.
A couple of details make it usable in practice. The scanner first blanks out the insides of string and character literals, raw strings and comments, so a test fixture that has assert!(true) inside a string of generated code doesn't trip it, and the brace counting only sees real code. After that it allows for the usual ways a real test hides its assertions. #[should_panic] tests are skipped because the panic is the assertion. Property-test macro blocks are skipped. A test that calls a helper in the same file, where the helper's body has failure signal, counts as asserting. So does any call named like assert_*, check_* or verify_*, which covers helpers in other files by convention.
It's built to miss things rather than throw false positives, and the module documents what it misses: assert_eq!(1, 1) and other spelled-out tautologies, a check_* call whose result gets dropped, an .unwrap() on something that can't fail, and helper chains more than one hop deep. A noisy gate ends up bypassed or ignored, so I'd rather catch the obvious cases with no arguments than go after the subtle ones and lose trust in the gate. The silent-skip test above actually passes the scan, since it has a ?. That one gets caught by policy instead.
One side effect I liked is in new-crate scaffolding. The template's test module comes with its example test commented out, because an empty test body is exactly what the gate rejects. So every new crate starts without a placeholder test pretending to be coverage.
The policy around it
The scan handles the cases with zero failure signal. The rules are for the shapes a lexer can't see.
Never mock. Hit the real dependency or skip the test. A fake at the boundary is fine as long as the test asserts on the decisions made in front of it, but a test where every collaborator is mocked gets rejected in review. A connection-refused error from a port with nothing listening is fine as a real negative path, because the library returns it deterministically.
Fail loud or move it out of CI. Tests that need credentials or hardware go behind a separate configuration flag that CI doesn't compile, and they .expect() their environment variables. If CI runs a test, the test has to actually run.
Tests live inline, next to the code, in a #[cfg(test)] module in the file they cover. This rule started out as a build-time fix. The workspace had 364 integration-test binaries across 142 crates, each linking its crate's whole dependency tree, and I had them collapsed into inline modules. It helps here too. A test next to the code can reach private items, which takes away the most common excuse for assert!(true), and a reviewer sees the test and the behavior in the same diff. The agent tooling now refuses writes that would add a new test binary.
What I'd tell someone else
The scan's own comments ended up being part of the lesson. The scanner module's header still says it isn't wired into the pipeline and that hooking it up is a separate change. But the CI pipeline started calling it in the same push and still does. Later, another agent wrote a different scanner and repeated the stale claim in its own docs, citing it as a cautionary example of a scanner that exists without being a gate. Comments drift the same way tests do, and the only check that turned out to be reliable was reading the call site.
More generally, a green suite written by agents tells you less than a green suite written by people who expect to get paged when it breaks. A static gate on the obvious cases costs milliseconds per pull request and gets rid of a whole category of fake tests. For everything else, I ask one question of every new test: if I broke the function, would this go red? If the answer is "probably not," I don't count it as coverage yet.