The temptation
LLMs will happily write a thousand tests for your codebase. They'll look professional. The coverage report will turn green. You'll feel good for about 20 minutes.
Then you'll discover that almost none of them test the things that actually matter.
What AI-generated tests are great for
- Smoke and contract tests — does this endpoint return 200, does this function not throw on happy path.
- Snapshot tests for static markup.
- Property-based test scaffolds that you then refine by hand.
- Test data factories — boring, repetitive, perfect for AI.
We use AI heavily for all of the above.
What AI-generated tests are bad for
- The edge case the customer actually hit at 3am.
- The race condition that only shows up under load.
- The bug your colleague specifically asked you to prevent recurring.
- Anything that requires understanding the business intent behind the code.
Those tests we write by hand. They're the ones that pay rent.
Our rule of thumb
If a test is going to catch a bug that costs more than five minutes of an engineer's time, a human writes it. Everything else can be AI-assisted.
The win isn't "more tests." It's the right tests, and time freed up to write the ones that matter.