Prompting for Unit Test Generation
Why Generated Tests Need Special Care A unit test checks that a small unit of code, usually one function or method, behaves correctly for specific inputs. Writing tests is one of the most popular uses of language models in development, because it is repetitive and time-consuming.
There is a hidden risk. A model asked to "write tests for this function" usually reads the function and writes tests that confirm what the code currently does. If the code has a bug, the tests confirm the bug. Such tests pass, look thorough, and protect nothing.
Good test generation therefore follows three principles:
- Tests come from the specification (what the code SHOULD do), not only the implementation.
- Tests cover categories: normal cases, boundaries, invalid input, and edge cases.
- Tests are checked: they must fail when the code is wrong. This lesson provides prompts that follow these principles, and a simple technique, mutation testing, for checking whether generated tests actually catch bugs.
Recipe 1: Tests From a Specification Use it when: testing a function whose intended behavior you can describe.
Prompt Write pytest tests for the function described below.
Specification: {specification}
Function signature: {signature}
Write tests in these categories, with at least 2 tests each:
- Normal cases
- Boundary values: values exactly at limits, and just inside and just outside them
- Invalid input: values that must raise errors, using pytest.raises
- Edge cases: empty, zero, very large, or unusual but valid inputs
Rules:
- Base expected values on the specification, not on any implementation.
- Show the calculation for any computed expected value in a comment.
- One behavior per test; use descriptive test names.
- Use @pytest.mark.parametrize where several inputs test the same behavior. Example Specification is_refund_eligible(days_since_purchase: int, percent_completed: int) -> bool Returns True if the purchase was at most 7 days ago AND less than 20% of the course is completed. Raises ValueError if either value is negative or percent_completed > 100. Example Output (Abbreviated) import pytest
from refunds import is_refund_eligible
@pytest.mark.parametrize( "days, percent, expected", [ (3, 10, True), # clearly eligible (7, 19, True), # both values at the inside edge of their limits (8, 10, False), # one day past the 7-day limit (3, 20, False), # exactly 20% completed is not "less than 20%" ], ) def test_refund_eligibility_rules(days, percent, expected): assert is_refund_eligible(days, percent) is expected
@pytest.mark.parametrize("days, percent", [(-1, 10), (3, -5), (3, 101)]) def test_invalid_values_raise_value_error(days, percent): with pytest.raises(ValueError): is_refund_eligible(days, percent) Why It Works Tests come from the specification, so they define correctness independently of the implementation. Boundary categories are where most bugs are: <= versus < mistakes appear exactly at limits such as 7 days and 20%. Comments explaining expected values make each test reviewable. The comment "exactly 20% completed is not less than 20%" shows the reasoning, so a reviewer can confirm it. parametrize keeps many similar cases readable in one test function. Recipe 2: Tests for Existing Code (Characterization Tests) Use it when: code has no tests and no reliable specification, for example before refactoring legacy code.
A characterization test records what the code currently does, so that later changes can be checked for unintended differences.
Prompt Write pytest characterization tests for the function in . These tests should record its current behavior, so we can refactor safely.
- Cover normal inputs, boundaries, and edge cases you can identify from the code.
- Do not "correct" behavior: record what the code does now.
- Mark any behavior that looks like a possible bug with a comment starting "# QUESTION:".
{code} Why It Works Characterization tests are honest about their purpose: they protect current behavior, not correct behavior. The # QUESTION: comments turn the model's observations into a list of possible bugs for a person to decide on, instead of silently encoding them as correct.
Recipe 3: Mocking External Services Use it when: the code calls something external, such as an API, an email service, or a payment gateway, which tests should not call for real.
A mock is a stand-in object that replaces a real dependency during a test, returns controlled values, and records how it was called.
Prompt Write pytest tests for the function in , which calls {external_service}.
- Replace the external call with unittest.mock.patch, patching it where it is used.
- Test: a successful response, an error response, and a timeout.
- Assert both the function's result and how the external service was called (arguments and number of calls).
- Do not make any real network calls.
{code} Example Output (Abbreviated) from unittest.mock import patch
from notifications import send_class_reminder
@patch("notifications.sms_client.send") def test_sends_one_sms_with_class_time(mock_send): mock_send.return_value = {"status": "sent"}
result = send_class_reminder(phone="+919800000000", class_time="18:00")
assert result is True mock_send.assert_called_once() assert "18:00" in mock_send.call_args.kwargs["message"] Why It Works "Patching it where it is used" addresses the most common mocking mistake. If notifications.py imports sms_client, the patch target must be notifications.sms_client.send, not the module where sms_client is defined. Error and timeout cases test the paths that real services occasionally take, which are the paths most often left untested. Asserting how the mock was called checks that the right data is sent, not only that something happened. The example assumes send_class_reminder calls sms_client.send with a message keyword argument. Generated mocks must match the real code's call structure; review them against the implementation.
Recipe 4: Django View and Model Tests Use it when: testing Django views, forms, or models.
Prompt Write Django tests (django.test.TestCase) for the view in .
Test:
- A successful request with valid data: status code, response content, and database changes
- Invalid data: status code and error messages
- Access control: anonymous user, logged-in user without permission, permitted user
- Use setUpTestData to create shared test objects
- Use self.client to make requests; do not call view functions directly
{view_and_url_code} Why It Works Database changes are checked, not just responses: an enrollment view should create exactly one enrollment record. Access control tests are frequently missing from generated tests, and they protect against serious security problems (Lesson 90, Recipe 8). setUpTestData creates shared data once per test class, which keeps tests fast. Recipe 5: Finding Missing Tests Use it when: a test suite exists, and you want to find gaps.
Prompt Here is a function in and its existing tests in .
List the behaviors and cases that are NOT tested, grouped by:
- Boundary values
- Invalid inputs
- Edge cases
- Error handling paths Then write tests only for the 3 most important gaps. Why It Works Asking for gaps before tests focuses the model on what is missing, instead of duplicating what already exists. Limiting it to the three most important gaps keeps the output reviewable.
Checking Test Quality With Mutation Testing The Idea Mutation testing asks: if the code had a bug, would the tests notice? It creates small, deliberate changes to the code, called mutants, such as changing <= to < or and to or, and runs the tests against each one.
If the tests fail, the mutant is killed: the tests caught the bug. If the tests still pass, the mutant survived: the tests would not catch that kind of bug. A Minimal Demonstration The following example compares two test sets on the refund function. It uses the function's source code as a string so the mutants can be created easily.
SOURCE = '''def is_refund_eligible(days_since_purchase, percent_completed): return days_since_purchase <= 7 and percent_completed < 20 '''
def weak_tests(f): assert f(3, 10) is True assert f(30, 90) is False
def strong_tests(f): assert f(3, 10) is True assert f(7, 19) is True # both boundaries, just inside assert f(8, 10) is False # just past the day limit assert f(3, 20) is False # exactly at the completion limit assert f(30, 90) is False
MUTATIONS = [("<= 7", "< 7"), ("< 20", "<= 20"), (" and ", " or ")]
def load(source: str): namespace = {} exec(source, namespace) # only our own, known source code return namespace["is_refund_eligible"]
def mutation_report(tests) -> list[str]: report = [] for old, new in MUTATIONS: mutant = load(SOURCE.replace(old, new, 1)) try: tests(mutant) report.append(f"SURVIVED {old!r} -> {new!r}") except AssertionError: report.append(f"killed {old!r} -> {new!r}") return report
print("Weak tests:") print("\n".join(mutation_report(weak_tests))) print("Strong tests:") print("\n".join(mutation_report(strong_tests))) Output:
Weak tests: SURVIVED '<= 7' -> '< 7' SURVIVED '< 20' -> '<= 20' SURVIVED ' and ' -> ' or ' Strong tests: killed '<= 7' -> '< 7' killed '< 20' -> '<= 20' killed ' and ' -> ' or ' Code Explanation MUTATIONS lists small changes that represent typical bugs: wrong comparison operators and the wrong logical operator. load() creates a function from source text. It uses exec, which is acceptable here only because the source is our own known code. Never use it with untrusted code (Lesson 58). mutation_report() creates each mutant, runs the tests, and records whether an AssertionError was raised. What the Results Show Both test sets pass on the correct code. But the weak tests let all three bugs survive: they only test cases far from the limits, where <= and < behave the same. The strong tests include boundary values (7 days, 19% and 20%, 8 days), which kill every mutant.
This is why boundary tests matter, and why "the tests pass" is not enough to judge test quality.
Real Mutation Testing Tools For real projects, use a dedicated tool such as mutmut for Python, which creates many mutants automatically and reports which survive. The demonstration above shows the principle in a few lines.
Common Mistakes Mistake 1: Tests Derived Only From the Implementation What it is: Asking "write tests for this code" with no specification.
Why it happens: The code is available; the specification is not written down.
What it causes: Tests that confirm existing bugs.
How to fix it: Provide the specification, or use characterization tests labeled as such.
Mistake 2: No Boundary Tests What it is: Testing only typical values.
Why it happens: Typical values are the first that come to mind.
What it causes: Off-by-one and comparison bugs go undetected, as the mutation demo shows.
How to fix it: Require boundary categories, and check with mutations.
Mistake 3: Mocks That Do Not Match Reality What it is: Mocks that return data in a shape the real service never returns, or patch the wrong location.
Why it happens: The model guesses the external interface.
What it causes: Tests that pass while the real integration fails.
How to fix it: Provide the real interface or sample responses, and patch where the dependency is used.
Mistake 4: Unreviewed Expected Values What it is: Accepting calculated expected values without checking.
Why it happens: The tests look precise.
What it causes: Tests that enforce wrong results.
How to fix it: Require comments showing calculations, and verify them yourself.
Mistake 5: Judging Tests by Count or Coverage Alone What it is: Assuming many tests or high line coverage means good tests.
Why it happens: Counts and coverage percentages are easy to measure.
What it causes: Large test suites that miss important bugs.
How to fix it: Check whether tests fail on broken code, using mutation testing.
Best Practices Generate tests from specifications. Independent definitions of correctness catch real bugs.
Require test categories. Normal, boundary, invalid, and edge cases.
Make expected values reviewable. Show calculations in comments.
Mock external services correctly. Patch where used, test error paths, and match real interfaces.
Label characterization tests honestly. They record behavior; they do not prove correctness.
Check test quality with mutations. Surviving mutants show missing tests.
Try It Yourself Exercise 1: Specification-Based Tests Write a specification for a function that calculates a course's early-bird price, valid only before a deadline date. Use Recipe 1 to generate tests, verify every expected value by hand, then implement the function and run the tests.
Exercise 2: Mutation Check Add two more mutations to MUTATIONS, such as changing 7 to 6, and run mutation_report() with the strong tests. If a mutant survives, add a test that kills it.
Exercise 3: Mock an External Service Write a function that fetches a course's seat count from an external API using the requests library, and returns "Full" when no seats are left. Use Recipe 3 to generate tests that mock the API for a normal response, a sold-out response, an error status, and a timeout. Confirm that no real network call is made.