Testing Rules
Test Organization
tests/test_<module>.py — unit tests for each source moduletests/agent/ — agent core, observer, batch, regression teststests/providers/ — provider-specific, model registry, streaming, format teststests/rag/ — RAG pipeline, chunking, hybrid search, vector store teststests/integration/ — cross-module integration teststests/tools/ — tool system tests
Mock Provider Pattern
class FakeProvider:
name = "fake"
supports_streaming = True
supports_async = True
def complete(self, *, model, system_prompt, messages, tools=None, **kw):
return Message(role=Role.ASSISTANT, content="response")
Always include tools=None in signature — missing it silently hides bugs.
Recording Provider Pattern
Use to verify exact args passed to provider methods:
class RecordingProvider(FakeProvider):
def __init__(self):
self.calls = []
def complete(self, **kwargs):
self.calls.append(("complete", kwargs))
return Message(role=Role.ASSISTANT, content="ok")
Regression Tests
- Go in
tests/agent/test_regression.py
- Each test class documents the specific bug it prevents
- Name pattern:
TestXxxHandling describing the failure mode
- Include a docstring explaining what broke and when
Assertions
- Model counts: update when adding/removing models
- Observer events: verify run_id is passed to all events
- Streaming: verify
ToolCall objects are yielded (not strings)
- Policy: verify deny actually blocks execution
- Guardrails: verify block raises, rewrite modifies content
Running Tests
pytest tests/ -x -q # All tests, stop on first failure
pytest tests/ -k "not e2e" -x -q # Skip E2E (CI mode)
pytest tests/agent/ -x -q # Just agent tests