Testing Strategy¶

This document outlines the testing strategy for Bedrock Swarm, including test organization, coverage requirements, and best practices.

Test Organization¶

Tests are organized into the following categories:

Unit Tests (tests/unit/)
Test individual components in isolation
Mock external dependencies
Focus on edge cases and error handling
Integration Tests (tests/integration/)
Test component interactions
Use real AWS services
Verify end-to-end functionality

Coverage Requirements¶

We maintain strict coverage requirements for different components:

Core Components (95%+ coverage)¶

Calculator Tool: 100%
Events System: 100%
Validation: 94%
Time Tool: 95%
Send Message Tool: 92%
Agency: 91%

Model Layer (80%+ coverage)¶

Titan Model: 83%
Claude Model: 36% (In Progress)
Model Factory: 43% (In Progress)

Supporting Components (60%+ coverage)¶

Base Tools: 61%
Memory System: 53%
Base Agents: 33% (In Progress)
Thread System: 18% (In Progress)

Running Tests¶

Basic Test Execution¶

# Run all tests
pytest

# Run specific test file
pytest tests/unit/test_model_titan.py

# Run with coverage
pytest --cov=bedrock_swarm

# Run with verbose output
pytest -v

Test Configuration¶

The project uses pytest.ini for test configuration:

[pytest]
minversion = 6.0
addopts = -ra -q --cov=bedrock_swarm
testpaths = ["tests"]
markers =
    integration: marks tests as integration tests that should not be mocked

Test Fixtures¶

Common test fixtures are defined in tests/unit/conftest.py:

AWS Configuration

@pytest.fixture(autouse=True)
def mock_aws_session():
    """Mock AWS session for all unit tests."""
    with patch("boto3.Session") as mock_session:
        # Configure mock session
        yield mock_session

Mock Tools

@pytest.fixture
def mock_tool():
    """Create a mock tool instance."""
    return MockTool()

Mock Models

@pytest.fixture
def mock_model():
    """Create a mock model instance."""
    return MockBedrockModel()

Best Practices¶

Direct Method Testing

def test_execute_impl():
    """Test direct method implementation."""
    tool = CalculatorTool()
    result = tool._execute_impl(expression="2 + 2")
    assert result == "4"

Error Handling Tests

def test_invalid_input():
    """Test error handling for invalid input."""
    with pytest.raises(ValueError, match="Invalid input"):
        tool.execute(invalid_param="value")

Edge Cases

def test_edge_cases():
    """Test edge cases and boundary conditions."""
    assert tool.execute(value=0) == "0"
    assert tool.execute(value=MAX_VALUE) == str(MAX_VALUE)

Mocking External Services

@patch("boto3.client")
def test_aws_integration(mock_client):
    """Test AWS service integration."""
    mock_client.return_value.invoke_model.return_value = MOCK_RESPONSE
    result = model.invoke(message="test")
    assert result["content"] == "expected"

Test Documentation¶

Each test file should include:

Module Docstring

"""Tests for calculator tool implementation."""

Test Function Docstrings

def test_basic_arithmetic():
    """Test basic arithmetic operations."""

Test Categories

class TestCalculatorTool:
    """Tests for calculator tool functionality."""

Continuous Integration¶

Tests are run automatically on: - Pull request creation - Push to main branch - Daily scheduled runs

CI Configuration¶

name: Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run tests
        run: |
          pip install -e ".[test]"
          pytest --cov=bedrock_swarm

Test Development Workflow¶

Write Tests First
Define expected behavior
Write test cases
Implement functionality

Run Tests Locally

# Run tests with coverage
pytest --cov=bedrock_swarm

# Generate coverage report
coverage html

Review Coverage
Identify uncovered code
Add missing test cases
Verify edge cases
Update Documentation
Document test changes
Update coverage requirements
Note any special cases