As we iterate through the TDD process, we’ll build up a large suite of tests. When writing so many tests, it is easy to lose focus of the scope and the purpose of each test.

There are three broad categories of coded tests that we’ll discuss: Unit, Integration, and End-to-End

Up to this point in our Tic-Tac-Toe development, we have been writing Unit Tests.

Download the source code to follow along.

Unit Tests

A Unit Test tests 1 specific isolated bit of functionality, on 1 component, taking care to eliminate all external dependencies of the code being tested. External dependencies are replaced with hard-coded stand-ins (AKA mocking) to simulate different scenarios.

Specifically, a unit test has these traits:

• tests 1 public-facing component
  • all external dependencies should be replaceable
  • but if there are private nested classes that are not part of the public interface and only used within the scope of the component being tested, then it’s not necessary to mock these – presuming of course that you can test all the different code paths from the public interface
• tests 1 specific, small bit of functionality
  • it’s fine to have many asserts in the same test scenario as long as the functionality being tested is of a single logical concept
• makes no lasting changes
  • unit tests should not change any long-term state, should not depend on state created by other tests, and should not affect the outcome of other tests
  • if files are created during the unit test, they should be deleted
  • if a data layer is being tested, then the test should always execute against a clean data store
• is succinct
  • there’s going to be a lot of unit tests, so only write test code that is essential, that you don’t get overwhelmed with code
• is fast
  • there’s going to be a lot of unit tests, so try to minimize the processing time, that you don’t waste a lot of time waiting on all the tests to run
• is executed all the time
  • execute all your unit tests after every and any change – it should not take long and you will have instant feedback if you inadvertently broke something

For instance, let’s randomize which player gets to start first:

public Game CreateNewRandom(string playerName, string otherPlayerName)
{
    var swap = new Random().Next(2) == 1;
    var player1 = new Player(swap ? otherPlayerName : playerName, 'X');
    var player2 = new Player(swap ? playerName : otherPlayerName, 'O');

    return new Game
    {
        Player1 = player1,
        Player2 = player2,
        PlayerTurn = player1,
    };
}

How would we unit test this functionality?

[TestMethod]
public void GameManager_CreateNewRandom_with_random_0_should_not_swap_players()
{
    // Arrange
    var gameManager = new GameManager();
    var firstPlayer = Guid.NewGuid().ToString();
    var secondPlayer = Guid.NewGuid().ToString();
    // TODO force Random to return 0

    // Act
    var game = gameManager.CreateNewRandom(firstPlayer, secondPlayer);

    // Assert
    Assert.IsNotNull(game);
    Assert.IsNotNull(game.Player1, "game.Player1");
    Assert.AreEqual('X', game.Player1.Mark, "game.Player1.Mark");
    Assert.AreEqual(firstPlayer, game.Player1.Name, "game.Player1.Name");
    Assert.IsNotNull(game.Player2, "game.Player2");
    Assert.AreEqual('O', game.Player2.Mark, "game.Player2.Mark");
    Assert.AreEqual(secondPlayer, game.Player2.Name, "game.Player2.Name");
}

[TestMethod]
public void GameManager_CreateNewRandom_with_random_1_should_swap_players()
{
    // Arrange
    var gameManager = new GameManager();
    var firstPlayer = Guid.NewGuid().ToString();
    var secondPlayer = Guid.NewGuid().ToString();
    // TODO force Random to return 1

    // Act
    var game = gameManager.CreateNewRandom(firstPlayer, secondPlayer);

    // Assert
    Assert.IsNotNull(game);
    Assert.IsNotNull(game.Player1, "game.Player1");
    Assert.AreEqual('X', game.Player1.Mark, "game.Player1.Mark");
    Assert.AreEqual(secondPlayer, game.Player1.Name, "game.Player1.Name");
    Assert.IsNotNull(game.Player2, "game.Player2");
    Assert.AreEqual('O', game.Player2.Mark, "game.Player2.Mark");
    Assert.AreEqual(firstPlayer, game.Player2.Name, "game.Player2.Name");
}

Once we write the test, we see the problem – there is no way to control what Random gives us when this line of code executes. Random is an external dependency that needs to be mocked in order to isolate the code being tested.

(We’ll get to how to solve this problem in the next installment).

Unit tests are the nano-level of testing. Unit tests cover every minutiae of the code.

Integration Tests

When you widen the scope, you get Integration Tests: tests which cover a specific functionality that spans across different components or external dependencies..

An integration test has these traits:

• tests 1 public-facing component
  • the test code should still focus on a single entry point
  • some, if not all, external dependencies will execute their actual code
  • unreliable or unpredictable dependencies unrelated to the test should still be mocked – such as network calls or timing-sensitive code
  • the effects of the deeper dependencies may be tested and asserted
• tests 1 specific, larger bit of functionality
  • the functionality being tested should still be a single logical concept although the assertions may cover the effects of multiple components
• makes no lasting changes
  • same as with Unit Tests
• is executed frequently
  • run the integration tests after all the unit tests are passing

If we wanted to test how CreateNewRandom and Random work together, that would be an integration test:

[TestMethod]
[TestCategory("Integration")]
public void GameManager_CreateNewRandom_should_swap_players_50_pct_of_the_time()
{
    // need a large number of runs to ensure a smooth statistical distribution
    var testRuns = 10000;
    var numSwapped = 0;
    var numNotSwapped = 0;
    var player1 = Guid.NewGuid().ToString();
    var player2 = Guid.NewGuid().ToString();
    var gameManager = new GameManager();

    for (var i = 0; i < testRuns; i++)
    {
        var game = gameManager.CreateNewRandom(player1, player2);

        if (game.Player1.Name == player1)
        {
            numNotSwapped++;
        }
        else
        {
            numSwapped++;
        }
    }

    // expect 1:1
    var expectedRatio = 0.5d;
    // but give it 10% margin of error either way
    var marginOfError = 0.1d;

    var actualRatio = Math.Abs(numSwapped / (double) testRuns);

    Console.WriteLine($"Swapped: {numSwapped}, not swapped: {numNotSwapped}, total: {testRuns}, ratio: {actualRatio:0.000}");

    Assert.IsTrue(Math.Abs(actualRatio-expectedRatio) <= marginOfError, $"Expected a ratio of {expectedRatio:0.000} but got {actualRatio:0.000}");
}

[2] I like to decorate or denote Integration tests so that I can decide which tests to run.
[11] Note that there is no mocking of dependencies here, i’m testing against real code

Integration tests give us a feel for how multiple components work together.

End-to-End Tests

Finally we have End-to-End Tests: tests which execute an entire vertical slice of the codebase and verify the big-picture functionality. E2E Tests often align with high-level feature requirements.

An End-to-End test:

• simulates real-world usage as closely as possible
• aligns with a single high-level requirement
• should only mock dependencies when absolutely necessary (produces undesirable side-effects or prevents the test from succeeding)
• does not verify all the details (save those for unit tests), only the important ones
• may have side effects, but those should not affect other tests or subsequent test runs
• is executed rarely
  • run End-to-End tests after all the unit and integration tests are passing, and after completing a large effort

For Tic-Tac-Toe, a high-level requirement would be something like, “Can play a game to completion where X wins” – and this would be verified with an End-to-End test.

[TestMethod]
public void Can_play_a_game_to_completion_where_X_wins()
{
    var mgr = new GameManager();
    var player1 = Guid.NewGuid().ToString();
    var player2 = Guid.NewGuid().ToString();
    var game = mgr.CreateNew(player1, player2);

    var moves = new[]
    {
        // _|_|_
        // _|_|_
        //  | | 
        5,
        // _|_|_
        // _|X|_
        //  | | 
        2,
        // _|O|_
        // _|X|_
        //  | | 
        1,
        // X|O|_
        // _|X|_
        //  | | 
        9,
        // X|O|_
        // _|X|_
        //  | |O
        7,
        // X|O|_
        // _|X|_
        // X| |O
        4,
        // X|O|_
        // O|X|_
        // X| |O
        3,
        // X|O|X
        // O|X|_
        // X| |O
    };

    var expectedWinnerName = player1;
    var expectedWinnerMark = 'X';
    var expectedBoard = new char?[]
    {
        'X', 'O', 'X',
        'O', 'X', null,
        'X', null, 'O'
    };

    foreach (var move in moves)
    {
        game.MakeMove(move);
    }

    Assert.IsTrue(game.IsFinished);
    Assert.AreEqual(expectedWinnerName, game.Winner.Name, nameof(game.Winner.Name));
    Assert.AreEqual(expectedWinnerMark, game.Winner.Mark, nameof(game.Winner.Mark));

    for (var i = 1; i <= 9; i++)
    {
        Assert.AreEqual(expectedBoard[i-1], game.Board[i], $"Expected {expectedBoard[i-1]} at position {i}, got {game.Board[i]}");
    }
}

With End-to-End tests, we can have confidence that a feature is working as designed. There’s not much need to run through every possible scenario with these tests, because that should have been covered by the Unit Tests.

Tests, Tests, Tests

When you write all these tests, you will build up a comprehensive test suite that covers small, medium, and high-level features. This will give you confidence in your code. By thinking through all the scenarios, you will have a better and clearer understanding of the requirements and the design.

TL;DR

Test Type	Scope	Scale	Execute
Unit	nano	many	constantly
Integration	micro	some	regularly
End-to-End	macro	few	periodically

If you’ve found this tutorial useful, please share it! Got suggestions or feedback? Please leave a comment – thanks!