Faster Feedback Loops: Running Tests on Git Changes

In large projects, running the entire test suite can be a significant time sink. Waiting for all tests to pass before merging changes can slow down development and increase feedback loops.

We recently implemented a system to run only the tests related to files changed in a given commit. This dramatically speeds up our test runs and provides faster feedback to developers.

The Approach

Our solution leverages git diff to identify modified files and then maps those files to their corresponding tests. The mapping process involves several strategies:

• Direct Naming: If a file named SomeClass.php exists, we look for a corresponding test file named SomeClassTest.php.
• Use-Statement Reverse Indexing: For files that use specific classes, we search for tests that exercise those classes.
• Special-Case Rules: We've defined rules for specific file types like migrations, factories, policies, and resources that are generated by frameworks. This allows us to accurately identify related tests for these file types.

Implementation Details

The process can be summarized as follows:

1. Use git diff to get the list of changed files.
2. For each changed file, determine the type (e.g., PHP class, migration).
3. Apply the appropriate mapping strategy to find related tests.
4. Execute only the identified tests.

For example, if we change a class UserService.php:

class UserService {
    public function createUser(array $data) {
        // ...
    }
}

Our system would identify UserServiceTest.php as the relevant test and execute only that test.

Benefits

• Faster Feedback: Developers receive test results much faster, allowing them to iterate more quickly.
• Reduced Pipeline Load: Running a smaller subset of tests reduces the load on our CI/CD pipeline.
• Improved Developer Experience: Faster test runs improve the overall developer experience and increase productivity.

The Takeaway

By intelligently selecting and running only the necessary tests, we've significantly improved our development workflow. This approach can be adapted to various project structures and testing frameworks. Identifying and executing relevant tests based on code changes accelerates the development process, improves feedback loops, and enhances overall efficiency.