Improve test speed in Symfony with in-RAM database

When implementing the fundamentals of my.typo3.org, an API based on Symfony was built to feed data to applications in the TYPO3 universe, e.g. the aforementioned my.typo3.org or the Certification Platform. This API must be rock-solid, thus it has a decent test coverage for each single piece gluing the application together. With development going further, the amount of tests increased, including API endpoints that get their data from a database. In out test scenarios, we use a sqlite database as this needs no additional setup.

At some point in the development process, executing tests became slower and slower as the amount of tests and the respective amount of fixtures increased:

Time: 10:06.275, Memory: 916.00 MB

OK (771 tests, 3120 assertions)

The full test run needs ~10 minutes and consumes over 900 MB of RAM. Of course the whole test suite doesn't have to re-run when changing a single controller, but some changes are low-level and trial & error by letting the CI do the job is not really feasible.

Disclaimer: I'm developing on a 2019 Dell XPS 15 7590 with hexacore CPU Intel i7-9750H, 32 GB RAM, an M.2 NMVe SSD, Ubuntu 20.10 and every project runs with ddev.

Use the RAM, Luke

Scraping data from and pushing data back to the SSD shouldn't be that time-consuming, but I'm not that deep into Symfony and sqlite internals to properly explain what's going on here. Luckily, Symfony allows to store the database in RAM very easily by setting the database URL to sqlite:///:memory: instead. However, the first run didn't go well:

ERRORS!
Tests: 771, Assertions: 1575, Errors: 39, Failures: 299.

All failures are caused by the exception Doctrine\DBAL\Exception\TableNotFoundException, right after priming the database the tables are not available anymore after importing the fixtures. After some research I found out that Symfony keeps the database in RAM until its kernel gets shutdown, either on purpose or when a new kernel is created. This happened at three specific places:

• before priming the database
• after importing the fixtures
• booting a web client to call the API endpoints in the tests

A typical setUp() looked like this:

protected function setUp(): void
{
    parent::setUp();
    $this->prime(); // calls static::bootKernel() as well
    $this->importFixture('path/to/fixture.php');
    static::bootKernel();
}

Importing fixtures

The issue has been identified, let's start fixing it. The kernel is now booted at first in the test's setUp() method and the primer demands an already booted kernel. If the primer cannot find a kernel, a \LogicException is thrown which reveals non-adopted test classes. The primer is a trait being imported in the test classes extending \Symfony\Bundle\FrameworkBundle\Test\KernelTestCase, checking for a booted kernel is straight forward:

trait DatabasePrimer
{
    public function prime(): void
    {
        if (!self::$booted) {
            throw new \LogicException('Could not find a booted kernel');
        }

        // ...
    }
}

The fixtures are imported by Doctrine's EntityManager, calling its persist() and flush() methods. This revealed another issue: the records imported to the database could not be found. The reason is that Doctrine maintains an identity map about the records which needs to be reset by calling clear() at the end of the import process, which is described in the Doctrine documentation as well:

Sometimes you want to clear the identity map of an EntityManager to start over. We use this regularly in our unit-tests to enforce loading objects from the database again instead of serving them from the identity map. You can call EntityManager#clear() to achieve this result.

For reference, here's the reduced importFixture() method:

public function importFixture(string $fileName): void
{
    $file = new \SplFileInfo($fileName);
    // ...
    $fixtureConfiguration = require $file->getRealPath();
    foreach ($fixtureConfiguration as $models) {
        foreach ($models as $model) {
            $entityManager->persist($model);
        }
    }
    $entityManager->flush();
    $entityManager->clear();
}

Call API

The last nut was tough to crack as it affected the way how API endpoints are called in the test scope. Remember, the Symfony kernel never must get shutdown in order to keep the database in RAM. However, the official Symfony documentation recommends using static::createClient() which shuts down any existing kernel on purpose. The aforementioned method gets the client from the named service test.client from the dependency injection container and does some assertions - we skip that and get the client via static::$kernel->getContainer()->get('test.client') only. Additionally, it was required to boot all bundles per web request again in a simple loop:

public function execute(Instruction $testInstruction): Response
{
    $kernel = $this->client->getKernel();
    foreach ($kernel->getBundles() as $bundle) {
        $bundle->boot();
    }

    $request = Request::create(
        $instruction->getUrl(),
        $instruction->getMethod(),
        // ...
    );

    $response = $kernel->handle($request);
    $kernel->terminate($request, $response);

    return $response;
}

Final tests

Once these changes were done, it was time for a new test run:

Time: 00:26.634, Memory: 230.00 MB

OK (771 tests, 3120 assertions)

Amazing! The previous runs with a on-disk database always took roughly 10 minutes, now the runs need less than 30 seconds, this is an improvement of ~95%, the RAM usage went down by ~75%. However, this change has a major drawback: it is quite impossible to debug the database in the middle of a test run in case something is off with the fixtures. In that case switching back to the on-disk variants seems to be without alternative, yet.

Header photo by Fabrizio Conti on Unsplash.