The rebuild system

Some data is derived from other data: the number of posts in a category, the number of topics in a section, the stored HTML output of a page, a search index. Derived data can become corrupt over time: an import, hand-written SQL, a library upgrade or an interrupted operation leaves counters wrong. The rebuild system recalculates this data from its source.

The rule#

Every module that produces content must include itself in the rebuild system.

The core cannot recount a table it has never heard of. If the Forum module does not register a rebuilder, the Rebuild screen cannot fix forum topics and counters. That is why the rule is not a preference but part of the module's contract.

The rule has two ways:

1. Register a rebuilder (below), or
2. Say why it is not needed: if the module has no derived data, write this into module.json:

"rebuild": {
    "exempt": "Notes keeps no derived counters or indexes; every value is stored as the user wrote it."
}

This rule is checked by cp:doctor: if an active module that has entities (a class in the Entity/ folder) neither registers a rebuilder nor declares an exemption, a rebuild.module_uncovered warning appears. The warning names the module that is missing.

php cp-core/bin/console cp:doctor

How it works#

Every class implementing RebuilderInterface gets the cpalius.rebuilder tag automatically; it is enough that the folder containing the class is defined in the module's services.yaml (Modules\Foo\Rebuild\: resource: '../../Rebuild/'). RebuilderRegistry collects all of them by id; if the same id is registered twice that is an error, because two jobs would overwrite the same route and progress state. An inactive module's rebuilder is not in the container, and the list shrinks by itself.

The job runs in batches. The runner in the browser calls rebuild($offset, $limit) on every request; if the returned number is smaller than $limit, the job is finished. This is the way to process millions of rows on cheap hosting without hitting the PHP timeout.

There are two screens:

"Rebuild all" runs the jobs in getPriority() order: content counters first, Core caches last.

Writing a rebuilder#

namespace Modules\Foo\Rebuild;

use App\Core\Rebuild\RebuilderInterface;
use Doctrine\ORM\EntityManagerInterface;

final class FooCountRebuilder implements RebuilderInterface
{
    public function __construct(private readonly EntityManagerInterface $em) {}

    public function getId(): string { return 'foo.counts'; }              // <module>.<what>; never changes once released
    public function getName(): string { return 'foo.rebuild.counts'; }    // translation key
    public function getDescription(): string { return 'foo.rebuild.counts_desc'; }
    public function getBatchSize(): int { return 100; }                   // rows processed in one request
    public function getPriority(): int { return 50; }                     // smaller first
    public function isStudioVisible(): bool { return true; }

    public function getTotal(): int
    {
        return (int) $this->em->getRepository(Foo::class)->count([]);
    }

    public function rebuild(int $offset, int $limit): int
    {
        $rows = $this->em->getRepository(Foo::class)->findBy([], ['id' => 'ASC'], $limit, $offset);
        foreach ($rows as $row) {
            // recalculate the derived value from its source
        }
        $this->em->flush();
        $this->em->clear();          // REQUIRED: a long run must not fill memory with the whole table

        return \count($rows);        // rows visited, including the ones that did nothing
    }
}

Rules:

• COUNT(*) is done here, not on the write path. When saving a post, do not count the table for the counter; increase/decrease the counter and use the rebuilder as a verifier.
• Be idempotent. Running twice must give the same result.
• Every batch must return quickly (keep getBatchSize() small).
• getTotal() must be cheap; it is only for the percentage bar.
• Trust the offset, keep no state. The runner calls from scratch on every request.
• Add the translation keys to the module's messages+intl-icu.<lang>.yaml file.

Examples#

Next: Hooks, cron, queue.