Is it possible to have dbt's Slim CI run on every pull request automatically?

dbt can select only what changed. Running Slim CI on every pull request, for every developer, is a different job, and the one dbdeux can take over. 

On most data teams, changing one model still triggers a full rebuild in Continuous Integration (CI). An engineer opens a pull request and CI sets off to build the entire project, hundreds or thousands of models, to validate a change that touched one of them while the team waits. Nobody decided it should work this way. It just never got fixed.

The strange part is that dbt has had the solution for years.

What Slim CI is

Slim CI is a simple idea: a pull request that should build only the models that changed and the ones downstream of them, not the whole project. The term comes from dbt Labs, and this capability lives inside dbt itself.

The mechanism has three pieces:

• The manifest. Every dbt run produces an artifact, manifest.json, that describes the whole project: every model, its config, and a checksum of its code. It is a snapshot of the project's state.

• State comparison. CI compares the pull request's manifest against a baseline manifest. The selector state:modified picks the models whose code changed; state:modified+ adds everything downstream of them.

• Deferral. Unchanged upstream models are not rebuilt. With --defer, their ref() calls resolve to the versions that already exist in the baseline environment, so a build of eight models can still reference the hundreds it depends on.

This whole mechanism is done in one command: dbt build --select state:modified+ --defer --state <baseline-manifest>.

Read from left to right: build only the changed models and their dependents (state:modified+), and let everything unchanged resolve to what already exists in the baseline (--defer --state). That is Slim CI in one line.

It already works in plain dbt

The first article in this series (dbt's development loop is where data teams lose time) ended on a number: change one model in a 1,001-model project, and a state-based build rebuilds 8, not 1,001. Here is the run behind it.

Native dbt, on a 1,001-model project. A full build touches all 1,001. After a change to a single model, dim_product, a state-based build touches 8, the model and its seven dependents. The other 993 are skipped. (Image dbt, Thiago Lima)

That is Slim CI, and it is entirely native dbt. No product is required. Which raises and obvious question:

Why doesn't every team have it?

If the capability is free and built into dbt, why do most teams still rebuild everything on every pull request? 

Because having the Slim CI capability and a complete running system are two different things. To turn state:modified+ into something that actually runs on every PR, someone has to build and maintain the machinery around it by:

• storing the production manifest somewhere the CI run can reach, and refresh it on every deploy so the baseline never goes stale.

• configuring --state and --defer to point at that baseline.

• creating an isolated CI schema for each pull request, so concurrent PRs do not overwrite each other.

• giving the CI run its own warehouse credentials.

• wiring all of it into the Git provider so it triggers on a pull request.

Then, keep the whole thing working as the project and the warehouse change, for every pipeline, indefinitely.

The environments nobody mentions

The one-line command hides a small system with several moving parts:

• A production environment. It holds the deployed models and the baseline manifest. It is both the source of "what changed" and the place unchanged references resolve to.

• An ephemeral CI schema. A throwaway, isolated schema where the changed models get built and tested for the pull request, kept separate from any developer's sandbox and from production.

• The data warehouse. Where the builds actually run. dbt orchestrates; the SQL executes in Snowflake, BigQuery, or whichever adapter the team uses.

• The Git provider. The pull request is what starts the run and where the result needs to land.

None of that is visible in the state:modified+ flag. All of it has to exist for the flag to be useful.

It is plumbing, and plumbing has no owner

Here is why the gap persists. None of this work is analysis or modeling. It is platform work, and it competes for time against everything the team is actually measured on. The person who could build it well is usually the person the team can least spare. So it sits on the packlog while CI keeps rebuilding the whole project, and everyone absorbs the wait.

That is the real distance between having the capability and not having it running. It is not skill, but ownership.

How dbdeux runs every pull request 

This is the layer dbdeux handles. The manifest is stored and kept current automatically, so the baseline is never stale. Each developer gets an isolated schema without setting one up. The slim run happens on every pull request, serverless, with no CI infrastructure to stand up and nothing to configure. So, it’s the same dbt selection a team could configure, without having to do so.

 dbdeux on the pull request: the new margin_bucket column, detected automatically and flagged as a safe, non-breaking change. No command-line run produces this on a PR. (Image dbdeux, Thiago Lima) 

The same change as impact: dim_product modified, and the seven downstream models it forces to rebuild, computed on the PR in milliseconds. (Image dbdeux, Thiago Lima)

The point is

The mechanism was never the hard part. dbt solved that years ago, and any team willing to build and maintain the machinery around it can have Slim CI for free. The hard part was having it, everywhere, on every pull request, without a person keeping it alive.

That gap, between the Slim CI capability and a team that actually runs it, is what dbdeux closes.

---

About the author: Thiago Lima is an independent consultant for BRF Consulting.

BRF Consulting specializes on Data Engineering, Artificial Intelligence, Software Development, Salesforce and CRM (Customer Relationship Management). For more information about our services, email us at contact@brfconsulting.com.