dbt's development loop is where data teams lose time

dbt solved transformation. However, everything around dbt’s development loop costs data teams valuable time.

Suppose a Data Engineer changes one model (say, a customer table). One.

They open a pull request (PR) and a Continuous Integration (CI) and start rebuilding the whole project. Then they wait and switch to something else, losing the thread, to come back fifteen minutes later to see if the cue turned green. Multiply that by every engineer and every PR, every day. 

But, none of that waiting has anything to do with dbt. dbt did its job. The friction came from the development loop around it.

dbt solves one layer of the loop 

dbt wins, and the argument is over: It made transformation version-controlled code, modular SQL, tests, docs, and lineage, instead of the stored procedures nobody wants to touch.

With that, dbt solves one layer. The rest of the development lifecycle around it gets stitched together instead.

Think about what it takes to ship a change to one model:

• In a local environment somebody has to set up and keep working on every machine.

• A CI process that usually rebuilds more than what changed.

• Some convention to keep developers off each other's schemas.

• A review step where the reviewer approves SQL without seeing what it does to the schema or the data downstream.

• Lineage and docs that live somewhere else.

Each of those is a separate tool, config, or script that one person on the team understands.

Shipping one model change touches six things. dbt solves one of them. The other five are a separate tool, config, or script. (Image: dbt, Thiago Lima) 

What the dbt loop costs

A team’s lead is measuring the same thing, just in different words: time from idea to production, onboarding time for a new hire, rework that shouldn't have happened.

Time. Setup, CI waits, and context switching add up, and they are not small.

Risk. A reviewer who approves a change without seeing the schema is basically guessing. Without that context the bug surfaces in production – in a dashboard, in front of a stakeholder – instead of in the PR, where a schema diff would have caught it.

Dependency. There is usually one person who knows how to get the environment running. When that person is out, the whole team slows down.

The bottleneck isn't where it seems

When a data team feels slow, the instinct is to look at the warehouse, the models, the SQL. But that part is already fast, and already solved. The bottleneck is the development loop, and speeding up the transformation won't move that bottleneck.

So stop treating the environment, the CI, the isolation, and the review as four separate problems, each solved with its own scripts and glue.

What a dbt loop looks like as one platform

That's what dbdeux from DataLakeHouse is built for. A team keeps dbt, its project, and

its Git workflow. What changes is the development loop around them in three pillars.

1. A cloud environment with no setup, isolated per developer.

Every developer builds their own schema in the company's warehouse, so people can work in the same repository at the same time without overwriting each other's tables. 

Onboarding stops being a setup ritual: a new developer opens the workspace and starts working.

2. Slim CI, without the assembly.

Let's be honest, building only the models that changed and their dependents is not new. dbt can already do this, with state:modified+ with deferral being around for years. 

The figure below is exactly a run on a 1001-model project: change one model, and instead of rebuilding 1001, the build touches 8, the model and its seven dependents. 993 are skipped.

Native dbt, on a 1001-model project. The selection is dbt's. Having it run on every PR with no setup is the part a team otherwise builds and maintains itself. (Image: dbt, Thiago Lima) 

So why does CI still rebuild everything for most teams? Because dbt being able to do it and a team actually having it on every PR are two different things. To get that slim run on every pull request, someone has to store the production manifest, keep it current, wire the state comparison into the CI provider, provision the computer, and manage the credentials for every pipeline, and then keep the whole thing working as the project changes.

That is what dbdeux does at this layer: the slim run on every PR, with no manifest to babysit and no CI infrastructure to stand up. The same dbt selection by a team could wire itself, without wiring it.

3. Review with context.

A pull request should show the reviewer what changed in the data, not only what changed in the text. state:modified+ tells what will rebuild. It does not put a schema diff, a sample-data comparison, or a breaking-change flag in front of the person approving the PR. Column-level lineage and those diffs are the part that is genuinely not a command-line flag.

None of the three is a new way to write SQL. Two of them aren't even new capabilities. What's new is that a team stops building and maintaining them by themselves.

Where to start

Track how much of a team's time goes into the transformation itself, and how much goes into everything around it: setup, CI waits, review back-and-forth, the recurring "how do I get this running" questions.

That ratio is the development loop problem.

---

About the author: Thiago Lima is an independent consultant for BRF Consulting.

BRF Consulting specializes on Data Engineering, Artificial Intelligence, Software Development, Salesforce and CRM (Customer Relationship Management). For more information about our services, email us at contact@brfconsulting.com.