Will dbt (data build tool) become the next trending data technology?
dbt (data build tool) has become one of the fast growing technologies that is used to transform your data at scale. If you are not sure about the power of dbt check out how dbt is used in a real world scenario ( https://tipseason.com/dbt-tutorial-real-world-scenario-guide/ ) .
Do you have dbt integration in your project ?
I feel like these ETL (T in the case of dbt), data analytics tools and processes are ripe for being consumed by graph driven ML (machine learning) models. Usually with ML you still have the transform step to prep data (hence the bias problems) just like you do for data analytics, but using more appropriate DBs for analysis like graph DBs really create less need to massage ("transform") your data since noisy data is easily filtered out if need be, this prevents good data getting transformed out by human data engineers, and preserves connections in the data that humans have a hard time making but ML is good at. Analyzing data in a graph DB data store is moderately simple and a natural fit.
That being said good ML engineers are even harder to find than data engineers so getting rid of ETL is probably just for the big budget companies.
Agree that Graph DBs are a great fit for some of the use cases. But even if you have graph database as underlying storage they can be mostly used as a data warehouse isn't it ? So its more like ELT as compared to ETL. Inorder to transform this graph database data into something meaningful we need tools like dbt right!?
Not really, native modeling ("something meaningful") comes with the graph network theory territory, but I think other use cases of dbt would still be valid for a while. Like testing data. Running a graph convolutional network (GCN) is still "expensive" so doing exploratory testing on new data might be better with a dbt.
Anyways, I think dbt is only for RDBMS & SQL anyways right? Some orgs will be stuck with (un)relational DBs forever probably.
This technology's growth feels predicted on the labor shortage of data engineering professionals. Despite the growing amount of bootcamps and STEM focus in USA universities, I continue to be surprised that his shortage is occurring. It doesn't look like this shortage is going away anytime soon, so data analysts are going to have to take on the work that SWEs aren't able to do. As long as someone knows SQL, they can use a DBT, so I imagine we will continue to see these types of tools proliferate.
Definitely true. I think there are two ways to look at it.
Having tools that can combine data analysis + data engineering can greatly help solve both the problems.
I think it already has 😉
We’ve been using dbt at my day job for the past two years (both self hosted and through their dbt Cloud offering) and the experience has been great. Happy to go into more detail if you’re interested.
Great to know. Just curious what is the specific use case that you are using it for ? Would love to learn more in detail.
We have two:
For internal analytics. Our primary application database is MongoDB. We built a small listener that forwards all Mongo change stream events to BigQuery in real time and have dbt Cloud jobs that run hourly to create structured tables from that raw data. More dbt models are built on top of that base layer to generate the meaningful analytics data that feeds into our BI tool.
For in-platform analytics. Very similar to the use case and approach above, but we self-host dbt and provide in-app analytics for our clients. Self-hosting because it’s easier to deploy and automate across ephemeral development environments + staging and production.
Interesting. This looks like a perfect use case we were trying to evaluate for our analytics. Thank you so much for sharing this.