Twirl: A new kind of orchestrator

Company Blog

Docs

A new kind of
orchestrator

Twirl comes with opinions and best practices built in, so data developers can be productive from day one

It’s 2024 and deploying
data pipelines is still
painful and slow

Slow iterations

It's hard to be sure everything works until it's live in production.

High maintenance

Cloud infrastructure requires a big up front time investment and ongoing maintenance.

Complex debugging

Debugging a job failure takes too long and requires looking in too many places.

There's an easier way with Twirl

Iterate with speed and confidence

Multi-language dev mode, unit tests, data contracts, and data validations— so you can be sure your code works before deploying.

Run everything on your cloud

We never see your data and you never do any infra work. Your dev-ops and compliance teams will love you!

Visualize everything in one place

See lineage across your databases and containerized jobs, and understand upstream and downstream impacts at a glance.

One opinionated framework
for all your data needs

Develop

Plain code

A Twirl job is plain code. No dsl, no decorators, no imports. If you can write SQL or Python, you can write a Twirl job.

Data environments

Twirl’s development mode reads input data from production but write outputs to a safe and isolated development sandbox.

Tight integrations

First class dbt support with zero overhead-- mix and match regular Twirl jobs and dbt models.

Data contracts

You may optionally define a schema which Twirl will enforce, preventing users from accidentally breaking other people's code.

Unit tests

Twirl has first class support for fixture oriented unit testing, making it easy to test edge cases and be confident your code works.

Deploy

Continuous Integration

Deploying a job in Twirl is as simple as merging a pull request. Twirl will run a Github Action pushing your new code to the orchestrator.

Your data never leaves your cloud

All data is stored in your cloud account, and all jobs execute there, so no sensitive data ever leaves your account.

Mixed scheduling

Twirl gives you full control over node scheduling, which means different jobs in the same pipeline can run at different frequencies.

Code packaging

Twirl automatically packages your code and dependencies, simplifying deployment and ensuring consistency between development and prod environments.

Monitor

Lineage

Twirl’s web app lets you easily check how all your data jobs are doing in one view.

Filter

Use tags or Twirl's selection syntax to easily zoom in on relevant jobs

Uncover

Twirl’s detail view shows you which upstream jobs failed, making it easy to debug stale tables.

Investigate

Twirl displays parsed log output so you can always see the right level of detail for your jobs.

Develop. Deploy. Monitor.

By integrating a local build tool with fully managed infrastructure, we provide a seamless experience for building and deploying data products.

Adored by world class
data companies

“Twirl is like dbt but with the added power of a full blown orchestrator... 10/10 would choose again.”

Victor Quinn

CTO and Co-founder at Texture Energy

“Twirl allowed us to move from three tools (dbt, airflow, and prefect) to just one and in doing so we reduced our code base by 30%.”

Tobias Morville

Head of Data at Monta

“Instead of hiring multiple engineers to build the same thing, we got started instantly.”

Stéphanie Cabrera

Head of Data at Bokadirekt

Build data products
effortlessly with Twirl