AI data engineering platform

Keep your warehouse fast, cheap and correct.

For ClickHouse, Cloud or self-hosted.

Book a free audit
  • Lower warehouse bill
  • faster dashboards
  • More agents running in the cluster

How Underpin works

Three AI agents. One platform underneath.

The agents are what you see: they find problems, propose fixes and answer questions. Underneath, the Underpin platform does the heavy work. It measures, tests and keeps the record each agent relies on.

Underpin platform The layer you don’t see. It measures, tests and records what the agents rely on. Tune AI agent Tune Faster queries. Same numbers. Finds your slowest queries every day Fixes them in a safe copy first Same numbers, guaranteed You approve. Then it ships. Ledger AI agent Ledger Your bill, under control. Real-time monitoring, 24/7 Every dollar traced to a table Alerts before the bill jumps Hot data fast, old data cheap and stored once Atlas AI agent Atlas Your data, fully understood. Every source, owner and metric in one place Broken data caught before anyone sees it Answers in your team’s own definitions Agents that know your business Your warehouse ClickHouse · one cluster, limited compute Tune AI agent Ledger AI agent Atlas AI agent Underpin platform The layer you don’t see. It measures, tests and records what the agents rely on. Tune Faster queries. Same numbers. Finds your slowest queries every day Fixes them in a safe copy first Same numbers, guaranteed You approve. Then it ships. Ledger Your bill, under control. Real-time monitoring, 24/7 Every dollar traced to a table Alerts before the bill jumps Hot data fast, old data cheap and stored once Atlas Your data, fully understood. Every source, owner and metric in one place Broken data caught before anyone sees it Answers in your team’s own definitions Agents that know your business Your warehouse ClickHouse · one cluster, limited compute

Why now

AI agents ask far more questions than people do.

Each question costs money and adds load to the warehouse. An agent also repeats whatever the data says, so one wrong number ends up in many answers. A warehouse that served a team of analysts well can turn slow, expensive or unreliable once agents start using it.

How we start

Start with a free audit.

  1. 1

    Free audit

    You see where your money and time go, and what to fix first.

  2. 2

    Fixes

    Underpin proposes each fix. You approve them one by one.

  3. 3

    Ongoing care

    Underpin watches the warehouse and warns you before the bill or the speed drifts.

Under the hood

The Underpin platform, in detail.

Tune

queries and models

Every change is built next to the current version and compared on output, memory and running time before it goes live.

Old vs new output branch: refactor-sessions-join
Sample · illustrative
model rows memory, old memory, new time, old time, new output
sessions_daily1,82618.4 GiB3.1 GiB142 s38 sidentical
active_users_weekly2619.7 GiB2.2 GiB64 s21 sidentical
retention_cohorts4,38022.6 GiB6.8 GiB211 s74 sidentical
orders_by_country_daily9,4124.1 GiB1.3 GiB19 s7 sidentical
4 of 4 models identical. Memory −76%, running time −68%. Safe to merge. Scroll sideways to see all columns.

Ledger

cost and monitoring

On ClickHouse you pay for the capacity you run, so the next agent question has no price of its own. Ledger keeps that capacity small. On self-hosted clusters it also plans storage table by table, because S3 bills per request and ClickHouse makes a lot of requests.

Monthly bill as the number of questions grows With per-query pricing the bill rises in a straight line with the number of questions. With capacity pricing it stays flat. Tuning lowers the flat line. per-query pricing capacity, as found capacity, after tuning questions per month → monthly bill →
The shape of the two pricing models.No data

Measured continuously

  • real timeCPU, memory and query load on every node.
  • per userWhat each person, dashboard and agent runs, and what it costs the cluster.
  • per tableStorage on each disk, parts and merges, and how far back it is still written.
  • real timeS3 requests by table and by operation.
  • dailyCost per table, reconciled against the AWS bill.

Alerts include

  • alertThe S3 write rate doubles against its baseline.
  • alertA new table breaks a storage rule.
  • alertDaily cost drifts from the AWS bill.
  • alertA table outgrows its share of SSD.
  • alertQueries run close to the memory limit.
  • alertS3 starts throttling requests.

…and more, tuned to your cluster.

Where S3 requests come from

Inserts Every insert writes a new part, and a part is several files.
Merges Merging parts on S3 reads them back and writes the result again.
Moves between disks Moving a part from SSD to S3 uploads every file in it.
Replicas Each replica writes its own copy, so every request above happens per replica.

Where each kind of table keeps its data

SSD, S3 and Iceberg placement by table type, against the age of the data Real-time event tables keep N days on SSD while small parts merge into large ones, then only large parts move to S3. Incremental dbt models keep their lookback window on SSD, then move to S3. On S3 each replica keeps its own copy. History older than a set number of months moves to Iceberg tables on S3, stored once for all replicas. User tables rebuilt from full history stay entirely on SSD. ×2 one copy per replica ×1 stored once Real-time event tables many small inserts N days on SSD while merges finish, then only large parts move to S3 Incremental dbt models delete+insert with a lookback lookback window on SSD then S3 User tables rebuilt from full history rebuilt every run, so the whole table stays on SSD written now older than N months → age of the data
Each table's SSD window comes from its own logs: how far back it is still written and merged. SSD is limited, so the windows are planned together. Scroll sideways to see the full diagram.

Old history moves to Iceberg tables on S3. Without zero-copy replication, each ClickHouse replica keeps its own copy on S3. Iceberg stores the data once, however many replicas read it. ClickHouse reads Iceberg directly, and a view unions recent data in ClickHouse with the history in Iceberg, so queries, dashboards and agents see one table.

Atlas

knowledge base

Atlas is the data knowledge base. Connectors take snapshots of the warehouse system tables and query log, dbt artifacts, Airflow, the BI tool and Kafka, and Atlas merges them into one graph by table identity, with lineage and usage per reader. On that graph it keeps owners, metric definitions, data health (test and run results, freshness) and the prompts and context the agents read. It reads metadata only, never the data itself, and runs as one small service on a DuckDB file or the ClickHouse you already have. Agents read it read-only through the HTTP API (an MCP server is planned) and query the warehouse as their own logged users.

orders_daily dbt model · ClickHouse table
Atlas sample · illustrative
About

One row per day and country: orders, revenue and refunds.

owner analytics  ·  tags finance, daily

Metrics
  • revenue = sum(amount) − refunds, net of tax
  • orders = count of paid orders
Lineage
upstream
  • stg_orders
  • stg_refunds
downstream
  • revenue_weekly
  • dashboard: Revenue overview
  • role: agent_analyst
Reads, 7 days
  • dashboards1,240
  • agent_analyst860
  • analysts210
Health
fresh · updated 06:10 · 4 of 4 tests passing last 14 runs: 13 ok, 1 failed

Use case

A consumer app’s warehouse, two months in.

ClickHouse Cloud to self-hosted on AWS, plus every heavy dbt model and dashboard tuned. Weekly figures: the last week before the tuning against the most recent full week.

Monthly bill $15k →

$6.5k

ClickHouse Cloud to self-hosted on AWS. No savings plan.

dbt runs out of memory, per week 167 →

0

While weekly model runs grew from 23k to 35k.

dbt memory, per week 1,715 GiB-h →

333GiB-h −81%

Memory × time across every model run.

Dashboard tiles failed, per week 219 →

0

Tiles that ran out of memory.

Dashboard tile, p95 9.5 s →

2.2s

Across 11,000+ tile queries a week.

Dashboard first screen, p95 30 s →

6.7s

From opening a dashboard to its first screen loaded.

Free audit

Three free audits open this month.

You choose how much access to give.

  • No access

    You export usage summaries and we look at those.

  • Read-only access

    Underpin looks at the warehouse directly and cannot change anything.

Send a message to book one.