Building an ETL Platform While Migrating to Project Antalya

At Mindwise Information Technology we built o_project, a data platform for big data and AI analytics. ClickHouse^Ⓡ is the analytical engine underneath it. That part has not changed and is not going to.

What changed was everything in front of it.

In Q3 2025 we came across Project Antalya and saw an obvious fit: keep ClickHouse as the query engine, move the bulk of our storage into an open data lake. We decided to integrate it into o_project and started migrating toward Apache Iceberg with Swarm Cluster Acceleration.

Then we looked at our ETL pipeline. It loaded data directly into ClickHouse — inserts, essentially, wrapped in some scheduling. Antalya offered something else: datasets written to S3 as Parquet, registered as Iceberg tables, queryable the moment they land. Not a configuration change. A different shape of pipeline.

We assumed the hard part would be the export itself, or the Iceberg registration. It was neither. Both turned out to be reasonably well-trodden ground. The hard part was everything around them — running exports reliably against a dozen different source systems, surviving the failures that happen at three in the morning, and knowing what state a job was actually in when it died.

That is where o_rabbit came from. It started as a small internal component for one migration and turned into a standalone platform for data ingestion, transformation, and orchestration.

One long-running process is a bad foundation

o_rabbit is built around an unglamorous observation: exports fail in boring ways. A connection drops halfway through a 400-million-row table. A worker gets OOM-killed. Someone deploys during a run. If the whole export lives inside a single process, every one of those means starting from zero.

So the first decision was to separate control from execution.

The master owns the durable state of the system — jobs, runs, tasks, leases, progress, artifacts, retries, cancellation, worker registration. It plans work and records what happened. It does not move data.

Workers do the actual labor. They read from source systems, convert records into Parquet, upload the files to S3-compatible storage, and report back.

o_rabbit architecture: the master plans and records, workers move the data.

That split is the difference between an ETL script and an ingestion platform.

Workers pull; the master does not push

Workers are not told what to do. They ask. Each one registers with the master, sends heartbeats, and requests work when it has capacity.

This sounds like an implementation detail, but it isn’t. A push model requires the master to know which workers are alive, which are healthy, and how loaded each one is — and to be right about all three at the instant it dispatches. Pull makes that question disappear. A worker that isn’t running simply doesn’t ask for anything.

It also makes workers cheap to think about. They run on a laptop during local development, inside containers, or on separate remote hosts, and you get more throughput by starting more of them — no change to the job definition, no rebalancing step. If one disappears mid-task, its lease expires and the master hands the work to someone else. The master is the control plane; workers are interchangeable execution units.

The database is the memory of the export

Every meaningful step of an export is written down. The master keeps its state in SQLite — serverless, a single file, nothing extra to operate. That last point matters more than it sounds, given that the whole purpose of the exercise was to reduce operational surface, not add to it.

The vocabulary is small. A connection describes where data comes from or where it goes. A job describes what should be exported. A run is one execution of that job, broken into tasks, and each task can have several attempts if it fails and has to be retried. When a task succeeds, its output is recorded as an artifact, with metadata about the Parquet file it produced.

The payoff shows up after something goes wrong. The question stops being “what was the process doing when it died?”, which is usually unanswerable, and becomes a query against a table. The run state is on disk. o_rabbit reads it and continues from a known point.

Turning one export into parallel work

Treating a large export as a single unit of work is simple and fragile. One slow query, one dropped connection, one failed upload, and the entire run is gone.

Instead, o_rabbit plans the export into smaller logical tasks before any data moves. For sources with an ordered cursor, the planner splits the work into ranges so different workers process different slices of the same table at the same time.

Two things fall out of that. Exports scale with the number of workers rather than the speed of one process. And failure gets smaller — a failed task is retried on its own while everything already written stays written.

Planning is configurable, because databases are not interchangeable. o_rabbit can pick concurrency and task sizing automatically, which is fine for most jobs. When it isn’t — a read replica you must not saturate, a source with expensive queries, a table with badly skewed key distribution — operators can set chunk size, planned task count, fetch limit, and maximum in-flight tasks by hand.

The goal was never to find one perfect setting for every database. It was to give the system enough structure to parallelize safely, without turning every export into a hand-built operation.

What it connects to

Sources sit behind a connector interface, which keeps the cost of adding a new one low. Most of our own volume goes through Postgres, MySQL, MongoDB, and ClickHouse itself — ClickHouse to Iceberg being the path that started this project in the first place. Alongside those, o_rabbit currently ships connectors for Oracle, MSSQL, MariaDB, Cassandra, Trino, Flight SQL, and Iceberg as a source.

The full and current list lives in the repository, together with connection examples. More are on the way; the part we care about keeping stable is the interface, not the count.

Closing the loop

The migration ultimately became much more than replacing one export pipeline with another. It fundamentally changed how our analytics platform is structured.

Today, o_project remains the user-facing platform where datasets are connected, ingestion jobs are configured, pipelines are scheduled, and dashboards are built. o_rabbit is responsible for reliably extracting data from source systems, writing it as compressed Parquet files into an S3-compatible data lake, and registering those datasets as Iceberg tables.

Instead of loading every dataset directly into ClickHouse, the data lake has become the platform’s system of record. ClickHouse, through Antalya’s Swarm Cluster Acceleration, now focuses exclusively on analytical query execution.

That separation gives us several advantages over our previous ClickHouse-only architecture.

Most importantly, storage and compute are no longer coupled. Data lives in inexpensive object storage while analytical clusters can be scaled, replaced, or upgraded independently without copying or rewriting datasets. Expanding query capacity is no longer tied to increasing storage capacity.

Operationally, this also simplifies lifecycle management. Analytical clusters become largely stateless compute resources, while the data lake maintains the canonical copy of the data with Iceberg’s transactional guarantees and snapshot history. If a cluster needs to be rebuilt or replaced, the data remains untouched.

The result is a clearer separation of responsibilities. o_project manages the user experience, o_rabbit handles ingestion and orchestration, the data lake stores data in an open transactional format, and ClickHouse does what it does best: execute analytical queries at high speed.

Where we are now

o_rabbit has been running in production since July 2026. It currently moves ~2 TB / 4 billion rows per week across 45 scheduled pipelines, feeding the Iceberg tables that o_project queries through Antalya’s swarm cluster.

The concrete difference is easiest to see on our largest dataset. A full export of 500GB took 86 minutes on the old single-process pipeline. On 4 workers it now takes 40 minutes, and a failure partway through costs us one retried chunk (several minutes) instead of the entire run.

None of this is a low-latency system, and it was never meant to be. It is batch ingestion with predictable behavior and recoverable failures, which for our workloads is the trade we wanted. Next on the list: support for new external sources, automatic master-worker discovery, and a managed service-based cloud solution.

o_rabbit is available at github.com/LevonGhukas/o_rabbit.