Shape and influence

A data platform you extend, not one you rebuild

A good move is not only worth something where it lands; it creates influence for the moves that follow. Bad shape grows weaker the longer the game runs.

Data architecture works the same way. A system thrown together runs straight away, but it constrains every choice that follows — and each round of investment costs more than the last. We build on a lakehouse architecture: storage separated from compute, tables that retain their history, pipelines managed as source code. Every later step adds to the platform rather than replacing it.

See the architecture When not to build it

The problem

Four signs your data has reached its limits

If two or more of these apply, the problem is not a missing reporting tool. It is that the data has no central home and no processing pipeline behind it.

Management figures depend on one person

Summary reports are assembled by hand from several sources. Nobody else can check them, and when that person is away the reporting period slips.

One indicator, several different numbers

Each department pulls from a different system and calculates it a different way. Meetings are spent arguing over which figure is right instead of what to do next.

Figures reach leadership only once they are out of date

This period's decisions rest on last period's position. That delay is an opportunity cost, even though it appears on no financial statement.

Past figures cannot be reconstructed

Line-of-business systems overwrite prior states. When a figure from an earlier period has to be explained, or an audit requires it, it cannot be rebuilt.

Tiering

Bronze, Silver, Gold

Data passes through three tiers, each with its own level of reliability. This tiering lets you trace where an incorrect figure originated instead of re-examining the whole pipeline.

Bronze
Raw tier

Data is loaded exactly as it arrives from the source, with nothing altered. Faulty and duplicate records are kept as they are.

Why this layer is needed When transformation logic turns out to be wrong, everything can be recalculated from scratch without asking the source systems for the data again.
Silver
Cleansed tier

Data types are standardised, duplicates removed, keys reconciled across sources, and missing values handled under a single set of rules.

Why this layer is needed This is the tier data engineers and analysts work from. One record per row, one meaning per key.
Gold
Serving tier

Summary tables by business subject, indicators calculated in advance, dimensional models built for reporting.

Why this layer is needed Reports read directly from here, so they open quickly. Each indicator has exactly one definition, written down.

Data moves in one direction only, from the bottom up. Each tier retains the version from the tier below, so an incorrect figure can be traced back to its source.

Technology stack

Eight components, all open source

There are no licence fees on this list. The cost sits in operations, and that is the part we take responsibility for.

Sources

Line-of-business systems Oracle, PostgreSQL, MySQL

Unstructured data Files, logs, documents

Ingestion

Airflow On a schedule

Rust, Python Stream processing

Storage

Iceberg on S3 Bronze, Silver, Gold

PostgreSQL Metadata

Consumption

Trino Distributed queries

Superset Reports, dashboards

Data moves in one direction, from line-of-business systems to the analytics layer. There is no return arrow — the analytics layer only reads, it never writes to the source systems.

Layer	Technology	Role
Table format	Apache Iceberg	Versioned tables, the ability to read back the state at any point in time, and schema changes that do not require rewriting all the data.
Storage	S3 operated in Vietnam	Object storage, separated from the compute layer so it scales independently and at low cost.
Query	Trino	Distributed queries over data in object storage, with the ability to join across to line-of-business databases in the same statement.
Orchestration	Apache Airflow	Scheduling and monitoring of pipelines, management of dependencies between steps, and re-running a failed step without re-running everything.
Transformation	dbt	Transformation logic written in SQL, managed in git, with data tests and automatically generated documentation.
Metadata	PostgreSQL	Holds the table catalogue, run history, data quality logs and pipeline configuration.
Processing	Rust and Python	Python for most pipelines and integration work. Rust for steps that demand high performance or run continuously.
Visualisation	Apache Superset	Reports and dashboards, with access controlled by role and by organisational unit.

Processing languages

Python for breadth, Rust for the hot paths

The language is chosen to suit each processing step, rather than one language for everything.

Python for most of the workload

Python has the widest ecosystem of data libraries, and your organisation's technical team is usually already familiar with it. Ingestion pipelines, integration work and most transformation steps are written in Python.

Rust for steps that demand performance

Steps that run continuously or handle large volumes — log file parsing, event stream normalisation, long-running services — are written in Rust, which reduces server resources and removes a class of memory errors.

No replacement of what already runs well

We do not propose moving Python components that work well over to Rust. We apply it only where the benefit can be measured, and we state the trade-off: Rust engineers are harder to find.

Points to weigh

Four questions to ask before you start

Data projects usually fail for reasons that are not technical: they are started too early, or built larger than the actual need.

Which organisations do not need a lakehouse?: Most of them. If the data fits in a single relational database and reports run in a few seconds, adding Iceberg, Trino and Airflow raises operating costs without solving anything. We recommend starting with a separate PostgreSQL analytics store and expanding only once you genuinely reach its limits.
What does this architecture really cost?: Not licence fees — the entire stack is open source. The cost is in operations: you need people who understand Trino, who can respond when a pipeline fails outside working hours, and who maintain the dbt logic as the business changes. That is the part we take on.
Does it mean replacing the systems we run today?: No. The analytics layer reads data out; it does not write back into the line-of-business systems. If the whole data platform stops, day-to-day operations carry on as normal. This is a design constraint.
Where do we start?: With one business subject where someone is genuinely waiting for the figures. We build a complete pipeline from source to report so your organisation sees results before anything is expanded. Building the full platform first and then looking for work for it is the most common way a data project fails.

Start with one business subject

We build a complete pipeline from data source to report for one subject where someone is genuinely waiting for the figures. Your organisation sees the result first, then decides whether to expand.

Call +84 969 45 98 68

or write to cnk@haidang.network