As data costs have gotten cheaper and cheaper, data teams have a tendancy to store more and more data. This can lead to situations where the database turns into a complicated mess where no one is really sure what anything is.
The solution to this is to have good documentation and a set of rules that those with write permissions have to abide by. Whilst there are many different options and different names in this blog, I will be describing medallion architecture.
Medallion architecture is a layered data design pattern popularized for organizing data in lakehouses such as Snowflake or Databricks. It structures data into three progressive tiers—Bronze, Silver, and Gold—each representing increasing levels of refinement, quality, and usability.
Raw data enters at Bronze and flows through transformations to become business-ready at Gold, enabling scalable ELT pipelines, governance, and multi-use analytics.
-Bronze Layer: Raw Ingestion
The Bronze layer captures raw data from sources system in its original format—JSON, CSV, logs, or parquet, generally with no cleansing or schema enforcement what so ever . It acts as an immutable audit of the data which arrived in the system. An exception to the changes rule might be the addition of an ingestion time stamp in order to understand and troubleshoot time gaps.
This is classically the relm of the data engineer who would use tools like kafka or Fivetran in order to ingest there data.
-Silver Layer: Clean and Validated
As we saw in our diagram, our silver layer comes after our bronze. Where the data is refined often this will be through cleansing, deduplication, normalization, and schema enforcement. It resolves inconsistencies like nulls, formats dates, and joins related records into a normalised model.
Here we have Analytics engineers at work who will use tools such as dbt or SQL to carry out the abouve mentioned transformations to create validated datasets ready for analysis. This layer balances detail retention with usability for the data specialist.
-Gold Layer: Business-Ready Assets
The Gold layer is where we will apply business logic: aggregations, and enrichments for specific use cases like Tableau/Power BI Visualisations. It delivers curated views—daily metrics, KPIs optimized for consumption by analysts, executives who don't have the skills or time to delve into the bronze and gold layers.
It is normally consumed by data analysts who will produce reports and visualisations for the buissness.
