From Ledgers to Intelligence Part 7: The Cloud Data Warehouse How Redshift, BigQuery and Snowflake Rewrote the Rules

Digital Transformation | June 2026

The data warehouse had been the centrepiece of enterprise analytics for twenty years. Organisations had spent millions sometimes tens of millions on Teradata appliances, Oracle Exadata systems, and IBM Netezza boxes. They had built teams to operate them, refined physical designs over years, and embedded the warehouse as a foundational dependency for every analytical system in the business.

Then, between 2012 and 2015, three products appeared that made the on-premise data warehouse economically and technically obsolete for most use cases. Not immediately the transition took a decade but the trajectory was irreversible from the moment Amazon Redshift launched in November 2012.

Amazon Redshift: The First Affordable MPP Warehouse

Amazon Web Services launched Redshift in November 2012. It was a massively parallel processing (MPP) columnar database the same class of architecture as Teradata, which had dominated large-scale analytics for two decades. The difference was price. A 2-terabyte Teradata system cost, in capital and maintenance, approximately $10,000 per terabyte per year. Redshift launched at roughly $1,000 per terabyte per year and fell steadily from there.

Redshift stored data in a columnar format grouping all values for a given column together on disk, rather than storing complete rows. For analytical queries that aggregate a small number of columns across many rows, columnar storage dramatically reduces I/O: a query summing the revenue column does not need to read the customer name, address, or product description columns at all. Combined with aggressive data compression (columnar data compresses well because adjacent values in a column tend to be similar) and parallel query execution across a cluster of nodes, Redshift delivered data-warehouse-class performance at a fraction of on-premise costs.

Redshift was not technically groundbreaking it was built on a fork of ParAccel’s MPP database but its economics were. For the first time, a mid-market organisation could run a production data warehouse without a multimillion-dollar hardware purchase, a specialised DBA team, and a dedicated data centre. Redshift changed who could afford enterprise analytics.

Google BigQuery: Serverless SQL at Scale

Google had been running its internal BigQuery service a distributed query engine built on the Dremel research project (Melnik et al., 2010) since before Redshift launched. The public version, released in 2011 and made generally available in 2012, offered a fundamentally different architectural model: completely serverless. There were no nodes to provision, no cluster to size, no infrastructure to manage. A user uploaded data to Google Cloud Storage, wrote a SQL query, and received results Google handled all compute allocation, parallelisation, and infrastructure management invisibly.

BigQuery charged per query, based on the bytes scanned a radical departure from both on-premise (capital purchase) and Redshift (hourly compute cost) pricing models. For organisations with highly variable analytical workloads intense analysis during business hours, near-zero activity overnight serverless pricing was dramatically more economical. The trade-off was cost predictability: a poorly written query that scanned terabytes of data could generate an unexpectedly large bill.

Technically, BigQuery used a nested columnar format (Dremel’s record-shredding approach) that handled semi-structured data (JSON-like nested records) efficiently a significant advantage for organisations whose data included API responses, event logs, and other semi-structured formats that struggled in traditional columnar stores.

Snowflake: Separating Storage from Compute

Snowflake launched into public beta in 2014 with an architectural innovation that neither Redshift nor BigQuery had fully realised: complete, elastic separation of storage and compute. In Redshift, adding compute capacity required adding storage nodes you could not scale one without the other. In Snowflake, storage was managed separately in Amazon S3, and compute was provided by “virtual warehouses” independent clusters of EC2 nodes that could be started, stopped, resized, and suspended independently.

The practical implication was transformative. An organisation could run five different virtual warehouses against the same data simultaneously one for the ETL pipeline, one for the finance team’s monthly reporting, one for the data science team, one for the executive dashboard, one for ad-hoc analysis with each warehouse sized for its own workload and billed only when active. The contention that plagued shared on-premise warehouses (the ETL run slowing the reporting queries, the data science pipeline blocking the financial close) was architecturally eliminated.

Snowflake also introduced features that no on-premise system could replicate: near-zero-copy data cloning (creating a full copy of a dataset in seconds with no additional storage cost, for development/testing); time travel (querying the state of a table as it existed at any point in the past 90 days); and data sharing (sharing live data with external organisations without copying it). The Snowflake VLDB paper (Dageville et al., 2016) described these innovations in technical detail and established Snowflake as the subject of serious academic attention as well as enterprise adoption.

🏗️ Architecture: Cloud Data Warehouse Architecture (2012–2019)

☁️ Source Systems (SaaS, APIs, On-Prem DBs) → 📥 Cloud Ingestion (Stitch, FiveTran, AWS Glue) → 🗄️ Cloud Object Store (S3, GCS, ADLS Raw Storage) → 🏭 Cloud DW (Snowflake / Redshift / BigQuery) → 🔄 Transform (SQL Models, Stored Procs) → 📊 BI Layer (Looker, Tableau, Power BI)

The ELT Shift

The cloud data warehouse changed the direction of the data pipeline’s primary transformation step. Traditional ETL (Extract-Transform-Load) transformed data before loading it into the warehouse, because warehouse compute was expensive and limited. Cloud warehouses offered elastic, cheap compute making it economical to load raw data first and transform it inside the warehouse using SQL. This reversal ELT rather than ETL eliminated much of the complexity of external ETL tooling and moved transformation logic into the warehouse where it could be queried, tested, and versioned like any other database object.

The ELT shift had a second effect: it dramatically simplified the ingestion layer. If transformation happened in the warehouse, the ingestion tool only needed to move data faithfully from source to target without transforming it. This created space for a new category of lightweight, opinionated ingestion tools Fivetran chief among them that would become central to the Modern Data Stack described in Part 8.

---

References

1. Dageville, B. et al. (2016). The Snowflake Elastic Data Warehouse. Proceedings of the ACM SIGMOD 2016 International Conference on Management of Data.
2. Melnik, S. et al. (2010). Dremel: Interactive Analysis of Web-Scale Datasets. Proceedings of the VLDB Endowment, 3(1), 330–339.
3. Amazon Web Services (2012). Amazon Redshift: Fast, Simple, Cost-Effective Data Warehousing. AWS Launch Announcement.
4. Stonebraker, M. & Çetintemel, U. (2005). “One Size Fits All”: An Idea Whose Time Has Come and Gone. Proceedings of ICDE 2005.
5. Abadi, D. et al. (2013). The Design and Implementation of Modern Column-Oriented Database Systems. Foundations and Trends in Databases, 5(3), 197–280.
6. Zikopoulos, P. et al. (2012). Harness the Power of Big Data: The IBM Big Data Platform. McGraw-Hill.
7. Google Cloud (2021). BigQuery under the hood: Architecture, performance, and best practices.