There's nothing worse than a boss coming back from a pompous tech conference. Nowadays, they come back infected by the industry's latest buzzword: Data Lake. Our meticulous and structured Data Warehouse is outdated and we have to start dumping everything (databases, logs, images, audios) into a large centralized "lake".

When they come at you with this you have to take a deep breath. Although the Data Lake concept makes a lot of technical sense, the way it's sold to executives is pure smoke that hides a gigantic danger.

Schema-on-Write vs Schema-on-Read

The technical core of this discussion is where and when we enforce data rules.

In a traditional model (Data Warehouse), we use Schema-on-Write. I design a Ventas table with an INT for the ID and a DECIMAL for the price. If you try to insert a row where the price is the word "Twenty", the database rejects the transaction and throws an error. The data is always clean because the toll booth is at the entrance.

In a Data Lake, the model is Schema-on-Read. We dump JSON files, malformed CSVs and raw logs raw into an HDFS cluster or an S3 bucket, without validating anything. The structure is not enforced upon saving, but upon reading. When an analyst wants to view sales, they launch a distributed query engine like Presto, Apache Hive or Spark SQL that tries to interpret those files on the fly.

-- Using Apache Hive to query raw data as if it were a table
CREATE EXTERNAL TABLE ventas_lago (
    id_venta STRING,
    importe STRING -- Everything goes in as text because we don't know what's really there
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/datos_crudos/lago/ventas/';

-- And now we pray the CSVs don't have commas inside the amount
SELECT id_venta, CAST(importe AS DOUBLE) FROM ventas_lago;

Reflection: Welcome to the Data Swamp

The promise from software vendors is that you can store petabytes at a laughable cost and worry about the structure "later". The problem is that "later" rarely arrives.

If you allow any department to dump loose files into a shared Hadoop folder without metadata, without a data catalog and without corporate governance rules, you are not building a Data Lake. You are building a landfill. A Data Swamp.

It's useless to have 10 Terabytes of user logs if the analyst who has to process them doesn't know if the dates are in American or European format, or if the time zone is UTC. The "Data Lake" is a fantastic architecture for data scientists who need unmanipulated raw material, but it requires more discipline, more cataloging and better data flow engineering than the classic relational model. Otherwise, we will end up drowned in our own lake.