The McWarehouse (DW Toolkit, Ch.01 - Architecture)Written on November 29, 2022
The ‘fine-dining restaurant’ is a commonly used metaphor to describe Data Warehouse architecture. A core duty of a Data Warehouse Developer or Data Engineer is to extract raw data from a variety source systems, and transform it in a way that’s usable and consumable for the end-user. This is analogous to a chef who will hand-pick only the freshest raw ingredients, transforming them into an array of dishes that will be served to their patrons.
But not all data environments are fine-dining restaurants. When complexity is low and the needs are simple, then perhaps a spreadsheet is all that’s needed? Why hire a talented chef when a supermarket trip by a home cook will meet our needs?
In most cases, if you’re assessing the need to build a Data Warehouse environment, then it’s likely that there is more size or complexity than a spreadsheet can handle, in which case it’s better to have resources who can pick, design, and transform these sources for you, so that our patrons: 1) recieve the order what they want 2) recieve a quality product 3) receive a quality experience
Should a data warehouse environment or restaurant fail to deliver any of these components, it’s likely they will start losing customers. Word gets around fast, so it doesn’t take too many justified negative reviews for seeds of distrust to spread.
But not all patrons are looking for a fine-dining experience. Perhaps they are willing to settle for a cheaper alternative?