Guiding Lights
- Make reversible, high-ROI decisions
- Requirements over building something cool
- Architecture first, technology second.
- Instead of designing the best architecture, try to design the least worst-architecture.
- Perfect is the enemy of good.
- Lindy effect: The longer a technology has been established, the longer it will be used.
- Use lindy effect as a litmus for determining immutable technologies (as opposed to transitionary)
Terminology
Terms that I need to know.
Examples of data architecture:
- Data Warehouse
- Cloud data warehouse
- Data mart
- Data Lake
- Modern Stack
- Lamda Architecture
- Kappa Architecture
- Data Mesh
Data warehouse
A data warehouse is a central hub used for reporting and analysis. Data in warehouse is typically highly formatted and structured for analytical use cases. It’s among the oldest and well-established data architectures.
Bill Inmon originated the concept of data warehousing and is defined as “a subject-oriented, integrated, nonvolatile and time-invariant collection of data in support of management’s decisions.”
Data Lakehouse (databricks)
A data lakehouse is a data management system that combines the benefits of data lakes and data warehouses.
Delta Lake
An optimised storage layer that supports ACID transactions and schema enforcement
Unity Catalog
A unified, fine-grained governance solution for data and AI
Domain
The real world subject area for which you’re architecting
Service
A service is a set of functionality whose goal is to accomplish a task.
Data Mesh
Each software team is responsible for preparing its data for consumption across the rest of the organization.