Architecture: Local optima versus maintainability

Aug 5 (edited) in General

Hello

I am new to this forum. I love data modelling, data vault and architecture. So I thought I make my introduction with some thoughts on architecture (just because someone compared data modelling with pineapple on pizza - not for everyone - and yes I disagree).

A data warehouse has an architecture and ideally exactly one location for each activity. The activity is carried out for all sources at this point. This reduces the maintenance effort, because you don't have to know the data intimately or remember all the details. It is sufficient to visualise the activity in order to determine where the necessary corrections need to be made.

With Data Vault also comes an architecture. A good architecture that includes configuration options for the interfaces, the Business Vault, the Data Mart and - if you are particularly adventurous - special Data Vault patterns.

And no matter how good the architecture is, there are often 'local optima' during development. If we only deviate from the warehouse guidelines for this small activity, we save ourselves a lot of work. And the exception for this source is small. Actually non-existent. Everyone can remember that.

Two years later, there are exceptions for every data source and you have to know the data exactly again to be able to make changes successfully. The maintenance effort increases. And after 5 years at the latest, only a few people will still be able to make changes and everyone will be talking about a redesign. All the small gains in speed are forgotten.

This does not mean that the architecture must not change. In fact, it should change. The only question is: will everyone benefit? Or just this data source alone. If everyone benefits, this change should also be made for everyone. Only then is it a universal change.

If you only change this for specific situations, then there are many individual customised solutions. Special solutions that need to be known before the existing implementation is adapted.

Maintaining an architecture requires a lot of effort. It is the only way to ensure that processes and the allocation of tasks to individual areas are clear. This effort is worth it, as it brings disproportionately high gains in terms of maintainability, the onboarding of new or temporary employees and enables quick or temporary changes between teams in the DWH.

Dispensing with local optima in favour of a clear architecture results in lower maintenance costs, easier onboarding and simpler - even short-term - team changes. All in all, this is very lucrative. A good architecture with the right guidelines pays off.

6 comments