There are more things in heaven and earth, Horatio, than are dreamt of in your data architecture.

William Shakespeare said that. “Billy no-dates” as his friends called him. But putting his feelings about temporal analysis aside, the philosophy is sound.

We’ve seen it ourselves of course, after the initial flooding of the data lakes in recent decades, which ultimately turned out to be shallow paddling pools, covering the grounding as far as the eye could see, with every pearl of wisdom hidden not only within the oyster that carried it but beneath 2 feet of saltwater besides. Drenched in observations of dubious provenance and even more dubious value.

The intent was both sincere and simple: we will put all of the data in the same place, respecting its original format, and connect it later when we need to. It was born out of the confluence of the rise of cheap commodity hardware and the parallel rise in the frustration of a priori transformation in an increasingly complex business landscape. In some respects it was a masterstroke. “We will solve all your ETL woes by allowing you to just… not. Remove the T altogether. Put it on ice.”

And while this was said, the sound of a million eyebrows being raised was heard among the database architect community.

But nevertheless, to some extent the idea did hold water, though of course the Data Lake was in dire need of rebranding after some bright oarsman suggested the S word to be a more appropriate description of the ensuing tangled sludge.

And so now we have the Lakehouse. All that is beautiful of the data lake – the wide vistas, the calm waters, the endless sky. And all that is industrial about the warehouse – the structure, the order, the predictability. All mixed together in a large glass of iced tea, sipped with friends beneath a lakeside gazebo on a warm summer’s eve.

Eventually the ice melted and we needed to decide what to do with the T. Stick it at the end, they said. And so we did. ELT was born. And eventually ETL/ELT, because many of us glanced through the glass of the sliding doors and realised that these terms were in fact the same thing. The minutiae of where the T is sipped turns out to be somewhat incidental.

With all this water and ice and tea, data architecture is a slippery business. Just when you think you’ve modelled the vision of the world as you and your colleagues see it, it frees itself from your grip and settles back into the puddles of incongruity and complexity. But fear not; you cannot account for every eventuality and, unless you want to end up inadvertently obscuring those pearls under several feet of drink, you shouldn’t try. So build for flexibility and accept change as being as inevitable as the ebb of the tide. Relax, take the long view with a nice sip of tea