Tips, Expertise, Articles and Advice from the Pro's for Your Website or Blog to Succeed
Data engineers discovered the benefits of conscious uncoupling around the same time as Gwyneth Paltrow and Chris Martin in 2014.
Of course, instead of life partners, engineers were starting to gleefully decouple storage and compute with emerging technologies like Snowflake (2012), Databricks (2013), and BigQuery (2010).
Have you ever stood in the slide projector’s glow and pitched furrowed-brow executives a change that could swing millions in revenue? Or flip the mission of hundreds of employees?
Shane and his data team navigated journalistic, product, and business interests to fine-tune the New York Times pay model. This highwire act helped transition The Gray Lady from an ad revenue to a subscriber-driven publication.
If you don’t like change, data engineering is not for you. Little in this space has escaped reinvention.
The most prominent, recent examples are Snowflake and Databricks, disrupting the concept of the database and ushering in the modern data stack era.
It's midnight in the dim and cluttered office of The New York Times, currently serving as the "situation room."
A powerful surge of traffic is inevitable. During every major election, the wave would crest and crash against our overwhelmed systems before receding, allowing us to assess the damage.
Data leaders across industries are embracing data mesh. However, it’s easy to be skeptical based on past trends that have come and gone. Those fads forced us to adjust our strategy, overhaul our tech, or re-skill our teams.
But the reason data teams want to understand better how to implement data mesh is that it solves genuine pain points. Specifically, the problems created by an exhaust-friendly data lake and the all-too-often disconnect between teams – of data producers, data consumers, and those in between – that contribute to the value chain of a data product.
Something interesting happened the other day. Our company experienced a minor data incident that our data observability platform didn’t catch for the simple reason that the data was trapped in a silo.
This got me thinking about other places data hides, not just from data observability platforms but from the scrutiny of the data team itself — and how these risks can impact our ability to successfully implement more modern or even experimental programs like data mesh. We sometimes refer to these as data silos, but really it’s the data equivalent of shadow IT.
Will Larson (writer of "An Elegant Puzzle" – recommended read) may have said it best when he wrote that one of the best kinds of reorganization is the one you don’t do.
However, data leaders inevitably reach a point where, due to team growth or evolving business demands, things just don’t work. Faced with these challenges, data organizations may swing back-and-forth between centralized vs. decentralized organizational structures until they achieve the right balance.
For many data teams, the past five years have witnessed an evolution of technology, teams, and processes that calls to mind another significant period in time: the Industrial Revolution.
From the late 18th century to the mid-19th century, the Industrial Revolution transformed economies with new tools, cheaper power sources, and more streamlined ways of organizing work in factories. And even now, visit a modern printing plant, and you’ll find a state-of-the-art operation with advanced tech and robotics.