Building a Data Organization - Evolution Mindset
Principles for Tech & People (Tech Series - #1)
Introduction
How do you know you're doing the right thing? Especially in an environment where you don't know what you don't know and you're simultaneously trying to hit a moving target.
Are you confident that your technology, team, processes, communication, culture are engaged in a process of continuous improvement?
This article attempts to convey principles based on the idea of evolution, aiming to help you hit a moving target in a rapidly changing environment.
Series Background
Goal and Context
Objective
This series offers insights into constructing an effective data analytics platform and organization. The principles distilled herein derive from my experience as a "Tech Lead Data Engineering" at a rapidly growing German startup.
Company Background
During the last two years, my great colleagues and I helped scale our startup's data organization from 3 to 35 members. The entire company also expanded from 100 to 400 employees in this period. Our platform engages 300 monthly active users at the BI level. (as of April 2023)
Technical Background
Our data analytics platform primarily employs BigQuery, dbt, Datafold, Looker, and custom Python data connectors.
Series Structure
This series aims to provide comprehensive insights into the successful building of a data organization revolving around four key pillars:
Evolution Mindset: Exploring the importance of flexibility, speed, and pragmatic, value-adding actions.
Product Mindset: Highlighting the approach of treating your data analytics platform as a product to serve internal customers.
Technology: Discussing how to effectively utilize technology, emphasizing simplicity, policy automation, and DevOps principles.
Data Organization Structure: Examining optimal structures for data teams to ensure efficient communication, effective resource allocation, and clear ownership.
Terminology
Data Analytics Platform
The tools and infrastructure employed for data analytics.
Data Organization
The company's division comprised data-centric roles that work daily with the data analytics platform.
Data Platform Team
The tech team owns the data analytics platform, mostly data engineers.
Data Analytics Teams
The data teams utilize the data analytics platform for data transformation and analysis. The customers of the data platform team.
Scaling Phases: Adapting Strategies to Growth
Since the following topics regarding "Evolution Mindset" are to be considered somewhat differently depending on the phase of scaling, I introduce the scaling phases before we dive into the actual topic of this post.
As your data organization expands, your strategies and implementations need to adjust. We identify three primary scaling phases that directly influence the application of the principles outlined in this series.
Phase-1: Pioneering Phase
The data platform team handles the end-to-end data analytics platform operations, from data loading to BI. The data analytics teams haven't yet joined the company at this stage.
Phase-2: Handover Phase
The data platform team manages data loading and transformation. The newly joined data analytics teams take over the latter part of the transformation step and handle BI.
Phase-3: Specialization Phase
The data platform team focuses on infrastructure and tooling, while the data analytics teams are responsible for data transformation and BI.
These scaling phases guide the evolution of your data organization. They influence decisions about technology, actions, feedback mechanisms, and learning methods, ensuring your data organization remains efficient and productive during growth.
The Evolution Mindset
The evolution mindset is divided into four components: flexibility & speed, actions, feedback, and learning. This article introduces principles for each of these topics. The principles are designed to help you build an environment that continuously improves.
Evolution Mindset: Adapting to Change and the Unknown
Principle: Embrace an evolutionary perspective.
Understand that change is the only constant and that you are making decisions with limited knowledge and information to hit a moving target. Under these conditions, a big one-shot-attempt won't get you far, and you will almost certainly miss your target.
Flexibility & Speed: Swiftly Adapt to Embrace Evolution
Principle: Utilize fast iterations on technology and the data organization structure to facilitate evolution.
Farmers transformed teosinte, a wild grass, into modern corn through numerous iteration cycles, continually refining and selecting superior plants each season.
Principle: Don't create anything without tangible business impact.
This principle applies both to technology and data organization.
Pragmatic Technology
Use technology to drive business value. When building your infrastructure, tie it to a concrete business use case. Aim for a simple output dashboard/report enabled by your infrastructure rather than attempting only to piece together the infrastructure and tools.
Pragmatic Data Organization
Let the data organization adapt, again having an impact in mind. Start with a centralized data platform team delivering end-to-end results. Hire other data analytics teams into business units based on your resource needs. Align knowledge transfer according to scaling phases (see also “Scaling Phases” above). Always critically asses the data organizational structure, especially centralization vs. decentralization of data teams, and adjust this concerning your business needs. There’s no blueprint for an organizational structure. It’s always a tradeoff, and you must find the sweet spot in your environment at the given time.
Feedback: Reveal the Flaws
Principle: Develop processes and measures to discern strengths and weaknesses, forming the basis for future enhancements.
In phase-1 , the end-to-end work of the data platform team is the primary feedback source. As more people start working on the data analytics platform, the feedback strategy should transition to retrospectives in phase-2 and include qualitative and quantitative KPIs in phase-3.
For example, my data platform team, in phase-3, did regular retros/interviews/surveys with our customers (data analytics teams) and started measuring many KPIs like "data freshness" or "perceived analyst productiity".
Learning: Enhance Your Decision-Making
Use your established feedback mechanisms (see above) to make better decisions. Based on your internal feedback, seek external resources (books, talks, etc.) to learn more and understand your flaws better. Define initiatives to level up and start your next iteration on tech on organizational data design.
Conclusion
In conclusion, harnessing an evolving mindset is the core of building an effective data organization. It promotes quick adaptability in technological implementations and organizational structures, just as farmers evolved teosinte into corn. This mindset encourages constant learning from actions and continually refining our approach based on feedback.
As we venture further into this series, we will delve deeper into the topics of cultivating a product mindset, the role of technology, and the nuances of data organization structure. These upcoming chapters will equip you with a comprehensive understanding of building a thriving data organization from first principles.
THANK YOU FOR READING!
Questions or feedback? Please comment and reach out!