Blogs
Because sharing is caring
Essence with Ivar Jacobson
Join Murray Robinson and Shane Gibson as they chat with Ivar Jacobson about Essence; a standard to develop patterns and pattern libraries that is being used by Scrum, SAFE, Lean and Kanban organisations. We discuss object oriented software engineering, use cases, UML, RUP, Essence and AI assisted method coaching. Join us to hear from one of the founders of the software engineering industry.
Attribution Model Patterns with Yorgos Moschovis
Explore the intricacies of marketing attribution with Yorgos Moschovis on the Agile Data Podcast. Dive into the complexities of tracking customer behavior across various channels and the challenges of connecting online and offline data. Learn about Yorgos’s journey from the Office of the Auditor General to a leader in data analytics, navigating through companies like Spark New Zealand and Silicon Graphics. Understand the evolution of attribution modeling, from basic first and last touch to sophisticated, multi-touch approaches. Discover the impact of third-party cookie demise on tracking and the emergence of identity management solutions. Gain insights into how different industries, like retail and insurance, approach customer journey mapping and the significance of continuous touches versus deeper histories. Hear Yorgos’s perspective on the diminishing role of static demographics in favor of digital behavior analysis. The conversation also delves into operational improvements through actionable insights, emphasising real-world applications and AB testing over theoretical model complexities.
Building a vibrant community with Scott Hirleman
Explore the art of building vibrant communities with Scott Hirleman on the Agile Data Podcast. Uncover key insights into community growth, operational strategies, and sustainability. Discover how Scott transitioned from a stock market enthusiast to a community management expert, specifically in tech and data spaces. Learn about the rapid expansion of the Data Mesh Learning Community, the importance of engaging members, and the role of timely content and responses. Delve into the operational nuances of community building, including automated onboarding, managing different fluency levels, and establishing community vibes. Understand the critical need for investment, sustainability, and how to overcome founder dependence. Gain insights into ideal team structures for community management and the significance of adaptability and realism in community growth. These comprehensive strategies and experiences offer valuable lessons for anyone looking to nurture and grow a successful community in the tech and data sectors.
Empowering teams to take initiative with Jose Corella
Join Murray Robinson and Shane Gibson as they dive into a discussion with Jose Corella on how leadership principles from the US Air Force can find applications in civilian organisations.
Recoding America with Jennifer Pahlka
Join hosts Murray Robinson and Shane Gibson in a conversation with Jen Pahlka, founder of Code for America and author of "Recoding America". They discuss the current challenges faced by governments in providing efficient digital services and how agile product...
Layered Data Architectures with Veronika Durgin
Dive into the Agile Data Podcast with Shane Gibson and Veronika Durgin as they explore the intricacies of layered data architecture, data vault modeling, and the evolution of data management. Discover key insights on balancing data democratisation with governance, the role of ETL processes, and the impact of organisational structure on data strategy.
How can data teams use Generative AI with Shaun McGirr
Discover the transformative impact of generative AI and large language models (LMS) in the world of data and analytics. This insightful podcast episode with Shane Gibson and Shaun McGirr delves into the evolution of data handling, from manual processes to advanced AI-driven automation. Uncover the vital role of AI in enhancing decision-making, business processes, and data democratization. Learn about the delicate balance between AI automation and human insight, the risks of over-reliance on AI, and the future of AI in data analytics. As the landscape of data analytics evolves rapidly, this episode is a must-listen for professionals seeking to adapt and thrive in an AI-driven future. Stay ahead of the curve in understanding how AI is reshaping the role of data professionals and transforming business strategies.
Product thinking and user story mapping with Jeff Patton
Join hosts Murray Robinson and Shane Gibson in a conversation with Jeff Patton, an advocate of agile and lean ways of working and author of the book “User Story Mapping.” In this episode, we focus on the concepts of product thinking and user story mapping.
The patterns of Activity Schema with Ahmed Elsamadisi
In an insightful episode of the AgileData Podcast, Shane Gibson hosts Ahmed Elsamadisi to delve into the evolving world of data modeling, focusing on the innovative concept of the Activity Schema. Elsamadisi, with a rich background in AI and data science, shares his journey from working on self-driving cars to spearheading data initiatives at WeWork. The discussion centers on the pivotal role of data modeling in enhancing scalability and efficiency in data systems, with Elsamadisi highlighting the limitations of traditional models like star schema and data vault in addressing complex, modern data queries.
AgileData App
Explore AgileData features, updates, and tips
Network
Learn about consulting practises and good patterns for data focused consultancies
DataOps
Learn from our DataOps expertise, covering essential concepts, patterns, and tools
Data and Analytics
Unlock the power of data and analytics with expert guidance
Google Cloud
Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows
Journey
Explore real-life stories of our challenges, and lessons learned
Product Management
Enrich your product management skills with practical patterns
What Is
Describing data and analytics concepts, terms, and technologies to enable better understanding
Resources
Valuable resources to support your growth in the agile, and data and analytics domains
AgileData Podcast
Discussing combining agile, product and data patterns.
No Nonsense Agile Podcast
Discussing agile and product ways of working.
App Videos
Explore videos to better understand the AgileData App's features and capabilities.
Martech – The Technologies Behind the Marketing Analytics Stack: A Guide for Data Magicians
Explore the MarTech stack based on two different patterns: marketing application and data platform. The marketing application pattern focuses on tools for content management, email marketing, CRM, social media, and more, while the data platform pattern emphasises data collection, integration, storage, analytics, and advanced technologies. By understanding both perspectives, you can build a comprehensive martech stack that efficiently integrates marketing efforts and harnesses the power of data to drive better results.
Anatomy of a Data Product
A graphical overview of the components required for a Data Product
Unveiling the Magic of Data Clean Rooms: Your Data Privacy Magicians
Data clean rooms are secure environments that enable organisations to process, analyse, and share sensitive data while maintaining privacy and security. They use data anonymization, access control, data usage policies, security measures, and auditing to ensure compliance with privacy regulations, making them indispensable for industries like healthcare, finance, and marketing.
Free Google Analytics 4 (GA4) online courses
TD:LR There is some great free course content to help you upskill in Google Analytics 4 (GA4) Here are the ones we recomend.Discover the Next Generation of Google Analytics Find out how the latest generation of Google...
5E’s
As Data Consultants your customers are buying and outcome based on one of these patterns – effort, expertise, experience or efficiency.
We outline what each of these are, how they are different to each other and how to charge for delivering them.
Agile-tecture Information Factory
Defining a Data Architecture is a key pattern when working in the data domain.
Its always tempting to boil the ocean when defining yours, don’t!
And once you have defined your data architecture, find a way to articulate and share it with simplicity.
Here is how we articulate the AgileData Data Agile-tecture.
Data Architecture as a Service (DAaaS)
TD:LR Data Architecture as a Service (DAaaS), is it Buzzwashing or not? As is often the case, it depends on your point of view. Our point of view? Nope its a real thing.
Myth: using the cloud for your data warehouse is expensive
TD:LR Cloud Data Platforms promise you the magic of storing your data and unlimited elastic compute for cents. Is it too good to be true? Yes AND No. You can run a cloud platform for a low low cost, but its will take...
Observability, Tick
TD:LR Data observability is not something new, its a set of features every data platform should have to get the data jobs done. Observability is crucial as you scale Observability is very on trend right now. It feels...
DataOps: The Magic Wand for Data Magicians
DataOps is a magical approach to data management, combining Agile, DevOps, and Lean Manufacturing principles. It fosters collaboration, agility, automation, continuous integration and delivery, and quality control. This empowers data magicians like you to work more efficiently, adapt to changing business requirements, and deliver high-quality, data-driven insights with confidence.
The language of data is not so natural
TD:LR The dream is we can just point the machine at our data, ask our question and get a useful answer. With ChatGPT we are closer than we have ever been, but we are not there yet,When Nigel and I first started...
Build Data Products Without A Data Team Using AgileData
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about our AgileData SaaS product and our focus on enabling analysts to do the data work without having to rely on a team of...
How To Bring Agile Practices To Your Data Projects
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about combining agile patterns and practises with those from the data domain. Listen to the episode or read the transcript....
App Engine and Socket.IO
We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.
I can write a bit of code faster
TD:LR To get data tasks done involves a lot more than just bashing out a few lines of code to get the data into a format that you can give it to your stakeholder/customer. Unless of course it really is a one off and...
The Focus Podcast – Agile Data Governance Patterns
Early in 2022 Shane Gibson was lucky enough to talk to the Focus podcast crew about agile governance in the data domain. Watch or listen to the episode.
ELT without persisted watermarks ? not a problem
We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured
Three Agile Testing Methods – TDD, ATDD and BDD
In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing.
Using a manifest concept to run data pipelines
TD:LR … you don’t always need to use DAGs to orchestrate Previously we talked about how we use an ephemeral Serverless architecture based on Google Cloud Functions and Google PubSub Messaging to run our customer data...
“Serverless” Data Processing
TD:LR When we dreamed up AgileData and started white-boarding ideas around architecture, one of the patterns we were adamant that we would leverage, would be Serverless. This posts explains why we were adamant and what...
A Data Engineer an Agile Coach and a Fish walk into a bar…
This is the first of a series of articles detailing how we built a platform to make data fun and remove complexity for our users





























