How do you empower engineering teams to improve, at scale?

Following the success of our recent workshop at SEACON, Matthew Skelton, Team Topologies co-author and Director at Conflux and Benedict Steele, Chief Delivery Officer at Armakuni discussed in this webinar how the Multi-team Software Delivery Assessment (MSDA), approach can be used to assess and empower engineering teams in fast paced businesses.

With fast flow approaches like Continuous Delivery and Team Topologies, different teams and streams use a range of different practices and techniques.

 So, how can you find and emphasise the good practices? How do teams know what good looks like? And how can you assess these practices across tens or hundreds of teams on a regular basis?

Watch the replay here:

Webinar Q&A responses:

• How do you measure if a team is more engaged?

  We gather informal qualitative feedback after a team session, but more detailed engagement metrics would need to be done in a separate tool. We’ve also been told anecdotally by leaders and managers that there is an uplift in morale after we’ve run the sessions.

• Who has access to see these patterns / themes?

  It’s completely up to you who has access to the output of the MSDA exercise. Ideally teams would share their results openly and discuss their successes and failures whilst trying to improve. However, we’re aware that the culture of some organisations aren’t as generative as that yet. We recommend discussing with the teams beforehand how they’d like the data shared (openly, anonymously, or not at all). In our experience openly is always best. Access to the themes in the MSDA, are available via softwaredeliveryassessment.com

• Do you have qualitative & quantitative metrics to assess the impact of using this?

  We usually gather qualitative data immediately after an MSDA team session. Eg. How valuable was the session for the team? The quantitative metrics emerge over time, but we strongly recommend that you do not use the metrics to compare teams directly. Instead, use metrics in aggregate to show trends across groups of teams or the whole organisation.

• How was the MSDA applied and worked through with Data Science teams and, more specifically, Machine Learning Ops Teams?

  In our experience of running this with Data Science teams, we’ve found it has helped reveal the gaps in their understanding and shown the unknown unknowns. Especially in the space where Data Science crosses over into modern engineering practices. For MLOps teams, we’ve found it covers a lot of what they need, but there are some gaps. We’re working on a more enhanced version for these types of teams, so watch this space - or if you’d like to get involved then please get in contact.

• We have gone through a very similar exercise. However, the experience we have had so far is that it takes too long to run (closer to a day rather than 2 hours) as we don’t tend to limit the discussions themselves, which are very interesting. What would you suggest we do to strike a balance between filling in the assessment responses vs initiating and facilitating useful discussion points?

  If teams want to spend a long time discussing details, there could be a wider need for more discussion on practices. Consider making more time - perhaps via lunchtime talks or an Internal Tech Conference - so that the MSDA sessions can remain more focused because people have already had a chance to talk through the background details. You may also be going into a level of detail that isn’t needed at that moment in time - or your reason for gathering the information isn’t clear. If you're trying to understand your whole ecosystem, then you may end up in analysis paralysis. If that’s the case, we’d recommend a focussed look at one or two dimensions (the ones where you believe the most pain is felt) and work to improve those

• How much do you do, and how much do you look at but ask the company to do?

  In the long term, the aim is to empower the end customer to run and interpret the MSDA themselves. In the early stages, however, a partner like Armakuni or Conflux would supervise and guide the assessment, combining it with other activities to help shape a transformation.

• What sources do you rely on to show the validity of the Spotify Squad Health Check? Other tools included in the MSDA seem to have little to no empirical evidence.

  The Spotify Squad Health Check has been used by many teams around the world over many years as a foundation or part of an assessment approach. Any “validity” comes from its use in multiple organisations over time. The practices and criteria in the MSDA are based on accepted good practice in organisations around the world. Some of the measures are backed up by the DORA and Accelerate findings, and others are based on sound expert and practitioner advice. We’re not claiming any particular empirical basis, but we have confidence in the approaches based on our practical experience.

• Are facilitators also evaluating teams' practices?

  Facilitators help teams to assess their own practices but do not directly themselves assess teams.

• If I type in softwaredeliveryassessment.com it takes me to GitHub, is that right?

  That is correct. We may provide a dedicated website in the future, but for now the details are directly on GitHub.

• Will it be useful to discuss the assessment with senior leaders, perhaps to get buy-in for more investment, for example, or is it purely for the team to self improve?

  We always see it as having two benefits - if you’re in a team, it provides a baseline for you to track progress against. If you’re a leader it helps you reveal blindspots and identify cross-team patterns where extra investment or intervention may be needed.

• On the softwaredeliveryassessment.com website do you give elaborative details on each of the assessments. What do the categories mean, for instance ‘Inter-team dependencies’?

  This is an example of where we need informed facilitators who know what the criteria and concepts mean. The details behind the questions or criteria are typically all found in books referenced in each dimension, so facilitators can read the books to deepen their knowledge ahead of an MSDA session. As well as the knowledge, we’d also recommend facilitators having some experience in the practises discussed.

• Similar exercises have been done in the past, but with depressing results for teams who already know they aren’t in a great place. Do you have any tips for handling these cases?

  Absolutely. You need to establish that the exercise isn’t one of judgement (or even assessment really), it’s to take a baseline for them to improve upon. Once you’ve run the exercise, focus on the issues that are within their gift to control. Make time for them to improve those issues. Thirdly, use the output from the MSDA to articulate with others outside the team the impact that any issues those others are having - use the MSDA as a case for investment or for change.

• How would you compare this approach with Comparative Agility?

  The assessment at Comparative Agility is comprehensive, but proprietary. The MSDA is “open core” meaning that you can use the assessment without licence costs. The MSDA is focused on software delivery whereas Comparative Agility addresses additional areas.

• How does MSDA differ from DORA metrics as there seems to be a number of overlaps?

  Yes: the MSDA includes some DORA metrics deliberately, but MSDA tackles some additional dimensions. MSDA and DORA are very well aligned and both are based partly on the same underlying books (especially Continuous Delivery and Accelerate) and practices.

• How do these differ from the Continuous Delivery Survey?

  he primary focus of MSDA is to empower teams to improve. MSDA does that by encouraging discussion within and across teams about what good looks like. A survey by itself does not include any discussion or self-improvement opportunities.

• How have you addressed issues around self assessment as opposed to focusing on empirical data? I.e. Can MSDA measurements be gathered through automated processes as opposed to individuals judging themselves?

  MSDA is a team-centric approach, so it’s teams that self-reflect, not individuals. Given that the key focus of MSDA is to empower teams to improve at scale, we think it’s vital to make the core of the assessment a self-assessment. However, there is a place for automatically-collected metrics for things like cycle time, deployment frequency, failed deployments, Work-In-Progress, flow efficiency, and so forth. Remember to ensure that any automatically-collected metrics feel like empowerment to the teams, otherwise teams will disengage and the assessment will be counterproductive.

• These kinds of evaluations are obviously a Manager's dream, but I'm not convinced it's a great source of psych safety.

  A key part of MSDA is exactly that teams feel a sense of psychological safety when involved in the assessment process. The goal must be to empower teams to improve, not to compare teams and rank them.

• How do you deal with a situation where very senior management (CTO or CEO) just don't "get it”? E.g. If they're pushing for team vs team or dev vs developer comparison?

  It’s possible to protect teams by careful use of the data collected from the assessment. Avoid giving managers or C-level people access to the team-level data and instead aggregate the data and show only trends and averages.

Meet our speakers

Matthew Skelton, Founder at Conflux

Matthew Skelton is co-author of Team Topologies: organizing business and technology teams for fast flow. Recognized by TechBeacon in 2018, 2019, and 2020 as one of the top 100 people to follow in DevOps, Matthew curates the well-known DevOps team topologies patterns at devopstopologies.com. He is Head of Consulting at Conflux and specializes in Continuous Delivery, operability, and organization dynamics for modern software systems.

• Mastodon: @matthewskelton@mastodon.social

• LinkedIn: matthewskelton

Benedict Steele, Chief Delivery Officer, Armakuni

Benedict is a creative technology leader who balances lived experience with emerging best practice. He has almost 20 years experience in technology, product, and engineering - transforming organisations of all sizes by helping them adopt new methods, techniques, and ways of working.

He is the Chief Delivery Officer at Armakuni, responsible for client delivery.

Outside of work he has very little free time due to the demands of his tiny humans. When he does get some free time he spends it wishing that he'd got a dog instead.

• LinkedIn: Benedict Steele