Organisational Dysfunctions Mirrored as Architectural Complexity
We have come to acknowledge in recent years that there is a strong relationship between the design of a software architecture and the shape of the organisation it is developed by.
Within this broad space, one sub-topic that frequently captures my attention is the relationship between organisational dysfunctions and their influence on software architecture.
Often, significant and unnecessary technical compromises in a software system will be introduced as a result of politics, misaligned incentives, or poor communication.
I’m aware that for any reasonably sized organisation the software architecture will be shaped for sociopolitical reasons, often beneficial. However, I consider dysfunction-mirrored-complexity to represent technical complexities that are extremely costly, yet unnecessary, and avoidable through improvements in social interactions.
One pattern I’ve observed in large and smaller organisations is The Accountability Protection Buffer. This pattern arises when a team needs to shield themselves from problems inherited through services they depend on owned by other teams.
Accountability Protection Buffer
When teams are accountable for delivering a level of service, but they are not fully responsible for delivering that level of service there is a political risk. The bigger the gap between accountability and responsibility, the greater the level of risk and severity for political and technical dysfunctions.
Think about this scenario: in a large organisation there are two departments. One department (Department A) is entirely responsible for the company’s website, while the other department (Department B) provides APIs consumed by the website and also third parties.
Department A is accountable for providing the best possible web experience - features, performance, usability, and more. Yet they cannot fully control each of those characteristics, they partially depend on Department B’s APIs.
Department B regard Department A as one of many customers. Their purpose is not just to serve Department A, their APIs are primarily designed for other customers. Department A encounter problems:
- The APIs are not fast enough
- The APIs are too fragile
- Department B have long lead times on adding new features whereas Department A are practising continuous delivery and want to move quickly
These problems directly result in a slow, unreliable, and un-innovative web experience. Department A has inherited these problems as a result of their dependence on Department B’s APIs.
Consequently, customer satisfaction is low and website-driven business is low. Both external customers and internal stakeholders blame Department A — the website is slow, unreliable, and rarely seems to add enhancements. Department A own the website, the website is not good enough, and therefore it’s Department A doing a bad job and letting the business down.
As a result of the inherited problems, Department A become confrontational with Department B over the lack of improvements or empathy. Their relationship deteriorates into a feud.
Confrontation and aggression yields no improvements. In fact, they make things worse. Department A are now desperate to find a way to fix their problems. They decide to implement an Accountability Protection Buffer…
They create an importer which runs overnight. It sucks all of the data out of the APIs and stores it in their own local database. This then enables them to dramatically improve the performance, reliability and usability of the website and allows them to implement some types of features much more easily.
Finally, they are now responsible and accountable.
Department A’s success was not without criticism. It took them three months to properly implement and stabilise the importer during which time no major improvements to the website were made.
Soon after, more criticisms start to emerge as a new wave of problems is encountered:
- Out-of-date information is being shown on the website — old prices and products
- New products are not being shown on the website until one day later
- The website is showing different information compared to internal systems
It becomes a full-time job for Department A to investigate and resolve data issues. Due to the poor relationship between departments A and B, the time taken and amount of energy wasted investigating data problems is dramatic.
Eventually, the complaints from customers and internal stakeholders are far worse as the two system drift further apart and the problems grow. Customers are seeing one price on the website but when they try to make a purchase they are being given a higher price. The company’s brand is being damaged, and customer loyalty is disappearing.
The Technical Costs of Organisational Dysfunctions
A significant amount of time was invested in making the situation between Department A and Department B in many ways worse.
Nobody knows if Department A’s aggression or Department B’s uncooperative-ness was the main cause. Regardless, it’s quite clear that the lack of cooperation resulted in huge loss of opportunity for the business.
Instead of innovating on core business opportunities, Department A spends half of its time developing and maintaining a technical wall so that they don’t have to interact with Department B. And still the customer experience is poor.
I’m sure you’ll have seen similar situations in many, if not every, large system you’ve worked on. Maybe you’ve been part of the problem, I’m sure I have at some point.
We should stop and frequently ask ourselves: Are we trying to architect around organisational dysfunctions? Are we exacerbating the problem?
How to Prevent Dysfunction-mirrored-complexity?
I won’t claim that preventing organisational dysfunctions from becoming technical complexity is an easy problem to solve. It’s a problem that touches on every layer in the organisation.
Firstly, incentives need to be aligned. If the organisation does not have a clear portfolio-level focus and teams are being stretched in many different directions, they won’t be able to satisfy those who depend on them. They need to know where to focus their efforts.
As always, the problem relates to how the teams are financed. If Department B isn’t given the budget to hire enough people to support all of the teams that depend on their services, there are clearly going to be problems.
For example, if Department A is given a budget to develop a suite of new website features, but Department B has not been given budget to implement the API changes, the organisation has designed a massive conflict in the system which cannot end well, and will probably reduce the productivity of all teams as they become locked in political battles.
Within each team, every individual must strive to collaborate on what is best for the overall software architecture and the business in general. Meaning more discussion, empathy, and trying to find a deal that works for all parties.
We can only expect people to behave that way if leaders create an environment which incentivises them to behave that way. Where frequent and respectful collaboration are ingrained in the culture. Where cross-team success is rewarded and individual goals do not conflict. Where blame is shared and seen as a collective problem and where teams are free to self-organise to resolve them.
A good software architecture which drives business innovation is an artefact of a well-designed sociotechnical system. The role of engineering leadership in aligning, empowering, and incentivising teams is crucial to the success of your IT-powered organisation.
Don’t hesitate to contact me if you’re looking for engineering leadership or domain-driven design expertise, consulting, or training.
