The Power of the Architecture-driven Organisation

Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations
– Melvin Conway

Project Positron: A Lack of Business Engagement

As an example, imagine Project Positron initiated by its business project sponsor as a modern compute platform for use by other parts of the business. The development team is excited and engineering is going swimmingly.

The only issue is that during each biweekly retrospective a few team members express concern that they don’t have a clear understanding of the use case. User engagement and acceptance testing is talked about, but it never really materialises. Maybe the product owner gives assurances that he or she will take it up with other parts of the Business. Everything else is going well though, so this does not receive a whole lot of attention and with delivery pressure and other priorities during the sprint the follow-up quickly gets delayed to the next retrospective.

Such a lack of engagement doesn’t happen all the time, thankfully, but I’ve nevertheless seen it more often than I’d like to. The reasons are never simple. They range from other projects taking higher priority – a situation which can be addressed – to the project sponsor not achieving sufficient buy-in from other stakeholders in the business – a situation which is more serious. Either way, even if the engineering delivery is first rate, the resulting uptake of Project Positron, once delivered, will still be lower than expected because ready and willing users are not available. In the worst case it could be sidelined and mothballed, or scrapped entirely.

Engineers in an agile development team generally do not have much control over this scenario, but as a consultant it is something I would definitely want to highlight in order to give Project Positron the best chance of success in its organisation. We should not look at the success of a project only in terms of its engineering deliverables, but also in terms of its larger context in the organisation. If the business isn’t engaged with the solution, it may be a sign of organisational blockers that need removing, or it may be a genuine lack of broader demand.

Project Datatron: Architecture-driven Collaboration

This isn’t the only organisational problem an engineering consultant may encounter. A more nuanced case arose during a recent client engagement.

We were asked to iron out some of the creases in a new Data Producer platform involving one of the Data Consumer pipelines (let’s call it Project Datatron). At first it looked like a fairly straightforward data engineering problem. We were introduced to the relevant Consumer, namely the Business Data team. Their main requirement to integrate with Project Datatron’s new Platform was articulated as collecting data from the relevant data stream and making the data available as a view to their business users, either via a Data Lake or a Data Warehouse. They were having difficulty deciding on the best of the two options, and needed some help.

On the surface the problem seemed to be clearly defined: data from a stream needed to be transferred to a data repository. What could be simpler? In reality it was a bit more complicated. Hidden below the surface was a clash of paradigms: an Online streaming paradigm on the Producer side was now connected to a (previously Offline) batch paradigm on the Consumer side.

This resulted in various design discussions about files and creating a query layer on top of the file system (the Data Lake option), or alternatively batching data into the Data Warehouse using familiar ETL tools. It was perfectly understandable that the Business Data team would attempt to design the solution according to their existing toolset and experience, but it was becoming clearer that there were fundamental misunderstandings about the purpose of Project Datatron and the overall Data Platform architecture. This in turn was exacerbated by a lack of communication between the Producer and Consumer teams as well as a lack of end-to-end ownership of the architecture. There was no holistic overview, no shared domain language or any common forum for discussion.

There were obviously communication issues, but it was more than that. In many ways it was a textbook example of Conway’s Law (quoted at the top of this post). Until recently the Business Data team had been a part of the Business Department. They had become part of the IT Department only a few months before and their communication patterns, and even physical seating location, were still predominantly with and alongside their business customers within the Business Department, rather than with engineering teams in the IT Department. This was good from the point of view of their primary customers, but it meant that their engineering perspective was isolated from the rest of IT, and in fact reflected a set of legacy practices.

One could argue that this isn’t really an issue. After all, what does it matter that there are separate paradigms and legacy practices when there is an agreed handoff acting as the interface? Isn’t the most important thing that the system works as expected as long as each team is able to maintain the part of the overall system they are responsible for? The answer, of course, is that it depends.

I usually find it to be a good guideline to keep asking myself these two questions during the course of a project engagement:

1. What is the problem we are trying to solve?
2. What is the value to the organisation?

In the case of Project Datatron the overarching problem to be solved could be described as how to make the relevant upstream data available for downstream consumption by the Business Data team’s business users (data scientists, business intelligence analysts). This part was readily understood.

The value to the organisation, however, was less obvious. The Business Data team’s interpretation may have been that the value to the organisation lay in what the downstream business users accomplished with the data (producing business intelligence and actionable insights).

At this point we decided to run it past our engagement sponsor, who also happened to be the CTO of the organisation. The CTO saw things differently, envisaging Project Datatron as an opportunity to:

1. Empower teams to use new technologies and adopt best practices
2. Create resiliency by adopting common standards and technologies

In other words, the real value to the organisation lay in empowering engineering teams and creating a Community of Practice. It was a forward-looking vision that went beyond business as usual and the way things were currently being done.

This finally helped to clarify the nature of the problem. The Business Data team’s existing practices were out of sync with the rest of the IT engineering teams, and lacked more recent engineering practices such as automated builds, tests and deployments. Project Datatron offered an opportunity to bring the Business Data team into the fold.

Once we had uncovered this state of things the real solution presented itself more easily. The team developing the Producer side of the Data Platform had already adopted modern engineering practices: automated cloud services, software microservices and CI / CD pipelines. Compared to the Business Data team’s existing engineering approach our sponsor was spot on. It was an opportunity for them to adopt these shared practices and begin the move away from a strict ETL and batch paradigm supported by manual operations.

Our job as consultants then transitioned to helping them translate the larger IT organisation’s best practices into a design and implementation suitable to the Business Data team’s objectives. Most of this was new to the Business Data team, but they could now see the advantages of the approach, and, through pairing and technical mentoring, were willing to be guided through the implementation.

Conclusion

Not all stories of change have a happy ending, but when no technical solution seems to fit the problem it is often worth looking at it from an organisational perspective by asking “What is the value to the organisation?”

The example of Project Datatron shows that when an organisation is open to change a strong architectural vision can help individual teams take their first steps on a journey of transformation and modernisation.

This blog is written exclusively by the OpenCredo team. We do not accept external contributions.