Software Architecture for Scale-ups

You’re part of a small start-up. There’s only one thing on your mind: shipping a product and rapidly finding product-market fit. Code is king! Software architecture? Sounds like a quaint concern from people who don’t ship. But, as it turns out, every system has an architecture. Whether or not it’s a good one, you’ll discover when the product takes off and you move from start-up to scale-up.

At Picnic, we’re scaling up for sure. First delivering groceries in just a single city in The Netherlands, then throughout the country, later expanding to Germany, and recently to France. We’ve got the product and found our market. Picnic’s systems and teams are growing at a corresponding pace. Around twenty product teams build and maintain services that grew out of the initial core system. We’ve more than doubled the size of our tech team in the past year. Still, software architecture across these services remained an ad-hoc and rather implicit concern.

From a team perspective architectural choices may seem optimal, but looking at the system as a whole, they’re often not. In this post we’ll explore how we improve this by giving software architecture a more prominent role. This post won’t be about particular architectural styles or technologies: that really depends on the products you’re building. What we can share is the process we devised to increase the quality of the architecture of our systems.

While it’s great that product teams build and deliver (micro-)services autonomously, several issues emerge when scaling up. Especially around bigger initiatives, with development spanning several products. More than once, key technical dependencies were discovered late into the process. This either leads to duplicate work or re-work. As another example, multiple teams shifted to more event-based architectures, but in different ways and on different timelines. More generally, insights on how to efficiently apply new technology remain siloed in teams.

Architecture Working Group

At this point, we decided that fully decentralized architecture decisions (or lack thereof) would hurt our mission in the long run. At the same time, the pendulum shouldn’t completely swing in the other direction either. Fully centralizing all architecture decisions is neither feasible nor desirable in a scale-up context. Gatekeeping behavior slows teams down and undermines autonomy.

Thus, the Architecture Working Group (AWG) was born: a small group of experienced engineers from the various product teams, who offer part of their time to address architectural questions and challenges for the whole tech team. The goal of the AWG is to empower individual product teams to make the right decisions. To this end, the group needs to have a rough overview of architecturally significant developments, and a process to gather input from, and provide feedback to the teams.

We settled on a lightweight process where product teams are invited to write Architecture Proposals for significant developments. Significant means it’s either a big project, or one that involves many teams and systems and therefore poses unique risks and challenges.

Architecture Proposals serve as input for a review process by the AWG. This way, the Architecture Working Group has the chance to weigh in on architecturally significant changes that are coming up. The following outcomes are achieved by introducing this process:

• Coordination: potential conflicts or opportunities across product teams can be spotted upfront, avoiding duplicate work.
• Quality: by providing holistic feedback on architecture proposals, we increase the uniformity and quality of the overall architecture.
• Documentation: important architectural decisions are documented through the Architecture Proposals, available to everyone.

Together with the teams the AWG identifies upcoming major developments, and determines which can benefit from a proposal and review. After a team finishes their proposal in Confluence (more on the format later), two members from the Architecture Working Group start reviewing. Feedback is provided directly on the proposal itself, and a feedback session is planned with the proposal writers. Usually all follow-up points are addressed by the product team, but if the feedback gives rise to larger changes to the proposal, another round of review may be scheduled.

Again, the goal of this process is not to block initiatives, but to help product teams ask and answer the right architectural questions. We want to foster a culture where it’s encouraged to design and discuss important decisions upfront.

Architecture Proposal Template

Thinking about architectural changes in such a deliberate way doesn’t come naturally to everyone. One of the tools the AWG developed is a template for teams to use. It’s a straightforward Confluence template asking the writer to at least address the following areas:

• Goals/Requirements: describe the business outcome of the proposed change. If this section cannot be clearly written, it’s impossible to judge the validity of the proposed architectural changes. Sometimes, proposals are put on hold until there’s enough clarity on the actual goals and requirements.
• Current solution: briefly describe the relevant parts of the current domain model, and provide a context diagram of the existing services and their interactions.
• Proposed solution: describe the change and its architectural impact, focusing on new services, API design, data modeling, integration with existing systems, and possibly new technology to be adopted. We provide guidance on when component diagrams or sequence diagrams are appropriate.
• Operational concerns: outline the load and performance targets, along with any security considerations for the change. Additionally, teams are expected to provide a roll-out plan if there are for example breaking API changes in the proposal, or other dependencies that need to be managed.
• Risks: architecture is always about trade-offs. Any risks identified during the architectural design phase should be documented, along with an impact assessment and mitigation strategy.

Each section contains pointers on what diagram types are best suited, and what level of abstraction is expected. All proposals (including feedback) are open for the whole tech team to see. This way, everyone can learn and keep up-to-date with other teams.

Fine-tuning the process

Since we’ve started the AWG and introduced the proposal process at the end of 2020, we’ve learned some lessons along the way.

Initially we aligned the process with our quarterly planning cycle. All tech leads would be invited to a joint meeting to uncover the topics requiring an architecture proposal. Proposals would then be completed on a strict timeline before the quarter starts, resulting in a clear go/no-go decision. Inadvertently, we turned the AWG into a sort of Change Advisory Board, adding stress and uncertainty to an already packed period around quarterly roadmap planning. So we changed that. Now, we don’t do big quarterly meetings. Instead, we organise individual check-ins from the AWG with product teams on a rolling basis to identify their needs and reflect on the current state of the team’s architecture. We want to be there to advise the teams on their crucial changes, not block or burden them at difficult moments.

Writing an architecture proposal is not easy. Having good examples, and sharing those with other teams really helps people to get started. We evolved the template based on feedback from the initial proposals. It also became clear that writing proposals often uncovered non-trivial issues with the goals and requirements of an initiative. However great the remainder of the proposal is, if the fundamentals aren’t right, it doesn’t matter. To counter this, we now ask teams to first iterate on the goals and requirements section until all stakeholders agree. Only then does the team write the remainder, designing the architecture in sufficient detail.

What’s next for us…

Introducing the Architecture Working Group with a lightweight proposal process works well for Picnic while scaling up. We don’t define a grand centralized architecture that every team should conform to. Instead, we want to put architectural thinking into the hearts and minds of all developers. Rather than limiting people’s creativity, the AWG helps assess whether architectural choices work out well in the broader Picnic context. It also brings teams together at a technical level to work on ambitious initiatives spanning multiple products.

We certainly haven’t reached our final state yet. With more of the product teams finding their way to the AWG, we’ve expanded the group to keep up. Transitioning from a quarterly ‘pull-model’ to a more self-service model (from the perspective of product teams) also happened recently. Finding the right balance on when to involve the AWG is an ongoing journey. So far we’ve benefited from many architecture proposals that lead to valuable insights upfront, increasing the quality of our systems. What’s your way of designing systems for the next big step?