man at a laptop

Rewrites and transitional architecture

Words By Zak Islam7 mins

Back in 2015 at a previous company I took over a product that was riddled with technical debt.

We managed between 70–100 high severity events on a good week. The system did not have basic protocols like rate limiting on the APIs, prioritisation of work, or defensive capabilities such as isolating noisy neighbours but it easily processed many millions of requests per second. In other words, it was a massive cluster f&^k under the hood but it worked. When we did our first assessment of the system, we honestly didn’t know where to start. The challenge ahead of us seemed unsurmountable. We knew a rewrite of the system was necessary but we’d have to earn the right to rewrite the system. One of the core engineering principles that we had was:

[…] Engineers are grateful to our predecessors. We appreciate the value of working systems and the lessons they embody. We understand that many problems are not essentially new.

Simply put, this means respect what came before. At the time I don’t think we really appreciated how this principle would shape how we would think as engineers and leaders as we progressed in our career.

Rewrites

The second-system effect (also known as second-system syndrome) is the tendency of small, elegant, and successful systems to be succeeded by over-engineered, bloated systems, due to inflated expectations and overconfidence.[1]

While the system was in a less than desirable state at the time, it didn’t need a rewrite; it just needed some love ❤.

Transitional architecture

Root Cause Distribution graph

Iteration 1

Iteration 2

Iteration 3

  1. A low latency data distribution protocol to report request-rate-by-customer from hosts across the fleet
  2. A semi-decentralised service that can make some decision based on heuristics
  3. A automated system to update request limits
  4. Oh, and this system had to be highly resilient and fault tolerant

The perfect system

The take aways

  1. Avoid the temptation to rewrite: On the surface, rewriting systems feel like the right thing to do to avoid the inherent complexities. When you appreciate the value of working systems and the lessons they embody rewrites are often less appealing. (Sometimes you have to rewrite a system. That’s ok. When you have to rewrite it, stick to replicating what works — stick with what works and make the smallest amount of change necessary)
  2. Transitional architecture: North stars are good to have but often they should be left as just that — a vision. An iterative or transitional architecture can help eliminate unnecessary complexity and deliver a solution that is good enough.

Credits

Words By Zak Islam

7 mins

You might also like these

Jumpstart your corner of the internet today

Aboriginal Flag
Torres Strait Islander Flag
We acknowledge the Traditional Custodians of the land on which our office stands, The Wurundjeri people of the Kulin Nation, and pay our respects to Elders past, present and emerging.

Linktree Pty Ltd, 223 Liverpool St, Gadigal (Darlinghurst) NSW 2010