We should be boring
A few years ago, during a team meeting, we tackled an issue with the database. To be precise, we were tackling tens of problems that we have now, and we can easily foresee tens of new ones. The problems ranged from the performance of operations over the database, like schema change, through how we are using the database from application code to our database going out of support, and we will not be able to have it on the cloud provider in a few months.
Like with every problem, there were many solutions or steps we could take to postpone the inventible—we are reaching all possible limits of our current database setup, and we need to shift, and that shift will be costly. Not doing it will not allow us to grow and onboard new customers onto our SaaS product. Committing to change will influence the whole organisation, which will be a lengthy process.
Like on triage, we discussed options and pros and cons in our context. We spanned that discussion across a few meetings, with spikes validating ideas and propositions. That’s what we like to do. There is a new shiny toy that promises scalability and performance without overhead. Let’s go :)
We were already after two spikes that yielded mixed results:
To benefit from one, we need to make all changes to how we use the database from an application perspective—we need to drop FK, fix n+1 queries, fix querying for the same information ten times during a request, etc.
The other showed that we would need to pay 100-fold of the current database cost to maintain the same performance but eliminate all other issues we have.
Both spikes showed us that we don't know what we don't know, and making them even somehow work was problematic. Finding help in the community or from outside was impossible - finding a technology specialist of technology that is one or two years old is hard.
During one of the summarization sessions, one of my colleagues said, "We just should be boring".
Boring? What is it? And why?
He explained that he meant that we should not look at other shiny new tools as we can't predict problems that we will have with them. We have a strong team handling all edge cases with our current database; they know how it works, what to expect, and, most importantly, how to fix it. They are doing this on a daily basis with a database that was already proclaimed dead a few times before. The new product only guarantees that it will fix some of our usage problems with DB, which will only add to the complexity of migration.
And he was right. After spending a few months on spikes and gathering issues and possible solutions, we finally saw the end.
Thanks to that comment and the linked article by Dan McKinley Choose Boring Technology, from which he used that phrase, we unanimously decided to stay in our current database and tackle each issue separately. That gave us focus; we were able to shape a road map of actions that we needed to do, which, in the end, would allow us to make educated decisions about change if required.
Looking from a time perspective, it was a good decision. We have reduced the cognitive load for the developers and everyone else, allowing them to concentrate on delivering customer features.
In this situation, we've settled on:
Stability: we know how our database works and how to make it stable;
Maintainability: we know how to tackle problems of taking care of our database;
Specialist availability: we could reach a database expert who probably already solved that issue with our database;
Cost: we knew the cost of running this database at scale;
Expertise level: we could talk in detail about how to improve our code as we know how our database works and reacts; we think we know where to look for the issue;
It's easy to add new technology and change things around, but it's hard to predict what this will bring us. One of the team members introduced Lua, and then that person left. We now had two problems: a non-working product and a lack of Lua developers to fix it (back then, it was hard to find, too).
Ok, that situation was predictable. So, let's look at a different example: introducing tRPC to a project. What could go wrong? Your frontend devs love it’s type-safe on the client side out of the box, 🤯 (yes, I know we can have the same thing with server actions...). But then, after going to staging/production, the operation team is having a bad day trying to configure WAF to pass that serialised JSON in the query string generated by tRPC and not to fire too many errors (yes, WAF usually run rules using regex..., but who would have thought? And on dev, who is using WAF?).
Closing
That said, going boring does not mean going with an old version or niche/old software. Going boring means selecting technology in the current context of the organisation and product, such as skill matrix, operation, maintainability, extension, and sometimes even market situation. Then, document that decision with a list of trade-offs.
It's not bad to use a new product, language, or framework; just think about the consequences of introducing it. It does not matter how senior you are in the organisation; it's a matter of trade-offs and discussion. Sometimes, all you need to do is talk to humans. Sometimes, all you need to do to improve performance is update the Java version and not necessarily migrate to Rust.
As an Individual Contributor to the organisation, you will face many situations where your decision will impact the organisation and other teams. Don't take that lightly. Boring is good.