Configuration management
Almost every project I’ve worked on has had some problem with configuration and secret management. There are a lot of problems, but in most cases, the reason was the same: the high-level strategy wasn’t created, so different groups of stakeholders tried to force the best solution for them. If you already know the group of stakeholders in your organization for config management, that’s good, and you can skip the next paragraph 🙂.
Stakeholders
Devs – the most obvious group. Their perspective is simple: great development experience. Simple “git push & run” is the best option. In such cases, they believe that simple files are the best, no matter how many secrets are in the project.
Ops – the second most obvious group. Of course, if your organization believes in “you build it, you ship it,” it can be the same guys as in the first group. But their goal is different. It is rather how to make sure that each environment has a proper configuration.
Production Ops – In large organizations or software houses, it is quite typical that for production, we have a totally different set of people involved, especially when you release your software rather rarely, like mobile apps, desktop apps, or even web apps. Not everyone deploys & releases 10 times a day.
Sec team – another group of stakeholders whose main responsibility is to make sure that our secrets are protected well enough. From a dev & ops perspective, their requirements look like a pain in the ass, but from the organization’s point of view, their requirements are an important protection layer.
The stakeholders are defined; now it is time to define types of configuration entries.
Types of configuration entries
Configuration entries can be split into the following groups:
Internal - something that is in our infrastructure or app. It is responsible for the components which are under our control. A few examples: Connection String to our database, URL to our service. But be careful; the last example works only for backend, because in desktop, mobile, or even SPA apps, it should be included in the next category.
External - something external that is not in our control. Again, a few examples: URL to weather service, API key to Google Maps.
Feature flags - something that changes our app behavior, for example: logging level, new GUI switch.
Above is only one dimension; the other one is the obvious: is it a secret or not. I’m sure that in most cases, we can agree that an API key or connection string is a secret, while a URL usually is not a secret at all. But the crucial word is “usually.” In some cases, even a URL is a secret, while a passwordless connection string is not at all 🙂. Moreover, depending on the software type, we cannot think about them in the same way. For example, no matter how much you try, you cannot protect an API key in a mobile app or a connection string in a desktop app. You can add some stuff, but it will be rather security by obscurity, not real protection.
In my perspective, there is also one more dimension: does it change in different environments? The most pragmatic example is logging level. We can change it, but unless we have an extreme problem, we don’t do that. The same goes with the convention over configuration or default values.
Dealing with secrets
As I wrote before, in desktop, mobile & frontend apps, you cannot protect secrets. The only way is to introduce authorization & authentication and make sure that the user can interact only with resources which belong to him/her.
In backend apps, there are two ways. The first one is simple: no secrets at all, using, for example, passwordless authentication, which is available in different forms depending on where you host your app. Of course, if we dig down, there are some passwords or certificates, but they are hidden from us, and the hosting environment makes sure that it is working like a charm. The second one is to introduce some kind of vault. If you have never heard about it, you can look at Hashicorp Vault, Azure KeyVault, AWS Secrets Manager, or Google KMS. At the end of the day, the security team is happy, but usually, the dev & ops teams have some (or a lot) problems.
Let’s start with the dev team. First of all, someone will use an argument: we have to use env variables, because according to the 12 Factor (see: https://12factor.net/config) that’s the best option. While of course, you use something that will convert values from vault to env vars in the running environment, it creates a tragic experience on dev machines, because setting up 10, 20, or 100 vars is a tragedy. Except that both options can have some security problems, our app needs to read configuration from at least 3 sources:
Environment variables aka env
Configuration file(s) aka file
Chosen vault solution aka vault
In some companies, the local environment doesn’t have access to the vault solution, but our app still needs logic to do that.
What about the order? If you load env -> file -> vault, the vault always wins, and there is no way to override something on the dev machine. The same goes with the file -> env -> vault.
If you decide to load env as the last one, you have only one option available: file -> vault -> env, because the vault cannot be the first one. It is an external service, and somewhere you need to put connection information. Such an approach gives you the flexibility to override values on the local machine, but it starts to be difficult to pass access to the key vault on the test & production environments.
It can be quite a challenge, right?
That’s why some teams decide to split configuration handling and load config in a different manner while debugging and while running applications.
Usually, another problem also appears on the radar. Let’s make a simple example: we have the URL of the service and the API key to use it. Should we treat them separately or in the same way? If so, we will put the URL in the env or config file and the API key in the vault. But does it make sense? In most teams that I worked with, the answer was no. On the other hand, it is quite useful. But at the end of the day, we are totally unhappy with manual secret management.
Git to the rescue
What if we could step back and store all secrets in the repositories, managing them the same way as code? Just use SaaC (Secrets as Code) or even CaaC (Configuration as Code). That idea is probably behind Mozilla SOPS (https://github.com/getsops/sops). Instead of plain values, we have encrypted values (see the video on https://github.com/getsops/sops to get the idea). Is it safe enough? Of course, it depends, but using a long enough key stored in a secure key vault seems good enough to me.
Does it solve all the problems? Of course not. It’s an idea we had a long time ago, but it fixes exactly one problem. What will be next? We will see.
Balancing security and usability
Wrapping up, managing configurations and secrets is a tricky task that requires us to consider what different stakeholders want. By sorting out configuration entries and using strategies like Secrets as Code (SaaC), we can make things more secure and easier to understand.
Even though there are challenges, especially with secret management in various environments, using existing tools and vault solutions can really help. The goal is to find a good balance between security and ease of use, so teams can work smoothly while keeping important info safe.
As technology keeps evolving, so will our methods, constantly improving to tackle new challenges. And that’s exactly what a staff engineer does –