Secrets management for business systems

A developer commits a configuration file containing a database password. They notice within the hour, delete the line, and commit again. The repository now looks clean, the incident feels closed, and everybody moves on.

Nothing has been fixed. The password is still valid, still in the repository's history, and still retrievable by anyone with access to that history — including anyone who cloned or forked it in the interval. The business has not resolved an exposure; it has stopped being able to see it. And because the credential still works, nothing will fail to remind anyone.

The engineering position that follows is uncomfortable and worth stating plainly: a secret that has been exposed is compromised, and the only remedy is to replace it. Everything else in this article follows from that, including the awkward discovery most businesses make at this point — that they cannot replace it, because nobody knows what would break.

What counts as a secret

Broader than most inventories assume. OWASP's secrets management guidance describes the category as API keys, database credentials, identity and access management permissions, SSH keys and certificates, among others. In a typical growing business that also covers SMTP credentials, payment gateway keys, cloud provider access keys, webhook signing secrets, licence keys, service account passwords, and the credentials embedded in a workflow automation platform.

Two categories behave differently enough that treating them as one problem produces a poor answer to both.

The distinction matters most at offboarding. Disabling a person's account handles their human secrets. It does nothing about a machine secret they created, knew, or embedded in something four years ago, which is why integrations authenticated as a departing employee fail on their last day — a failure mode covered from the integration side in the API integration article.

Deleting the line does not help

This is the single most important thing to understand about a leaked credential, and the platform vendors document it clearly enough that there is no need to argue it.

GitHub's guidance on removing sensitive data states that if you only rewrite your history and force push it, the commits with sensitive data may still be accessible elsewhere: in any clones or forks of your repository, directly via their SHA-1 hashes in cached views on GitHub, and through any pull requests that reference them. On other people's copies it is blunter still — you cannot remove sensitive data from other users' clones of your repository, and will have to send them the instructions.

The instruction that follows is the one businesses skip. GitHub states that if the sensitive data you need to remove is a secret such as a password, token or credential, as is often the case, then as a first step you need to revoke or rotate that secret. It goes further, and this is the part that reframes the whole incident: going through the extra steps to rewrite the history and remove the secret may not be warranted.

Read that carefully, because it inverts the instinctive response. The platform's own position is that rotating the credential is the remedy, and cleaning the history is optional housekeeping that may not be worth doing. Businesses reliably do the opposite: they spend an afternoon rewriting history, leave the credential in place because rotating it is disruptive, and record the incident as closed.

The provider may find out before you do

A detail that surprises businesses during their first leak, and one with a direct operational consequence.

GitHub documents that its secret scanning identifies hardcoded credentials including API keys, passwords, tokens and other known secret types, and that it scans beyond code into descriptions and comments in issues. Where the credential belongs to a partner provider, it states that when a partner secret is detected, they notify the provider so they can take action, such as revoking the credential. It also notes that partner secrets are reported directly to the provider and are not displayed in your repository alerts.

The practical sequence is therefore possible and does happen: a cloud or payment provider key is committed, the provider is notified automatically, the provider revokes it, and an integration stops working. The engineering team investigates an outage with no obvious cause, because the alert that would have explained it was never shown to them. Understanding that this can happen turns a confusing incident into a five-minute diagnosis.

It also means the window between exposure and consequence can be very short — shorter than a weekly review, shorter than a sprint, and considerably shorter than the time it usually takes to find out what a credential is used for.

Rotation is the control; storage is the hygiene

Most secrets management projects begin and end with storage. A vault is selected, credentials are moved into it, the project is closed, and nothing is ever rotated. The business now has better hygiene and exactly the same exposure to a leak, because the response to a leak was never the storage location.

OWASP's guidance is direct on the point: you should regularly rotate secrets so that any stolen credentials will only work for a short time. It is equally direct about how this should happen, warning that manual maintenance not only increases the risk of leakage, it also introduces the risk of human errors while maintaining the secret, and recommending automated rotation through pipelines, dynamic secrets and scheduled processes.

For credentials that are genuinely cryptographic keys rather than passwords, the principled treatment of lifetimes is NIST SP 800-57 Part 1 Revision 5, Recommendation for Key Management: Part 1 – General, finalised in May 2020, which sets out guidance on the protection each type of key requires and the functions involved in managing it.

On rotation intervals

Published advice on this is confident and largely unsourced — weekly for high-privilege credentials, monthly for low-privilege ones, and similar prescriptions that vary between vendors and trace to nothing. We are not going to add another number to that pile.

The more useful framing is that the interval matters far less than the capability. A business that can rotate any credential within an hour, on demand, is in a strong position whatever its schedule, because it can respond to an actual leak. A business that rotates on a calendar but takes three days and a change window to do it unplanned is in a weak one. Build the capability first; the interval is a policy decision that becomes easy once rotation is routine.

You cannot rotate what you cannot inventory

Here is why rotation is rare despite being universally recommended. Replacing a credential means updating everything that uses it, simultaneously enough that nothing breaks. That requires knowing what uses it — and in most growing businesses, nobody does.

The same database password is in an application's configuration, a reporting script on someone's machine, a backup job, a workflow automation platform, and a monitoring check that somebody set up two years ago. Rotating it breaks the two that nobody remembered, at an unpredictable moment, which is precisely why nobody rotates it.

This is the same problem OWASP names as API9, Improper Inventory Management, in its API Security Top 10, and the same register described in the API integration article does most of the work. For secrets specifically, five things need recording.

• What the secret is and what system it authenticates to.
• Every consumer that uses it — applications, scripts, scheduled jobs, automation workflows, monitoring checks.
• Where it is stored, and everywhere else a copy exists.
• What it is scoped to: read-only, write, administrative, or unrestricted.
• Who owns it by name, and when it was last rotated.

The second item is the one that makes rotation possible and the one most often incomplete. A useful test of the register's accuracy is simply to rotate something during business hours and see what complains — an exercise covered below, and one that is far cheaper to run deliberately than to discover during an incident.

Where secrets should live

There is a progression here, and most businesses are somewhere in the middle believing they are at the end.

The third row deserves emphasis because it is where a lot of businesses stop, having concluded the problem is solved. Environment variables keep secrets out of version control, which is a real and worthwhile gain. They do not provide access control, audit, versioning or rotation without redeployment, and a secret in an environment variable is readable by anything running in that environment — including, in an incident, whatever the attacker is running.

Two related placements are worth naming because they are so common and so easily overlooked. Credentials pasted directly into an automation platform's workflow steps end up in exported workflow definitions and version history, a failure mode covered in the workflow automation article. And credentials in a container image are in every copy of that image, in every registry it was pushed to.

Designing so that rotation is routine

Rotation is disruptive when the system assumes exactly one valid credential at a time. Designing that assumption out is what turns rotation from an incident into a task.

1. 1

   Support two valid credentials at once

   Where the provider allows multiple active keys, the sequence is: issue the new one, deploy it to every consumer, verify traffic has moved, then revoke the old one. No consumer is ever without a working credential, so nothing has to happen simultaneously.

2. 2

   Prefer short-lived credentials where they exist

   A token that expires in an hour and is renewed automatically limits the value of a leak without anyone scheduling anything. This is the mechanism behind OAuth refresh flows and cloud instance roles, and where a provider offers it, it removes most of the problem rather than managing it.

3. 3

   Give each consumer its own credential

   One shared key across six systems means rotating it affects all six. Separate credentials mean a leak is contained to one consumer and can be replaced without coordinating anything, and the audit trail identifies which system did what.

4. 4

   Rehearse a rotation deliberately

   Pick a credential, rotate it on a Tuesday morning with everyone available, and record what broke. What breaks is your inventory gap, discovered at the cheapest possible moment. An untested rotation procedure is a hypothesis, in exactly the way an untested backup is.

5. 5

   Automate the mechanism before the schedule

   A rotation that requires a person to log into four systems will be skipped under pressure. OWASP's position is that manual maintenance increases both leakage risk and the risk of human error during maintenance.

The fourth item is the highest-value exercise in this article. Businesses that have rehearsed a rotation respond to a leak in an afternoon. Businesses that have not spend the first day establishing what the credential is connected to, while it remains valid.

Scope determines what a leak costs

The other half of blast radius is what the credential is permitted to do. OWASP applies the principle directly, noting that engineers should not have access to all secrets in the secrets management system and that least privilege should be applied.

The same reasoning extends to the credentials themselves. A leaked key scoped to read one storage bucket is an unpleasant afternoon. The same key with unrestricted administrative access to the account is a different category of event, and the difference was decided when the key was created — usually by whoever needed something to work quickly and granted the broadest permission that made the error message go away.

Two habits do most of the work: create credentials with the narrowest permission that satisfies the task, and separate credentials by environment so that a development key cannot touch production data. Neither costs anything at creation time and both are expensive to retrofit across an estate.

Auditing and detection

A secret that is used somewhere nobody expected is the earliest available signal of a compromise, and it is only visible if usage is recorded.

OWASP sets out what should be audited: who requested a secret, whether the request was approved, when the secret was used, expiration events, and any authentication or authorisation errors. That last category is the most operationally useful and the most often ignored — repeated authentication failures against a service credential are either a misconfiguration you want to know about or someone attempting to use a credential they should not have.

On the prevention side, automated scanning of repositories for committed credentials is now standard and worth enabling wherever code is hosted. It does not remove the need for rotation when something is found; it shortens the interval between exposure and discovery, which is the variable that determines how much the exposure costs.

Responding to a leak

The order matters more than the individual steps, and the common failure is doing them in reverse.

1. 1

   Rotate first, before investigating

   The credential is compromised from the moment of exposure. Replacing it stops the exposure continuing while you work out what happened. Investigating first leaves a valid credential in the open for the duration of the investigation.

2. 2

   Then establish the window and the scope

   When was it exposed, who could have seen it, and what was it permitted to do. Provider audit logs are the evidence; this is where the scope decision made at creation time pays off or does not.

3. 3

   Check for use you did not authorise

   Look for activity from unfamiliar addresses, at unusual times, or of a kind the credential's legitimate consumers never perform.

4. 4

   Clean the history only if it is warranted

   GitHub's own guidance is that once the secret is rotated, rewriting history may not be warranted. Judge it on whether anything else in that history is sensitive, rather than treating it as the mandatory step.

5. 5

   Record what the inventory got wrong

   If rotating broke something unexpected, the register was incomplete. Fixing that is the durable outcome of the incident and the reason the next one will be cheaper.

Common mistakes

A representative scenario

A composite of situations we see repeatedly; no client detail is included.

A cloud provider access key is committed to a private repository in a configuration file. Within a short period the provider disables it automatically, having been notified. An integration that moves files between systems stops working. The team spends most of a day investigating an outage with no error that explains it, because the notification went to the provider rather than into their own alerts.

The recovery was straightforward once the cause was understood. What the incident revealed was more useful: the same key was in four places nobody had listed, it was scoped to full account access because that was fastest during the original build, and it had never been rotated since creation.

The remediation was sequenced by what reduces exposure fastest. Every consumer received its own narrowly scoped credential, so a future leak would be contained to one system. A register recorded consumers, storage locations, scope and owner. Repository scanning was enabled to shorten the gap between exposure and discovery. And a rotation was rehearsed deliberately on a quiet morning — which found one more consumer that the register had missed, at a cost of twenty minutes rather than a day.

Implementation checklist

• No credential is present in source code or in any file committed to version control.
• Human secrets are in a password manager with individual access; machine secrets are in a secret store the application reads at runtime.
• No machine credential is owned by a named individual; service accounts are used instead.
• Each consumer has its own credential rather than sharing one across systems.
• Every credential is scoped to the narrowest permission that satisfies its task, and separated by environment.
• A register records, for each secret: what it authenticates to, every consumer, every storage location, its scope, its owner and when it was last rotated.
• The provider supports two valid credentials at once, and the rotation sequence issues before revoking.
• Short-lived credentials are used wherever the provider offers them.
• A rotation has been rehearsed deliberately during business hours, and what broke has been added to the register.
• Repository secret scanning is enabled wherever code is hosted.
• Authentication failures against service credentials are logged and alerted on.
• The leak response is written down and starts with rotation rather than investigation.

Frequently asked questions

Conclusion

Secrets management is generally sold as a storage decision and is more accurately a preparedness one. The question that determines what a leak costs is not where credentials are kept but whether the business can replace one this afternoon without breaking something it forgot about.

That capability is built from unglamorous parts: a credential per consumer rather than one shared across many, the narrowest scope that satisfies each task, a register that records what uses what, and one rehearsed rotation to prove the register is accurate. None of it requires a product. All of it is what separates a leak that is handled in an afternoon from one that starts with a day spent working out what the credential was connected to, while it is still valid.