Developer secrets keep leaking. Can we stop the flood?

These leaks, stemming from the accidental exposure of API keys, security tokens, and other credentials in code, have led to a surge in recent security incidents. The technology industry has known about this issue for almost a decade, and there have been countless attempts to solve it.

But a recent report by Ars Technica just how severe the issue has become. It reveals that even the most sophisticated and security-conscious engineering organizations are not immune to lapses, indicating that more work needs to be done to reverse the trend.

Almost 4,000 leaked secrets discovered

The Ars Technica story centers on a study by GitGuardian that looked at exposed secrets in just one popular open source repository used by developers. The results provide a rare look into the scale of the problem today. In total, the report found:

• 3,938 unique secrets exposed across all projects.
• 768 of those unique secrets were still in active use.
• 2,922 projects contained at least one unique exposed secret.

Any exposed credential can be problematic because they create vulnerabilities that can be exploited by attackers, similar to exposing your password. But many of the secrets discovered in this report were especially concerning given the broad degree of access they provide. Here are some examples of the exposed secrets that GitGuardian discovered:

• Azure Active Directory API Keys
• GitHub OAuth App Keys
• Database credentials for MongoDB, MySQL, and PostgreSQL
• SSH Credentials
• Twilio Master Credentials

The problem seems to be growing. GitGuardian found that over 1,000 unique secrets have been exposed via new projects and commits in the last year alone. The ongoing proliferation of exposed credentials serves as a stark reminder that industry practices need to change, and fast.

Are developers to blame?

It’s tempting to blame the problem on immature coding practices, as the Ars Technica story does. After all, shouldn’t developers know better than to use plaintext secrets in their code?

But complexity is the real culprit.

Most engineering teams are under immense pressure to deliver projects on a strict schedule. To make matters worse, they often have to use tools that lack sensible defaults or simple workflows. These two factors often lead to mistakes. Small errors like setting a private repo as public, pushing internal packages public, or forgetting to add a file to .gitignore can result in secrets getting exposed.

One often-repeated solution is to use a .env file stored separately from the shared code repository. This is a simple text file used to store variables, such as API keys and database passwords, which are programmatically loaded into an application’s environment at runtime.

But .env files come with their own security risks and challenges, including:

• Lack of encryption. The contents of the file are typically stored in plaintext, which means the information is immediately readable if accessed by an attacker.

• Exposure on local disks. Attackers can access these files if they gain access to a developer’s machine, giving them full access to development and production secrets.

• Easily committed. .env files have no protections if they are accidentally committed with other files to a shared public code repository (for example, when building a package).

It’s clear .env files have implications for security, but what about their impact on team productivity?

Help! I’m in environment variable hell trying to keep prod, preprod, ci and dev environment variables up to date in my #Django #python #docker_compose app. Any advice? Symptoms include broken deployments, failed ci runs, broken dev environments… @graham_knapp@hachyderm.io

The constant headache of manually configuring .env files has mostly been normalized and is considered accepted complexity by developers. For them, it’s just an assumed cost of modern software development. It’s a problem that will be familiar to engineering teams, who are often forced to pick between speed and security. The future has to be both: how can developers hit their deadlines and ship great code while also being secure?

Security must be easy and fast

Security and engineering leaders need to encourage their organizations to hit pause and consider the culture, tools and workflows required to secure their developer secrets. We’ve all seen in the news that a huge leak can have massive financial ramifications. So it’s better to take a long-term view and do the right thing now, rather than ignoring the issue and letting the inevitable (hack) happen.

Wondering where to start? The best and least expensive time to catch secrets is before the code is ever committed to a shared repository. Developers should be equipped with processes and tools that integrate security into the software development lifecycle from line 1.

But it’s also critical to provide solutions that are simple and easy to use. User experience (UX) design emerged as a common discipline at software companies by the early 2000s, but only recently has a similar focus on developer experience started to gain traction.

Working with secrets should be nearly effortless for developers: find them, replace them with a reference, save them into encrypted storage, and inject them into production. Too many tools are the opposite, and require developers to run local vault servers or to shuffle around .env files.

Summary

Our industry has a choice. We can choose to stem the flood of developer secrets leaking into open source and enterprise code repositories. Or we can choose to tolerate the problem for another decade.

I know which option I would pick. Developers should be able to work securely and efficiently without interrupting their workflows. Let’s work together and make that happen for them.

Andrew Stiefel

Product Marketing – Business & Channels