Scanning for secrets

Projects, even of the open-source variety, sometimes have secrets that need to be maintained. They can range from things like signing keys, which are (or should be) securely stored away from the project's code, to credentials and tokens for access to various web-based services, such as cloud-hosting services or the Python Package Index (PyPI). These credentials are sometimes needed by instances of the running code, and some others benefit from being stored "near" the code, but these types of credentials are not meant to be distributed outside of the project. They can sometimes mistakenly be added to a public repository, however, which is a slip that attackers are most definitely on the lookout for. The big repository-hosting services like GitHub and GitLab are well-placed to scan for these kinds of secrets being committed to project repositories—and they do.

Source-code repositories represent something of an attractive nuisance for storing this kind of information; project developers need the information close to hand and, obviously, the Git repository qualifies. But there are a few problems with that, of course. Those secrets are only meant to be used by the project itself, so publicizing them may violate the terms of service for a web service (e.g. Twitter or Google Maps) or, far worse, allow using the project's cloud infrastructure to mine cryptocurrency or allow anyone to publish code as if it came from the project itself. Also, once secrets get committed and pushed to the public repository, they become part of the immutable history of the repository. Undoing that is difficult and doesn't actually put the toothpaste back in the tube; anyone who cloned or pulled from the repository before it gets scrubbed still has the secret information.

Once a project recognizes that it has inadvertently released a secret via its source-code repository, it needs to have the issuer revoke the credential and, presumably issue a new one. But there may be a lengthy window of time before the mistake is noticed; even if it is noticed quickly, it may take some time to get the issuer to revoke the secret. All of that is best avoided, if possible.

Over the years, there have been various problems that stemmed from credentials being committed to Git repositories and published on GitHub. An article from five years ago talks about a data breach at Uber using Amazon Web Services (AWS) credentials that were mistakenly committed at GitHub; a much larger, later breach used stolen credentials to access a private GitHub repository that had additional AWS tokens. The article also points to a Detectify blog post describing how the company found Slack tokens by scanning GitHub repositories; these kinds of problems go further back than that, of course. A 2019 paper [PDF] shows that the problem has not really abated, which is no real surprise.

GitHub has been scanning for secrets since 2015; it began by looking for its own OAuth tokens. In 2018, the company expanded its scanning to look for other types of tokens and credentials:

Since April, we’ve worked with cloud service providers in private beta to scan all changes to public repositories and public Gists for credentials (GitHub doesn’t scan private code). Each candidate credential is sent to the provider, including some basic metadata such as the repository name and the commit that introduced the credential. The provider can then validate the credential and decide if the credential should be revoked depending on the associated risks to the user or provider. Either way, the provider typically contacts the owner of the credential, letting them know what occurred and what action was taken.

These days, GitHub has a long list of secret types that it scans for, which are listed in its secret-scanning documentation. When it finds matches in new commits (or in the history of newly added repositories), it contacts the credential issuer via an automated HTTP POST to an issuer-supplied URL; the issuer can check the validity of the secret and determine what actions to take. Those could include revocation, notification of the owner of the secret, and possibly issuing a replacement secret.

GitHub actively solicits service providers to join the program. In order to do so, they need to set up an endpoint to receive the HTTP POST and provide GitHub with a regular expression to be used to look for matches. In order to eliminate attackers misusing the scanning-message URL, the messages sent to the endpoint are signed with a GitHub key that can (should) be verified before processing the "secret revealed" POST.

A recent note on the GitHub blog announced the addition of PyPI tokens to the secret-scanning feature. GitHub and PyPI teamed up to help protect projects from these kinds of mistakes:

From today, GitHub will scan every commit to a public repository for exposed PyPI API tokens. We will forward any tokens we find to PyPI, who will automatically disable them and notify their owners. The end-to-end process takes just a few seconds.

GitHub is not alone; GitLab also added secret scanning as part of its 11.9 release in 2019. Since much of the code that underlies GitLab is open source, the code for the "secret detection" feature is available in a GitLab repository. GitLab is an "open core" project, so much of its code is available, unlike GitHub, which is proprietary. So far, at least, it would not appear that the fully open-source repository-hosting service SourceHut has implemented a similar feature.

The GitLab scanner is based on the Gitleaks tool, which will scan a Git repository for various regular expressions stored in a TOML configuration file. It is written in Go and can be run in a number of different ways, including on local files that have not yet been committed. Doing so regularly could potentially prevent the secrets from ever getting committed at all, of course.

The GitLab scanning documentation has a list of what kinds of secrets it looks for, which is shorter than GitHub's list, but does include some different types of secrets. GitLab's scanning looks for things like SSH and PGP private keys, passwords in URLs, and US Social Security numbers. The gitleaks.toml file shows a few more scanning targets that have not yet made it onto the list, including PyPI upload tokens.

It is better, of course, if secrets never actually make it into a repository at all. Second best would be to catch them on the committer's system before they have pushed their changes to the central repository; it may be somewhat painful to do, but the offending commit(s) can be completely removed from the history at that point via a rebase operation. Either of those require some kind of local scanning (perhaps with Gitleaks) that gets run as part of the development process. Having a backstop at the repository-hosting service, though, undoubtedly helps give projects some peace of mind.