The Unspoken Vulnerability of JWTs

JSON Web Tokens (JWTs) are the new thing. Blog after blog and book after book tell you how to generate them and use them to authorize access to web services. But there is one little detail that everyone is leaving out: it is much harder to secure a server that generates JWTs than a server that generates session IDs. This is because the JWT signing key must be protected, whereas there is little need to secure session IDs, and session IDs are easily secured by hashing, anyway. As a consequence, the push to use JWTs for local authentication is making sites more vulnerable.

Here you might dismiss me as a random loon for questioning the JWT love, but I do have years of experience professionally evaluating systems to assess and document their security. I've performed IV&Vs for NSA, evaluated NetWare's file system for a TNI Class C2 rating, and developed a reputation for being able to quickly identify security flaws in large software systems. Mind you, that was COMPUSEC, not INFOSEC, and that was years ago, but I like to think some skill remains.

The Vulnerability Decision

The security decisions we make determine the options that are available to an attacker. Our decisions can even affect the options that are available to an attacker who breaches the server. Consider how an attacker has more options obtaining cleartext passwords than obtaining hashed passwords.

This discussion concerns the decision to originate JWT tokens versus the decision to use hashed session IDs. If you use JWT tokens issued by an outside authority, the discussion applies to that outside authority and not to you, unless that authority happens to use symmetric signing keys (i.e. an HMAC algorithm, which verifies the signature with the same key used to make the signature).

The situation is analogous to storing passwords cleartext versus storing them hashed. JWT tokens are signed with keys, and to sign the key, the server must use the key in plaintext. Unless the server takes special measures to compartmentalize the key, an attacker who breaches the server can obtain the key. Having the key, the attacker can forge JWT tokens and access the server as if they were authenticated users, without the server knowing any better.

Here are the options available to an attacker who breaches a server with an exposed JWT signing key:
  • Use the key to spoof arbitrary users
  • Read data from the server
  • Write to the server and risk detection through inconsistencies they may introduce, if they happen to also get the necessary write permissions
On the other hand, an attacker who breaches a server that only stores hashed session IDs has the following options:
  • Read data from the server
  • Write to the server and risk detection through inconsistencies they may introduce, if they happen to also get the necessary write permissions
Hashed session IDs cannot be used to quietly spoof the server. The attacker thus has one less option.

Discussions with others show that it's a matter of option whether making it easy for an attacker to spoof users matters when the attacker has already breached the server (see discussion below), but it seems uncontroversial that hashed session IDs give attackers fewer exploit options.

A Convenient Assumption

It seems that many developers believe that if a server is ever breached, then the attacker is able and willing to do anything with the server. Under this assumption, there is no need to protect JWT signing keys because the attacker can add sessions, change keys, or obviate session and signature checks and directly manipulate software and data. This assumption gives developers permission to do whatever is most convenient with the keys, because all is apparently lost with a breach anyway.

I was unable to find statistics indicating what percentage of breaches result in attackers modifying software or data, so until such statistics show, let's look at this logically. First, accounts can generally read more data than they can write, so if attacks track account distribution, attackers will generally be able to read more than they can write. Second, software is fickle, so attackers who want to remain undetected are ill-advised to attempt to modify software or data, particularly when they don't have access to the source code. Attackers are better off quietly hijacking accounts that can do what they want done. Finally, if the breach is of an authentication system that is isolated from client services, the attacker has not yet breached client data. If this system does not expose credentials (e.g. passwords or signing keys), the attacker is forced to either patch the software or change credentials, one of which is risky and the other of which users or database triggers can detect.

In short, it's neither clear that breaches normally give attackers unrestrained access nor clear that attackers who wish to remain undiscovered would take full advantage of unrestrained access anyway. Besides, if after a breach the attacker finds a key that provides easy, unchallenged access to APIs as any user, why would the attacker risk directly mucking with application behavior or data? It seems that guarding against attacks that only steal data might be valuable after all.

The Vulnerable Scenario

Consider this scenario. You have deployed a server that authenticates users by username and password. Your server generates a JWT on successful authentication, signing the JWT with either a symmetric or asymmetric key. The client hands the JWT to web services to authorize access. Each web service verifies the signature, and if the JWT is valid and unexpired, allows the client to access the service as the user identified in the JWT payload. The payload may also specify the permissions granted to the user (e.g. "scopes"), but our scenario does not require that permissions be present.

Now suppose an attacker breaches the authenticating server and we don't learn about the breach right away. This risk always exists. There are many different ways to breach, but we do know that it is common for the attacker to gain access to sensitive data, such as usernames and passwords. We're not particularly worried about passwords though, because our server has hashed them.

But the server has not hashed the signing key. It cannot hash the signing key, because the original key itself must be used for signing. Without taking measures that exceed what we normally do to protect passwords, the attacker may readily acquire the signing key. Maybe we thought to protect the key and split it between a file and the database, or maybe we stored the key encrypted in the file system. In these cases, both key halves might still be as readable as the password hashes, and the key for decrypting the encrypted key might be as readable as the encrypted key. Simple measures for securing the key are likely insufficient. In our scenario, insufficient measures were taken.

Specifically, we assume that the attacker has acquired the JWT signing key.

Our scenario is taking advantage of the most touted benefit of JWTs: by trusting the payload claims of a JWT, each server is spared from having to hit the database to authorize each request. When load balancing, the servers of a cluster are also spared from having to cache user data and from having to dedicate themselves to clients for "sticky" sessions. If we were not trusting the payload claims, we would be ignoring them and treating the JWT as an opaque token or session ID.

Exploiting the Scenario

Once the attacker has a signing key, the attacker can forge unexpired JWTs forever—or at least until the breach is detected and keys are changed. Having seen the data files, the attacker probably has user IDs and permission codes and so can put together functional JWTs. The attacker may therefore be able to access any web service as anyone, and no one will know until after the damage has been done.

It's possible to detect attacks by confirming IP addresses and device names, but servers that do so will not be benefiting from the touted scaling benefit of JWTs, because these confirmations require hitting the database. To scale, servers must trust the payloads of properly signed JWTs. Besides, the IP addresses of mobile clients dynamically change as the devices switch from network to network.

The exploitation exists even when using refresh tokens. It's common for an authentication server to issue both refresh tokens and access tokens (which are both JWTs). A refresh token controls the maximum period of client inactivity, while an access token controls the maximum period of unchecked access. When the access token expires, the client requests new refresh and access tokens. At this time, the server can validate the refresh token against a database to make sure that the token is legitimate and that the user's access hasn't been revoked for some reason. However, the attacker can keep forging unexpired access tokens to prevent the refresh token from ever being checked.

If the attacker gains both read and write access to a server database or file system, the attacker could potentially do anything without having to forge JWTs. But software is extremely finicky, so attackers taking this approach are at greater risk of impairing servers and being detected before gaining an advantage. Attackers wishing to exploit a service are best off secretly employing the service as implemented. In any case, because server accounts generally have permission to read more data than they can write, attackers are more likely to only gain read access. Forging JWTs is therefore both the preferred approach and the approach that's more likely to be available.

JWTs vs Session IDs

It is tempting to dismiss this vulnerability because no system is ever completely secure, but JWTs introduce this vulnerability. Traditional session IDs need not be vulnerable this way. Moreover, this vulnerability can be seriously damaging when a breach goes undetected. Traditional session IDs here include session tokens and otherwise opaque authentication tokens.

News of big name websites being breached is all too common. Sadly, it is also common for breached sites to report that they were storing passwords in the clear, allowing the attacker to gain access as the user undetected and possibly to exploit accounts on other websites. Companies are proud when they can report that only password hashes were stolen and accounts not likely compromised, because hashes don't give attackers free access to accounts having strong passwords.

It is uncommon for companies to report whether they were storing session IDs in the clear. An attacker can use the ID of an active session to gain access as the owner of the session. However, because servers typically expire sessions after short periods, it can be hard for an attacker to successfully exploit a session ID. Sessions with short expiration periods can also be tied to IP addresses to harden them further. The savvy server will generate strongly random session IDs and only ever store their hashes, making sessions virtually impossible to exploit.

It is easy for servers to secure session IDs.

Contrast this with our JWT scenario. If a server that stores unprotected or naively-protected JWT signing keys is breached, the company may have to report that all accounts were exposed despite having hashed passwords. The company that incautiously jumped on the JWT bandwagon finds itself in a less enviable position than the company that stuck with conventional session IDs. Mind you, that word "incautiously" is important here, as it is possible to secure JWT signing keys.

No Inherent Protection

Seeing that tutorial after tutorial places JWT signing keys in the file system, we may be inclined to assume that something as-yet-unmentioned about the file system or about JWT protocol provides the protection. Let's consider some of the possibilities.

A JWT can store a session- or token-unique in the "jti" claim of the payload. To gain a security benefit from this unique, the server would have to both confirm the unique on each request and ignore the rest of the payload. This allows the JWT to be used as a session ID, but it does so at the cost of obviating most of the benefit of JWTs. There does however appear to be one benefit to signing a unique: the server can verify the signature before looking the unique up in a database. This can reduce the impact of a DoS attack by keeping the database from being accessed.

Perhaps an attacker can be kept blind to valid JWT payload values by having the server only store hashes of these values. That way, it would not be sufficient for an attacker to steal a signing key, because the attacker would also need the unhashed values that are only known by clients. Let's try this out on the user ID, given by the "sub" claim, one of the crucial pieces of the payload. The unhashed user ID would need to be returned in the JWT to the client at authentication so that the client can include the ID in the JWT with each service request. Well, because a given user can authenticate multiple times, whether on multiple devices or on the same device periodically after logging out, the server must be able to repeatedly put the unhashed user ID in the JWT. It can only do that if it is storing the user ID unhashed, leaving the server unable to blind the attacker this way.

JWTs are commonly used with the Oauth protocol. Oauth 2 provides a service for "token introspection" that allows a server to verify the validity of a token such as a JWT. This service exists to allow tokens to be revoked on events such as deletion of the user, change of password, or change user permissions. The service can report that a token is invalid even if the token is properly signed because the service does not rely on the signature. A server could use this introspection service to validate each token on each request. But doing so would obviate the benefit of having JWT payload data: there is a lookup hit for each request, and the introspection service could simply provide the data in response to an opaque token. A server could validate tokens against the service at intervals, but the JWT could be exploited for the duration of the interval, and servers would need to maintain indefinite blacklists to prevent themselves from accepting JWTs that were previously found invalid.

Many JWT tutorials that offer advice about storing the signing key suggest putting the key in an environment variable that is set in the shell profile or in a script that runs at server launch. In these cases, the server runs as the user that can read the file that contains the key. It is hard to confidently secure anything but a very simple server, and an attacker working server's entry points may find a way to get the server to return the file. If the file is accessible to a group of users, such as the site developers, the attacker has more chances to acquire the file. In any case, the site admin is now set with the task of carefully protecting file privileges, a task that's less crucial when using session IDs.

The approach of storing the key encrypted in the file system has already been mentioned. While providing a level of indirection is an impediment, this just moves the problem to protecting the decryption key, which most of us would again make somehow readable by the server's user.

The Problem of Protecting Keys

It seems that if we're going to use JWT tokens in a way that is at least as secure as traditional session IDs, then we must take extraordinary measures to protect the signing keys. In particular, we should not take the most obvious approach of putting the key in a file that is readable by the server. I am not an expert in how this should be done, but I do know a few ways it can be done:
  • Hardware security modules (HSMs) may provide the most secure way to protect a key. A special hardware device is attached to the server. Private keys resides in the device and no where else. The server delegates responsibility for signing JWTs to the device. Anyone who breaks into the server would then have to find a way to break into the device to get the keys.
  • The server that receives authentication requests can delegate responsibility for signing keys to a simple backend server. The backend server is not be on the Internet, and its network interfaces are carefully designed to ensure the security of the system. The key may be stored on the file system of this backend server, but after breaching the Internet-facing server, the attacker would have the additional task of having to breach the more-protected downstream server.
  • The signing keys are only stored in RAM and not in the file system. When the server goes down, the keys are lost. If asymmetric keys are being used, the public keys can be stored in the database or file system because they can't be used to sign JWTs. If the server goes down, the private keys are lost, but all JWTs that were signed with them can still be verified using the public keys. On a load-balanced cluster, each server can generate its own keys at launch and store the public halves in the database where other servers can read them to verify JWTs.
  • The signing keys are stored in the file system accessible only to a single user that is not used for any other purpose, and a second process runs on the server as this user to provide JWT-signing services to the web server process. The web server process communicates with the JWT-signing process via strictly memory-based IPC to get JWTs signed.
These approaches are probably listed most-secure first. This StackExchange answer lists some other possibilities as well. OWASP provides the following general guideline for storing keys:
Keys should remain in a protected key vault at all times. In particular, ensure that there is a gap between the threat vectors that have direct access to the data and the threat vectors that have direct access to the keys. This implies that keys should not be stored on the application or web server (assuming that application attackers are part of the relevant threat model). [OWASP - Cryptographic Storage Cheat Sheet]
Symantec's Internet Security Threat Report 2016 (p. 72) has this to say:
Make sure to get your digital certificates from an established, trustworthy certificate authority that demonstrates excellent security practices. Symantec recommends that organizations:
  • Use separate Test Signing and Release Signing infrastructures.
  • Secure keys in secure, tamper-proof, cryptographic hardware devices.
  • Implement physical security to protect your assets from theft.
If a key is ever used on a server to sign a JWT, the key must occur in plaintext in RAM. It is not clear that RAM is any more secure than the file system. For example, there are several ways that RAM can end up in the file system. When more software is loaded than there is RAM to hold it, the operating system can page-swap RAM out to files as needed to make room for the software that is waiting to run. Similarly, when an error occurs or a diagnostics operation is performed, the contents of RAM can be written to a file as a core dump. There may also be ways for an attacker to read RAM. This exposure could come from within the server by the attacker executing software, or it come through faulty interfaces to the server that allow an attacker to remotely extract RAM for inspection. Hence, keys are safest residing completely outside of Internet-facing servers.

Most programmers seem to find all this overkill. Most seem content just to make things tricky for attackers, such as by splitting a key and storing its pieces in multiple locations or by transforming the key after loading it from a file and using the transformed key. Security professionals call this class of approaches "security through obscurity." Security through obscurity may slow some attacks down, but it only stops novice attackers. Real security requires compartmentalizing accessibility. For example, security is gained by putting one half of a split key in the file system accessible by one user and the other half in a database that is not accessible to that same user. This requires an attacker to make two successful breaches instead of just one. A web server that has both privileges to the file and credentials for the database provides a single point of access to both halves of the key.

Unfortunately, the problem of protecting keys is becoming more pervasive as web service APIs grow in popularity. When a company such as Google or Flickr provides web services, they issue an API key to the software developer to use for accessing the services. If the developer writes software that uses these services to provide additional downstream services, the developer must store the key on a server that is somehow connected to the Internet, even if indirectly connected. Each API uses its own key, so developers may find themselves needing to protect multiple keys. Protecting keys requires careful design for security, and it's likely that most sites instead keep things simple and insecure. Read more about this growing problem at ProgrammableWeb and the Cloud Security Alliance.

Trusting Authentication Servers

While the vulnerability we're highlighting affects all clients of a service, the authenticating server is the source of the vulnerability. This is the server that receives client credentials, such as username and password, and returns a signed JWT in response to successful authentication. The risk to the client depends on the degree to which the authenticating server has secured its signing keys. As we have seen, it takes extraordinary measures to protect keys. Most servers that roll their own authentication and issue JWTs probably cannot be trusted. But what about the big name authentication services?

Google, Yahoo, Facebook, and Twitter all offer Oauth 2 single sign-on services for third party websites. The user chooses which service to use for login, and the third party website delegates authentication to that service. The authentication service provides access tokens for the client to hand to the third party website. The third party website trusts these tokens. I've read that Google uses JWTs, but I don't know about the others. These third party websites can use the Oauth introspection service to verify each token on each request, but they don't have to. If they strive for scalability by trusting the signature and the JWT payload, these third party services are potentially vulnerable. The developers of third party websites need to know whether to trust the JWT tokens.

It seems crucial that authentication services say something about how they are protecting JWT signing keys, so that third party services can decide whether to leverage JWT scalability as a function of the authentication service. If this information exists for these services, I cannot find it.

There are also a number of tiers such as Auth0 and StormPath that provide backend authentication services for websites. Websites can delegate authentication and possibly authorization to these services so that users can login with usernames and passwords on the website without the site having to implement its own authentication mechanism. That way, developers can leverage the security of a single base of code that has been hardened across many applications. However, I am again unable to find where these sites are reporting how they protect keys from theft by breach.

At the moment, we seem to be assessing trustworthiness purely by brand name.

The Missing Warning Label

The takeaway here is simple. Developers should not be implementing JWT authentication unless they take pains to protect the signing keys. Developers should instead either stick with traditional session IDs or strictly delegate authentication to a trusted brand name authentication service. Moreover, authentication services should really be describing their efforts to protect keys so that developers can make informed decisions about trusting access tokens.

This information is missing from the dozens of treatments of JWTs that I've seen. I've created a warning label that really ought accompany any discussion of developing JWT-based servers:

Unless the convenient assumption really turns out to be true, it is worth designing to protect against breaches in which data is only stolen. In that case, I offer this JWT authentication warning: Only implement authentication that signs JSON Web Tokens if you also compartmentalize access to the signing keys. Otherwise delegate authentication via Oauth or stick with session IDs.


Popular posts from this blog

How Spiders Get Bigger by Molting

Mystery of the Misumena fidelis Crab Spider

A Book Written by Spiders