Why SSH security practices need to change (and how 1Password can help)

Earlier this year, we introduced the 1Password SSH agent as part of our commitment to bring developers the kinds of things developers want to see. Today, I’ll discuss a re-evaluation of the security properties and habits some of us old-timers may have regarding SSH keys, and which of those habits are outdated.

The short version is that SSH was originally a drop-in replacement for rsh (remote shell) and rlogin, which were centered around one machine trusting another machine, or one account on a machine trusting an account on another machine. SSH private keys were associated not just with individuals, but individual accounts on particular hosts.

Some of us old-timers need to adjust how we think about SSH keys.

SSH key management tools and conventions grew out of that environment. A key pair didn’t so much belong to a person, but a user logged on to a particular host.

This all made sense at the time SSH was invented. But certain security practices - and unintended consequences - followed, and they’ve been problematic since. The 1Password SSH agent and its integration fix those problems, but it means that some of us old-timers need to adjust how we think about SSH keys.

It’s about people now, not machines

We designed the 1Password SSH agent with the belief that:

1. SSH key pairs belong to people, not to the machine those people SSH from.
2. SSH private keys should be locked and unlocked when other high-value user credentials are locked and unlocked.

I don’t think either of these beliefs are controversial. But when in the course of geeky events it becomes necessary for one set of geeks to dissolve practices and conventions, a decent respect to the opinions of others requires that they should declare the causes which impel such a break with tradition.

First, we need to understand the reasons for the conventions we’re breaking. Only then can we evaluate whether we’re doing something good or bad. TL;DR: We’re doing something good.

Next, I’ll go into some history in an attempt to explain what those original reasons were and why the conventional wisdom was what it was. This will also make it clear that the reasons for some of the SSH key management practices became obsolete quickly.

In the before time

Back in the 1980s, very few individuals used their own Internet-connected computers. Instead, they used terminals connected to somebody else’s computers. In my case, the computers were owned and operated by the universities I attended.

The user accounts on these computers were managed by system administrators, and ordinary users had limited rights on them. I could do stuff within my own account but I couldn’t touch the system’s configuration.¹ As more computers in these environments were made available, it became useful for users to log in to one of them from another.

For example, imagine two machines called russell and whitehead. Both are administered by the same people with similar policies. It makes sense that if I’d logged on to russell as user goldberg, and the operators configured whitehead to trust russell, I should be able to jump over to whitehead with little fuss. I could use rsh to hop over to whitehead if that machine listed russell as a trusted device for this purpose.

There were many problems with this scheme.

There was some ability to configure things for specific users but this was fundamentally a machine-to-machine trust relationship. Anyone with the same username on both machines could log in to one from the other with no further authentication.

There were many problems with this scheme. One was that it didn’t provide a reliable way for russell to prove to whitehead that it really was russell.² And as machines became cheaper and more connected to the network, the people who ran the system could no longer rely on the idea that every machine on their network was honest. It was too easy for someone to get a machine to masquerade as russell to whitehead.

SSH was a solution to that, and many other problems.

The point here is that rsh, along with the related rlogin, were based on trust between machines. SSH was designed as a replacement, but with cryptographic goodness.

User on a machine

Consider a network with a machine called russell that’s carefully managed and receives all security updates within months instead of years. (Things were bad in the bad old days.) Other machines, including quine, don’t get the updates as quickly. If SSH were configured like its predecessor – to trust machines and the usernames they provided – very bad things could happen.

For example, imagine Mr. Talk (my neighbor’s cat and the enemy of my dogs, Patty and Molly) has an account on quine and Molly doesn’t. If Mr. Talk gains enough control of quine to create the user molly, then he can SSH over from molly@quine to molly@russell.

System administrators recognized the problem fairly quickly and no longer configured SSH to trust their peers had the same usernames. This is when users like Molly set up SSH key pairs for each of their accounts, instead of key pairs just belonging to machines.

The practice had shifted from securely authenticating a machine to securely authenticating a user from a particular machine.

So Molly might create an SSH key pair on whitehead, and Molly’s private key for that key pair would only live on whitehead. She would add the molly@quine private key to her list of authorized keys³ on russell. Molly would also have to create a key pair on each of the hosts that she wanted to SSH from and add the public part as an authorized key on all of the hosts that she wanted to SSH to.

When Molly SSH-ed from quine to russell, the magic of public-key cryptography proved to the SSH daemon on russell that the other side of the connection possessed the corresponding private key. This, of course, was done without transmitting any secrets. The SSH daemon on russell was assured that it really was molly@quine connecting.

The practice had shifted from securely authenticating a machine to securely authenticating a user from a particular machine.

Enter the agent

At first, most people didn’t set passphrases on their SSH private keys. As long as you were able to read the private key file, you were able to connect to any location authorized to use that key.

It was still enormously more secure than rsh and rlogin, but it meant that (as with rsh and rlogin), if you walked away from your desk, someone could use your logged-in session on one machine to go anywhere you could.

This made the private key something like a token. You would create one for each machine you might SSH from. I might have a key with the name jpg@russell and another key named jpg@whitehead. The private part of each key only lived in one place. Access to the private part of jpg@russell was considered proof that I could log in to the account jpg on the machine russell.

There are two security problems with this. I had to be very, very careful about locking my workstation if I walked away for a moment, particularly as my jpg@russell key gave me access to more and more places. And anyone who ever gained read access to my disk (or the backup tapes used for these machines) would have identical access. The private key for jpg@russell was only supposed to live in one place, but even if I did everything right, there were many ways an unauthorized person could get hold of it.

It was really hard to get people to type in the passphrase for their SSH key every time they wanted to use it.

That security practice might seem horrible today, but remember this had been a drop-in replacement for passwordless rsh and rlogin. Sure, we could tell people their SSH keys must be password protected, but few people complied.

This was all happening while central management of machines on a host was becoming a thing of the past. There was no way for the administrator of russell to see how easy it would be for an attacker to get hold of Molly’s private key on quine. SSH private keys were too easy to steal, and it was really hard to get people to type in the passphrase for their SSH key every time they wanted to use it.

The solution to this was ssh-agent.

People could password protect their SSH private key files, but would only need to type in their password for that key once per login session. This created some attack opportunities, but it meant that read access to someone’s private key was no longer sufficient for an attacker. This was a human-centric security compromise.

It was unreasonable to ask or expect everyone to enter their SSH key password each time they wanted to use SSH, so by opening up a one attack vector, we precluded what would otherwise be a far easier attack.

Bringing us to yesterday

That history brings us to yesterday.

The standard ssh-agent requires you to unlock your private key once; it’s then usable for the duration of your local login. As soon as SSH private keys were password protected, there was no reason to treat them as tokens any longer: Having access to the private key files wasn’t enough to authenticate.

But old habits die hard.

When SSH was first introduced, keeping the private key files secure was paramount. You had one for each machine you logged in from – if an attacker gained read access to that disk, that one key could be de-authorized.

Today

With the 1Password SSH agent, your SSH private key is protected by the same security model used to protect dozens if not hundreds of your secrets. 1Password protects your SSH private key the same way it protects other secrets in the event an attacker gains read access to your disk (or ours). You unlock your SSH key just as you unlock your other credentials within 1Password.

If an attacker could get the SSH private key merely by having read access to the host, then it made sense to tie that key to that host. In short, keeping a private key on a particular host was reasonable if the private key was available to anyone who could read the disk.

In those days, the SSH daemon on russell trusted the private key of molly@quine by trusting that no attacker gained read access to Molly’s files on quine. But in the case of 1Password, the security of the private key isn’t threatened by an attacker who gains read access to the disk so there’s no reason to tie the private key to a particular host.

Here at 1Password, we can understand the tradition of having an SSH private key that’s never supposed to leave its host. But we understand the reasons that tradition shouldn’t apply when the private key is properly encrypted.

So I return to where I started.

1. SSH key pairs belong to people, not to the machine those people SSH from.
2. SSH private keys should be locked and unlocked when other high-value user credentials are locked and unlocked.

This enables new, and easier ways of working with SSH.

Jeffrey Goldberg

Principal Security Architect