> HOME > PROJECTS > SSH GATE
With many servers comes more and more complex systems and the need to manage a
web of interdependencies. The following describes a setup I have employed that
we have come to see as invaluable.
OpenSSH is an incredibly powerful piece of software, and by default most of its
features are barely utilized. Each of the following sections shows how to add
more juice to a standard OpenSSH setup, and each gets progressively closer to
our goal: effortless, encrypted, authentication and communication between Linux
The Default: Secure Password Authentication
# ssh username@remotebox
This is the default way ssh handles authentication. A user is asked for
his/her password on remotebox, the password is transmitted encrypted over the
wire, checked against the remotebox database, and voila: remotebox shell.
- - Very secure, have to re-authenticate with every new connection.
- - Have to re-authenticate with every new connection.
- - Unattended cron jobs and scripts are a problem since we need a password.
Better: Shared Keys
SSH also supports authentication using shared keys. We generate a pair of
keys, one private and one public. When we attempt to authenticate, remotebox
generates a random number and encrypts it using our public key (that is all a
public key is good for). This is sent back to us, and if we can successfully
decrypt it using our private key, we have proven we are who we say we are.
OpenSSH supports RSA and/or DSA authentication (SSH1 and SSH2) using shared
keys. We generate our keys using the ssh-keygen program. We will generate a
DSA key, which requires SSH2. Also, make sure NOT to take the easy way out.
Do NOT create a key with no passphrase (this will be explained later).
# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
Now, when we concatenate the id_dsa.pub to a user's ~/.ssh/authorized_keys2
file on remotebox, (authorized_keys for rsa keys), we will be able to connect
to that machine as that user.
- - Still secure, have to re-authenticate with every new connection.
- - Still inconvenient, have to re-authenticate with every new connection.
If we had not entered a passphrase above, our private key would not be
encrypted. This is an easy way to avoid having to re-enter a passphrase with
every connection, and is one way to setup for unattended cron tasks between
machines. But it also means that if our localbox is hacked, the hacker would
be able to skate into any remote boxes we had access to. By entering a
passphrase, we encrypt the private key so it is safer to store on our disk. If
our box is hacked, it is still a significant obstacle to actually use the key
to impersonate us and gain access to other hosts having our public key.
Unfortunately, if we do things the "right" way, the result is still largely the
same as the default password setup... we are entering a passphrase for every
SSH-Agent is a program included with OpenSSH that allows us to properly use
encrypted keys, but also avoid having to re-type the passphrase. It does this
by caching decrypted private keys, allowing us a higher level of convenience
without sacrificing much security. Here is typical ssh-agent output:
SSH_AUTH_SOCK=/tmp/ssh-XXM9mJcq/agent.32461; export SSH_AUTH_SOCK;
SSH_AGENT_PID=32462; export SSH_AGENT_PID;
echo Agent pid 32462;
The output of ssh-agent is a series of bash commands setting up environment
variables. If we eval this command, these environment variables are exported
to our shell.
# eval `ssh-agent`
Once started (best run from our .bash_profile), we need to run ssh-add to add
our private keys to its cache:
# ssh-add ~/.ssh/id_dsa
Need passphrase for /root/.ssh/id_dsa
Now that ssh-agent knows about our key, we can access remote systems without a
- - Finally we enter a passphrase only once.
- - New ssh-agent for each login session, meaning we have to re-run ssh-add, etc.
- - Cron jobs do not inherit the necessary environment variables and cannot contact ssh-agent.
Enter Keychain to squash these last two cons. Keychain is an intelligent bash
front-end to ssh-agent, which allows us to use a single ssh-agent process per
system, avoiding re-authentication for every new login. Keychain is available
Once installed, add the following two lines to your .bash_profile:
source ~/.ssh-agent > /dev/null
The first line is the path to our private key (or many lines for many keys).
Read the man page for keychain to see lots of other powerful options. Also,
you'll need to make sure ~/.ssh-agent actually exists as later versions install
files in ~/.keychain (symlinks will suffice). This is the file that contains
the environment variables. Upon first login, keychain prompts us for the
passphrases to each of our private keys:
KeyChain 2.3.5; http://www.gentoo.org/projects/keychain
Copyright 2002-2004 Gentoo Technologies, Inc.; Distributed under the GPL
* Found existing ssh-agent at PID 11718
* Adding 1 key(s)...
Enter passphrase for /root/.ssh/id_dsa:
* Identity added: /root/.ssh/id_dsa (/root/.ssh/id_dsa)
- - Secure, encrypted keys.
- - Finally, re-authentication is rare.
- - Make sure private keys are on a well-protected box.
Subsequent logins do not ever have to re-enter the passphrase in this setup.
We have reached our goal... now let's surpass it.
More Improvements: SSH Forwarding Agents and Cron
In our current situation, we can only login to remote machines using our keys
from one source... our SSH Gate:
The trusted host sshgate can only connect to each box, the boxes can not
connect to one another in a passwordless fashion. To work around this, we
could have ssh-agent running on every box, with copie of our keys everywhere,
but that is a security hazard. Instead, we will create forwarding agents.
With a single line added to a machine's /etc/ssh/ssh_config (not sshd_config),
any machine can become a forwarding agent:
As a forwarding agent, all connections originating from that trusted machine
sshgate, indirectly or directly, can use sshgate's own ssh-agent process rather
than ssh-agent on the remotebox:
From the ssh_config manpage: Agent forwarding should be enabled with caution.
Users with the ability to bypass file permissions on the remote host (for the
agent’s Unix-domain socket) can access the local agent through the forwarded
connection. An attacker cannot obtain key material from the agent, however
they can perform operations on the keys that enable them to authenticate using
the identities loaded into the agent.
If untrust1 is not a forwarding agent, it means nothing if our connection TO
untrust1 was FROM a forwarding agent: the ssh-agent socket file from sshgate
(or an intermediary forwarding agent) is dumped into untrust1's /tmp. Make
sure the ssh_config file on sshgate (and any intermediary forwarding agents)
specifies that connections to untrust1 should not be forwarded with a Host
declaration indicating such.
Some implications for agent forwarding:
- Keychain only needs to run on one box, also setup as a forwarding agent.
- Other trusted boxes receive only the public key, but are also forwarding agents.
- Untrusted hosts receive only our public key, and these are not forwarding agents.
- - All of the above goals.
- - Minimal exposure of ssh-agent, avoiding possibility, however remote, of decrypted keys being extracted from untrusted hosts.
- - Minimal configuration on each host.
- - We can bounce from host to host within our trust domain!
- - As always, the Keychain box must be kept secure.
How does cron fit into this? We now have unadulterated access from the
Keychain box to run commands over ssh to any other box in our trust domain.
One possibility is a central enterprise task scheduler that runs commands on
remote machines. The cron task simply needs to be passed the ssh-agent info.
Here is an example cron entry:
*/2 * * * * root source /root/.ssh-agent; ssh root@somehost "/some/script/somewhere"
Other possibilities abound! For more info, see the article at
Hardening the Trusted Host
We have eliminated all of the cons we were concerned about, but that last one
will always remain... the security of the one trusted machine. Some
possibilities to address that last con:
These measures are beyond the scope of this document, but should be strongly
considered. Some possibilities with SELinux:users have no direct access to
their own private keys, no access to some /etc files, etc. Very powerful...
and doable. We have run variations on the above theme for years.
- Strict iptables rules only allow incoming connections for SSH, and perhaps DNS from specified hosts (no spoofing). Perhaps paranoid iptables rules allowing connections from a finite list of hosts.
- Users log into a custom shell that only allows a handful of commands. Root access closely guarded.
- SELinux to harden the machine.
No matter what measures you put in place, users will always be the weakest
link, and they need to be made well aware of the implications. In many ways,
we have created a castle gate.
If an intruder sneaks through our gate, he/she is not likely to spend a lot of
time trying to sabotage the gate, since it is armored, guarded, heavily
fortified, and of comparably little interest. Of far more interest is what
lies beyond the gate to which he/she now has full access.