Toolkit Overview

The main tools for end users are scp, sftp, and ssh. Use scp to copy files between your home computer and a remote host or between two remote hosts. For more flexibility in slinging files, use sftp, a drop-in replacement for trusty-old but insecure FTP clients. For the most generality, use ssh (aka slogin) to open an interactive shell on a remote computer. You can then issue whatever commands your remote account permits. You can also tell ssh to just run a command without opening an interactive shell. These three utilities look for default user options in ~/.ssh/config and default system options in /etc/ssh/ssh_config. Command-line options overrule user-default options; user-default options overrule system-default options; system-default options overrule built-in defaults.

For authentication with public-key encryption, you'll also use ssh-keygen and maybe ssh-copy-id from time to time. Use ssh-keygen to create your key pairs and to manage public keys for remote hosts. Use ssh-copy-id to install your public keys onto your remote accounts. You may also find ssh-keyscan handy on occasion should you wish to retrieve host keys explicitly.

When you protect your private key with a passphrase, you can use ssh-add to cache the key in memory and thereby reduce the number of times you need to type your passphrase. The cache holds your private key for the duration of your login session, by default. Or, you can limit the in-cache lifetime to your liking. The cache happily holds multiple private keys simultaneously as well. Under the hood, ssh-agent transparently manages your cache. You don't directly call ssh-agent yourself, however. Instead, your desktop session invokes ssh-agent on your behalf and without your intervention.

The default directory for a user's configuration and key files is ~/.ssh.

These notes were written with version 7.72p2 of OpenSSH:

-> ssh -V
OpenSSH_7.2p2, OpenSSL 1.0.2g-fips  1 Mar 2016

OpenSSH also offers a server, sshd. Its configuration file is /etc/sshd_config. The default directory for configuration and key files is /etc/ssh.

Package openssh provides ssh-keygen. Package openssh-clients provides the main client-side tools. Package openssh-server provides the server and its helpers. Package openssh-askpass provides passwords prompts.

For your GUI/X11 desktop, Fedora starts ssh-agent in such a way that any application you run from your session can find the agent when needed; see /etc/X11/xinit/xinitc. If you switch to a text/CLI console outside of X11, you'll need to start the agent yourself. For example, if you switch to a Bash console via Ctrl-Alt-F2, say, start the agent this way:

-> eval `ssh-agent -s`

Besides starting the agent itself, this double-duty command establishes environment variables that interested parties consult for directions to the agent:

-> env | grep --perl-regexp 'SSH_(AGENT|AUTH)'

Keep in mind that separate instances of the agent maintain separate and independent caches.

ssh-add may call on its helper gnome-ssh-askpass to prompt for a needed SSH passphrase. Environment variable SSH_ASKPASS locates the executable. Fresh Ports/openssh-askpass

Generating Keys

Here's how to generate an ED25519 key pair in files id_demo1 and

-> cd ~/.ssh
-> ssh-keygen -f id_demo1 -t ed25519 -C "Demo Key #1"
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in id_demo1.
Your public key has been saved in
The key fingerprint is:
SHA256:Fr1iO52hYHZDMVLSA6VqJjEOy44+JdR8MZ7LwBCfQ9A Demo Key 1
The key's randomart image is:
+--[ED25519 256]--+
| o+.  +==        |
| .oE.o +o+       |
|. B+. = o..      |
|.= B.= . . .     |
|o.o B = S o      |
|o. = = = * o     |
|..o     + o      |
|..       .       |
| ..              |

Option -C embeds a comment for your convenience when you subsequently review keys. If you prefer to see MD5 fingerprints instead of the default SHA256, you can append option -E md5 as well.

File id_demo1 holds the private key of the pair, and file holds the corresponding public key. Note the file permissions. The private key is readable only to its owner, while the public key is readable to all:

-> ls -gG id_demo1*
-rw-------. 1 444 Apr  4 11:28 id_demo1
-rw-r--r--. 1  92 Apr  4 11:28

These file permissions reflect the keys' roles: Keep your private key to yourself; share your public key as you wish.

If you do not need that wordy report, you can quell it with option -q:

-> rm id_demo1
-> ssh-keygen -f id_demo1 -t ed25519 -C "Demo Key #1" -q
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 

Here's how to generate an RSA key pair in files id_demo2 and

-> ssh-keygen -f id_demo2 -t rsa -b 3072 -C "Demo Key #2" -q
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 

Option -b 3072 sets the number of bits in the key to 3072 instead of the default 2048.

You can similary create a ECDSA key pair with option -t ecdsa, and you can select its key length with -b (256, 384, or 521). Likewise, you can create a DSA key pair with option -t dsa; option -b does not apply for DSA.

If you enter a passphrase, it is used to encrypt your private key. If you do not need a passphrase for your private key, simply hit enter at the prompts. Or you can say so from the get-go with option -P instead:

-> rm id_demo1 
-> ssh-keygen -f id_demo1 -t ed25519 -C "Demo Key #1" -q -P ''

You can also specify a real passphrase with option -P, but this exposes your passphrase in any record of your process. For example:

-> rm id_demo1 
-> ssh-keygen -f id_demo1 -t ed25519 -C "Demo Key 1" -q -P "Hello"
-> history 2
 1022  ssh-keygen -f id_demo1 -t ed25519 -C "Demo Key 1" -q -P "Hello"
 1023  history 2

This potential exposure is generally a bad idea on a shared machine.

You can change your passphrase easily with option -p:

-> ssh-keygen -f id_demo2 -p
Enter old passphrase: 
Enter new passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved with the new passphrase.

You can change your passphrase wholly from the command line, too, with additional options -P for the old passphrase and -N for its replacement:

-> ssh-keygen -f id_demo2 -p -P Hello -N 12345
Your identification has been saved with the new passphrase.

But again, this is generally a bad idea on a shared machine.

When you do give a passphrase, ssh-keygen encrypts the private key using cipher 128-bit AES. (See FILES section of man page).

-> head -3 id_demo2
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,766B5D5AC1DA1E869B124EE165C1D3C6

For storing ED25519 private keys, OpenSSH uses its own key format, rather than the usual and ubiquitous PEM format. The change stems from the the absence of ED25519 in OpenSSL, but the new format better resists brute-force attempts to crack the passphrase. For storing RSA private keys, OpenSSH uses PEM format by default. You can use the new format for an RSA key too, if you wish, by adding option -o when you generate the key or subsequently change its password (with -p). This new format is not accepted by versions of OpenSSH before 6.5, which added support for ED25519.

When you give a passphrase for your private key, OpenSSH encrypts your key with a cipher (AES) and stores only this representation in a file. The cipher needs its own secret key to work its magic—that's a second key, independent of your private key. A key derivation function, or KDF, generates the cipher key from your passphrase. OpenSSH's new format uses bcyrpt for the KDF, instead of MD5 with PEM. This switch is what gives the new format better resistance against brute-force attempts to guess your passphrase, as compared to PEM. You can tune the resistance with option -a, which selects the number of rounds for the KDF, an iterative algorithm.

Under the hood, your private-key file comprises a handful of parameters for the key's algorithm. If you're curious, you can peek inside an RSA key using the Open SSL toolkit. The parameters are big integers encoded as long hex strings; this example truncates these strings for brevity:

-> ssh-keygen -t rsa -f key_rsa_demo -P '' -q
-> openssl pkey -in key_rsa_demo -text -noout
Private-Key: (2048 bit)
publicExponent: 65537 (0x10001)

File key_rsa_demo stores the parameters using (encoded) ASN.1 (Abstract Syntax Notation One). This example again cuts the long hex strings:

-> openssl asn1parse -in key_rsa_demo | cut --bytes 1-80
    0:d=0  hl=4 l=1188 cons: SEQUENCE          
    4:d=1  hl=2 l=   1 prim: INTEGER           :00
    7:d=1  hl=4 l= 257 prim: INTEGER           :C23C74E2E9AB62AE30F49945FCB9C395
  268:d=1  hl=2 l=   3 prim: INTEGER           :010001
  273:d=1  hl=4 l= 256 prim: INTEGER           :35B4117D4BAE1F329A4686BA5B0DE18D
  533:d=1  hl=3 l= 129 prim: INTEGER           :EE36FC80CA5478BB7F2F4B8E3B4FEF26
  665:d=1  hl=3 l= 129 prim: INTEGER           :D0BCE76CC783D8405BBD4291FC71FA14
  797:d=1  hl=3 l= 129 prim: INTEGER           :DEC44F17859EBF14E0187378BC20C14D
  929:d=1  hl=3 l= 128 prim: INTEGER           :61C0855440DEF28FB926999D4A2E8A21
 1060:d=1  hl=3 l= 129 prim: INTEGER           :A1A8C669D78E8150E7A8FDFA3DB765D

See the man page asn1parse for an explanation of the report's format and for more options.

You cannot peek into an ED25519 key this way because OpenSSH uses its own format to store the key.

Cataloging Approved Hosts

When you first connect to an SSH server, OpenSSH will ask you to verify the fingerprint of that server's public key, called a host key, which the server itself remits. You'll see something like this:

-> ssh desktop.home
The authenticity of host 'desktop.home (' can't be established.
ED25519 key fingerprint is SHA256:sand/lPiVTIegY1mPqrNEMW+aMltbL2SzbOb0PpWzBw.
ED25519 key fingerprint is MD5:4b:ef:35:c9:c8:de:78:c7:bb:ba:aa:ed:be:06:ec:52.
Are you sure you want to continue connecting (yes/no)? 

You'll need to find the host's fingerprint independently of OpenSSH—perhaps on the host's validated website. Answer 'yes' only if the fingerprint remitted by the server matches the trusted fingerprint you retrieved elsewhere. This exchange ensures that you're talking to the Real McCoy and not some nefarious impostor in the middle.

You need to verify a host only on first contact. With your imprimatur, OpenSSH stores the host's public key in ~/.ssh/known_hosts for future reference:

Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'desktop.home,' (ED25519) to the list of known hosts.

You need not create or edit your known_hosts file (although you can if you wish). OpenSSH takes care of creating it and appending approved keys to it.

Use ssh-keygen to list the fingerprints of the host keys in your personal catalog:

-> cd ~/.ssh
-> ssh-keygen -f known_hosts -l 
1040 SHA256:I5Pz/5eo6lMOkGhNk/X9ChysmmTUD+wErD3Xrj6gUX4 router, (RSA)
256 SHA256:sand/lPiVTIegY1mPqrNEMW+aMltbL2SzbOb0PpWzBw desktop.home, (ED25519)
2048 SHA256:cXu4yA0YoW3TnciahoQlR3d5rTe0x/gKbg3IRFYyic4 desktop.home, (RSA)

You see the key's length, the key's fingerprint, the host IDs, and the key's type (algorithm). Add option -E md5 if you fancy MD5 over SHA256:

-> ssh-keygen -f known_hosts -l -E md5
1040 MD5:30:55:d1:f3:4b:34:d6:ab:68:f8:94:47:8b:a5:6c:40 router, (RSA)
256 MD5:4b:ef:35:c9:c8:de:78:c7:bb:ba:aa:ed:be:06:ec:52 desktop.home, (ED25519)
2048 MD5:07:2e:76:61:a2:30:2a:6e:63:96:d2:e6:44:24:47:52 desktop.home, (RSA)

This ditty cuts the fingerprint fat—just for posterity:

-> ssh-keygen -f known_hosts -l | cut --delimiter=" " --complement --fields=2
1040 router, (RSA)
256 desktop.home, (ED25519)
2048 desktop.home, (RSA)

You can use option -F to retrieve entries for a given host:

-> ssh-keygen -f known_hosts -l -F desktop.home
# Host desktop.home found: line 2
desktop.home ED25519 SHA256:sand/lPiVTIegY1mPqrNEMW+aMltbL2SzbOb0PpWzBw 
# Host desktop.home found: line 3 
desktop.home RSA SHA256:cXu4yA0YoW3TnciahoQlR3d5rTe0x/gKbg3IRFYyic4 

This report formats the output differently from the examples that do not specify a host.

Although your known_hosts file contains only public keys, it nevertheless reveals the hosts you connect to. You may prefer to keep this metadata private. Accordingly, your known_hosts is readable only by you:

-> ls -gG known_hosts
-rw-------. 1 2222 Apr  5 12:32 known_hosts

Your metadata are still exposed to user root on a shared machine, however. If this worries you, direct ssh-keygen to hash host names and addresses. For example:

-> ssh-keygen -f known_hosts -H
known_hosts updated.
Original contents retained as known_hosts.old
WARNING: known_hosts.old contains unhashed entries
Delete this file to ensure privacy of hostnames

The host names and addresses are now hidden. Compare before and after:

-> ssh-keygen -f known_hosts.old -l | cut --delimiter=" " --fields=3
-> ssh-keygen -f known_hosts -l | cut --delimiter=" " --fields=3

But you need to be vigilant here. First, new entries to known_hosts will not be hashed by default unless you activate option HashKnownHosts in your configuration file. Otherwise, you'll need to explicitly hash as above each time you add a host you wish to hide. That's safe to do because already-hashed entries are left in peace. Second, there's no translating back from hashed names to actual names. Option -F still works, though:

-> ssh-keygen -f known_hosts -l -F desktop.home
# Host desktop.home found: line 2
desktop.home ED25519 SHA256:sand/lPiVTIegY1mPqrNEMW+aMltbL2SzbOb0PpWzBw 
# Host desktop.home found: line 3
desktop.home RSA SHA256:cXu4yA0YoW3TnciahoQlR3d5rTe0x/gKbg3IRFYyic4 

By the way, if you're worried about keeping your contacts private, be careful not to reveal them in your configuration file, either. And remember to delete known_hosts.old if it retains unhashed entries.

If you wish to purge keys for hosts you no longer visit, you can have ssh-keygen remove the keys from your catalog of known hosts, like so:

-> ssh-keygen -R desktop.home
# Host desktop.home found: line 2
# Host desktop.home found: line 3
/home/ray/.ssh/known_hosts updated.
Original contents retained as /home/ray/.ssh/known_hosts.old

Keep this in mind if your host server changes it public key.

If an SSH server in your catalog changes its host key, OpenSSH will greet you with a dire warning when you next try to connect. For example, here's what happens when the demonstration server on desktop.home changed its host key:

 -> ssh desktop.home
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
Please contact your system administrator.
Add correct host key in /home/ray/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/ray/.ssh/known_hosts:3
ED25519 host key for desktop.home has changed and you have requested strict checking.
Host key verification failed.

Of course, you'll also get this warning if you're indeed the victim of a man-in-the-middle attack. Once you've satisfied yourself that it's just the benign case of a changed host key, remove the defunct key from your known_hosts, reconnect to the server, and verify the new fingerprint remitted.

For more about strict checking of host keys, alluded to in the warning, see option StrictHostKeyChecking in man page ssh_config.

You can also add hosts to your catalog manually so that initial connections do not pause for interactive approval. First, use ssh-keyscan to retrieve a server's host key. For example:

-> ssh-keyscan -t ed25519 desktop.home
# desktop.home:22 SSH-2.0-OpenSSH_7.2
desktop.home ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPu7fd///uEsQH4HByDySgxTUkYEpUuQJ4AApJo/eurv

That long string is the Base64-encoded host key. You can specify the key type you seek with option -t. For an RSA key:

-> ssh-keyscan -t rsa desktop.home
# desktop.home:22 SSH-2.0-OpenSSH_7.2
desktop.home ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiCP9N6qGanRBlgM3JUIa9Oop/9xg+/VlFyyuOUpy…

You can say ed255219,rsa for both types. See the man page for other key types and for default behavior.

Next, take the key's fingerprint to vet it against a trusted fingerprint you obtain independently of SSH. You can use md5sum or sha5sum; just remember to decode the key first. Here's how, for MD5:

-> echo AAAAC3NzaC1lZDI1NTE5AAAAIPu7fd///uEsQH4HByDySgxTUkYEpUuQJ4AApJo/eurv | base64 --decode | md5sum
4bef35c9c8de78c7bbbaaaedbe06ec52  -

Or from the get-go:

-> ssh-keyscan -t ed25519 desktop.home 2>&1 | tail -1 | cut -d' ' -f3 | base64 -d | md5sum
4bef35c9c8de78c7bbbaaaedbe06ec52  -

Once you're satisfied that the host key is legit, simply append the output from ssh-keyscan onto known_hosts:

-> ssh-keyscan -t ed25519 desktop.home 2>&1 | tail -1 >> known_hosts

If you want to check your work:

-> ssh-keygen -f known_hosts -l -F desktop.home -E md5
# Host desktop.home found: line 8 
desktop.home ED25519 MD5:4b:ef:35:c9:c8:de:78:c7:bb:ba:aa:ed:be:06:ec:52 

Don't forget to hash the added host names and IP addressed now if that's your style.

You can easily edit known_hosts manually with your favorite text editor. Each key spans a single line in the format remitted by ssh-keyscan; blank lines are ignored; lines starting with '#' are comments. To add a host, insert the corresponding output from ssh-keyscan. To delete a host, remove the corresponding line from the file. If you've hashed hosts' names and addresses, let ssh-keygen identify that line for you:

-> ssh-keygen -f known_hosts -l -F desktop.home -E md5
# Host desktop.home found: line 8 
desktop.home ED25519 MD5:4b:ef:35:c9:c8:de:78:c7:bb:ba:aa:ed:be:06:ec:52

Man page sshd documents the format for this file in section SSH_Known_Hosts File Format.

Authorizing IDs You Connect As

When you connect to an SSH server, you'll need to prove your identity before the server grants access to your account on the remote host—the computer running the SSH server. In a typical default, you prepend your user name to the host name and remit your password when prompted:

 -> ssh [email protected] 'hostname --long'
[ *** OpenSSH demonstration server *** ]

After a little setup, you can instead use your SSH key pair to prove yourself transparently and without providing a password. Residing on the remote host, file ~/.ssh/authorized_keys holds your public key. It's up to you to populate this file, and ssh-copy-id is here to help:

-> ssh-copy-id -i ~/.ssh/id_desktop [email protected]
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ray/.ssh/"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[ *** OpenSSH demonstration server *** ]

Number of key(s) added: 1

This utility will create directory .ssh and file .ssh/authorized_keys in your remote account as necessary. It will append to an existing authorized_keys so you can connect using different identities from different client hosts, for example.

Subsequent connections will not ask for your password:

-> ssh [email protected] 'hostname --long'
[ *** OpenSSH demonstration server *** ]

If you want to replace an existing authorized_keys to start afresh, you can use scp, like this:

-> scp ~/.ssh/ [email protected]:.ssh/authorized_keys
[ *** OpenSSH demonstration server *** ]                                         100%  108     0.1KB/s   00:00 

Be alert to copy the file for your public key—not your private key—via extension .pub; scp does not have your back here. The remote server will prompt for your password if the current key pair is missing.

Your authorized_keys file may be writable only to its owner. Otherwise, sshd simply ignores it.

You can tune some aspects of an SSH connection under a given key by adding options to the corresponding authorized_keys file on the server. Man page sshd documents the format and options in section Authorized_Keys File Format.

For example, suppose you want desktop.home to authenticate with key id_desktop only if the client's IP address is You can do this by prepending the string from="" to the line with this public key. Here's what the file looks like, with the key abridged:

-> cat authorized_keys
from="" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID3P4qU…

Now the server ignores this key unless the client connects from that IP address. The server may alternatively accept another form of authentication, such as a password; that depends on the server's configuration.

Running a Server

You can easily run your own SSH server, courtesy of OpenSSH. These notes give the basics geared towards a demo server on a protected home network.

The OpenSSH server is sshd. Under Fedora, use systemctl to start it:

-> systemctl start sshd

The server reads its configuration from file /etc/ssh/sshd_config, by default. Man page sshd_config describes the many options you can adjust. If you change your configuration file while the server is running, restart sshd to assert you will:

-> systemctl restart sshd

Under Fedora, three pairs of host keys are generated automatically, a pair each for ECDSA, ED22519, and RSA:

-> cd /etc/ssh
-> ls -1 ssh_host_*_key*

When it comes time to share your public keys' fingerprints, let ssh-keygen lend a hand, like so:

-> ssh-keygen -l -f 
256 SHA256:sand/lPiVTIegY1mPqrNEMW+aMltbL2SzbOb0PpWzBw no comment (ED25519)
-> ssh-keygen -l -f -E md5
256 MD5:4b:ef:35:c9:c8:de:78:c7:bb:ba:aa:ed:be:06:ec:52 no comment (ED25519)

Finally, the host machine's firewall must open port 22 to TCP connections (SSH service).

Here is the simple configuration file for the demonstration server in these notes:

-> ls -l /etc/ssh/sshd_config
-rw-------. 1 root root 477 Apr 18 12:30 /etc/ssh/sshd_config
=> cat /etc/ssh/sshd_config
# Say hello
Banner /etc/ssh/banner.txt

# Identify host keys to use.
HostKeyAlgorithms   ssh-ed25519,ssh-rsa
HostKey             /etc/ssh/ssh_host_ed25519_key
HostKey             /etc/ssh/ssh_host_rsa_key

# Select client authentication 
AuthenticationMethods    publickey password
PubkeyAcceptedKeyTypes   ssh-ed25519,ssh-rsa
AuthorizedKeysFile       .ssh/authorized_keys
UsePAM                   yes

# Enable SFTP subsystem
Subsystem sftp   /usr/libexec/openssh/sftp-server

=> cat /etc/ssh/banner.txt 
[ *** OpenSSH demonstration server *** ]

This configuration considers ED25519 and RSA keys only.

You can use sshd as a non-daemon command with option -t to check the validity of your configuration file. The command will report any objections or otherwise hold its peace. You can likewise use sshd with option -T to list all key-value pairs, both configured and default. For example:

=> sshd -T | grep -i HostKey
hostkeyalgorithms ssh-ed25519,ssh-rsa
hostkey /etc/ssh/ssh_host_ed25519_key
hostkey /etc/ssh/ssh_host_rsa_key

Under Fedora by default, three pairs of host keys are generated automatically, a pair each for ECDSA, ED22519, and RSA. Each pair is generated if it's missing whenever sshd starts—not at installation. You can refine what if any keys are so generated by tuning AUTOCREATE_SERVER_KEYS in /etc/sysconfig/sshd, which is self-documented. This file does not offer an official option to customize the names of the keys' files, but you can override the default choices by defining variables ED25519_KEY, RSA_KEY, etc. Shell script sshd-keygen does the work at the behest of sshd-keygen.service (/usr/lib/systemd/system/sshd-keygen.service).

The host's public keys are stored in /etc/ssh and are world-readable so that users may easily copy the keys into their known-hosts files and find the keys' fingerprints. OpenSSH does not use these files; it gets what it needs from the private-key files.

The "keyboard-interactive" method of authentication (AuthenticationMethods) may end-up defaulting to plain-old password authentication. But it offers more general authentication via challenge-response exchanges if so configured. What the heck is "keyboard interactive" authentication? and Difference between password and keyboard-interactive

Importing and Exporting Keys

To export an OpenSSH public key to a format suitable for applications beyond the OpenSSH suite:

-> cd ~/.ssh
-> ssh-keygen -e -f -m RFC4716
Comment: "3072-bit RSA, converted by [email protected] from OpenSSH"

-> ssh-keygen -e -f -m PKCS8
-----END PUBLIC KEY-----

-> ssh-keygen -e -f -m PEM

Despite what its man page says (for option -m), ssh-keygen does not export a private key. Instead, it exports the corresponding public key:

-> ssh-keygen -e -f id_demo2 -m RFC4716
Comment: "3072-bit RSA, converted by [email protected] from OpenSSH"

Convert OpenSSH private key into SSH2 private key