To attempt to rectify this SPF was created. To setup SPF you add either a TXT or an SPF record to the DNS zone you will be sending from. This record defines which servers are allowed to send mail that is coming from that domain. So on my domain danielhall.me I could publish an SPF record that says only my mail server is allowed to send mail that ends in @danielhall.me. Any mailservers receiving mail that is from my domain but not coming from an address listed in my SPF record can see that the mail is likely forged and throw it away. SPF works well in most situations but fails at a very common use case. If someone I send mail to tries to forward it to another address using an automatic process (no clicking forward in their client) then the mail will appear to come from my domain when it gets to the user it was forwarded to, however it will have came from the original recipients mailserver.

DKIM solves this problem by giving each sending mailserver a cryptographic key pair. The public keys is then published in a DNS record in that zone and stores the private key somewhere safe on the server. The server then proceeds to sign the headers (especially the From: header) and the body of all outgoing emails. This signature is then attached to the email as an extra header. When the receiving server get the email it gets the signature and uses that along with the list of signed headers to verify the signature against the public key of the signing domain. This means as long as the mail has passed through an authorised mailserver at any point it will be considered valid.

Setting up a DKIM is relatively simple process. You will need access to the  zone records for your domain and access to the configuration of all the mailservers which all mail originating at your domain passes through. You also need to be aware that signing mail makes it slightly more processor intensive to send an email. If you send a large amount of email this difference could be quite significant. If you’re using sendmail you may be able to alleviate it by switching to a less resource hungry MTA like Exim. You should also note that in some configurations DKIM can not be setup. For example if you use masquerading in sendmail DKIM will always fail as sendmail will modify the from header after signing.

Ultimately DKIM is a good move for the internet community at large, especially when combined with SPF. DKIM mail is assured to come from the sender and can be cryptographically proven so. While it does take more take a little more effort to setup and maintain it assures mail from your domain is secure and can be assured to have come from you or your company. Ultimately DKIM can protect your company against phishing attempts and boost your spam scores.

Random thought: What would Email look like if it were designed today?

So the key part of why a SSH agent and SSH agent forwarding forwarding is so useful is due to the way keys can be attacked. If I wanted to get your SSH private key I could find some flaw in the system that would give me that /home/you/.ssh/id_rsa file you have. Of course a malicious user with root access to the system could just go in and grab it. You can prevent this kind of attack by setting a passphrase on the key. Of course the root user could replace SSH with a special version designed to get your passphrase, steal the key out of memory or setup a keylogger. This means effectively that your private key is not safe on any system where a person you don’t trust has root access, or has other users and exploitable vulnerabilities.

Single Private Key on Multiple Machines

In this example you’re trusting the security of every single machine you have your private key on. Should it get compromised then you have to revoke you public key from every host, and regenerate private keys to place on every host. Every time you put your private key on a machine you increase the chances that it could be compromised.

Multiple Private Keys On Multiple Machines

So we’re getting a little closer to a good solution. In this instance we don’t have to generate our key and roll it out to all hosts in event of a compromise. You can also have segregate groups, on set of keys for work, another for home and so on. Your keys can still be compromised easily though, and once compromised they can be used until you revoke them manually.

SSH Agent Forwarding

There is a way to keep your key safe from compromise. Now I’ll have to explain how SSH authenticates you using your key. When your authenticating with SSH keys your key isn’t sent, the server sends you some random data and challenges your client to encrypt it with your private key. It then verifies the encrypted data by decrypting it with the public key and checking if it matches the data originally sent. Now the way most people would SSH from the second host to another third host is to utilise a private key on the second host to connect to the third host. Unfortunately this method means that you have to store a key (that is open for compromise) on the second host. SSH agent forwarding tells the SSH client on the second server to send the challenge data through to the SSH client (or ssh agent) on the first host. The agent encrypts the data and sends it via the SSH session to the third client.

The beauty of this method is that the second host never sees a private key, and the challenge data is useless to try and connect to a different host. Even if the second host is compromised there isn’t a private key there to compromise. It should be noted that if the second host is compromised it can still request the agent identify for a different host, or the session to the third host can be taken over. Both these are temporary though and unless the malicious user installs their key (something easy to notice) they cannot get back in.

Diagram detailing how an SSH connection is authenticated using agent forwarding.

If you want to know more about how this works, there is a wonderful tech tip at http://unixwiz.net/techtips/ssh-agent-forwarding.html.

But how?

SSH agent forwarding is even easier than copying keys all over the place. The first step is to generate keys for all the machines you log on to directly. You need to be sure these machines are secure and that your keys will stay safe, though this is sometimes not possible. You then add the generated public key to the authorized hosts file of all the machines you will connect to from this one, including ones that take two or more steps to get to. Finally you edit your ~/.ssh/ssh_config file to tell SSH to forward your agent through those hosts. Include the intermediate hosts in this list, but not the endpoints. You could also use SSHmenu to add the arguments automatically to those SSH commands. The following disables forwarding to all hosts, and explicitly enables it to fred, and aaron.missgner.com.

Host fred
  ForwardAgent yes

Host aaron.missgner.com
  ForwardAgent yes

Host *
  ForwardAgent no

Random thought: Linux has Plug ‘n Pray too, you plug the device in and pray the drivers aren’t proprietary.

To use SystemTap first you write a simple script, or borrow one from someone else. On a Fedora system you’ll fine some sample scripts in /usr/share/doc/systemtap-0.9.8/examples provided you have SystemTap installed. You then run the stap; command. The stap command immediately begins parsing the scipt looking for any tapsets that your script uses and if it does it includes them. It then converts your script into C code and compiles it into a kernel module. This kernel module is inserted into the running kernel and stap attaches to it. The kernel module stays in the kernel until it is cancelled by the user, it reaches an exit function or it encounters too many errors.

While SystemTap can be used to simply dump loads of data about what an application is doing in kernel space that is not its purpose. SystemTap scripts are able to drill down, extract, process and format the data its gathering. For example if you were trying to find out what files a process was writing to your disks could just output every single write call and print it out, or you could keep the statistics and every ten seconds print the top ten files written to. SystemTap is designed to help you filter out all the noise and monitor only what you want to monitor.

The simple way to get started with SystemTap is to download the Beginners guide or the Tutorial. On Fedora systems when you install SystemTap you’ll find the tutorial at /usr/share/doc/systemtap-0.9.8/tutorial.pdf. SystemTap skills are handy for system administrators and developers, so if you fit into those categories I’d highly recommend you check it out.

Random Thought: Where does /dev/zero come from and where does /dev/null go? What happens if you pipe /dev/zero to /dev/null?

top

Top, the grandaddy of all the Linux top commands, is most useful for  monitoring tasks running on your system. On my Fedora system its contained in package procps which on Fedora 11 was 3.2.7. Top has many keybindings to change its behaviour, for example ‘f’ is used to add and remove fields, ‘o’ will help you reorder those fields and the lesser-than and greater-than keys move the search field. You can type ‘h’ for a bigger list.

tload

You caught me! This one doesn’t end with top, but I put it here because on Fedora it comes as part of the procps packages with top, slabtop and others. tload is a good application to have in a small terminal in the background.  It comes packaged along with top. It displays a histogram of the current load for the system. I like to have it running in a transparent terminal that I leave open on my laptop.

htop

An improved, menu driven and colourised version of normal top. Htop allows you to get information on each thread of a program or combine all thread like normal top does. Some would argue that its more powerful, but others simply say its bloated. Whatever you believe, it has some nice features that any sysadmin will appreciate and you’ll soon be wishing htop was avaliable
everywhere.

iftop

top is to cputime what iftop is to your network interfaces. It displays a list of the top servers that are exchanging data over the selected interface. Because of the way it captures packets from the interface it needs root privleges to run.

iotop

iotop displays live system IO statistics. Like top it lists the top applications that are using IO. It can be toggled with the ‘o’ key to only
display programs currently performing IO, which is useful on large servers. You can read more about its keybindings on its manpage.

slabtop

slabtop is especially useful for kernel developers and pedantic system tuners. It displays a summary of all the slab objects allocated in the kernel. I can take options to tell it how to display its information, but only has two keybindings, spacebar is to refresh the screen and ‘q’ is to quit. You can see its options on its manpage.

xrestop

For X developers there is a utility called xrestop. xrestop displays a list of X server resources allocated. It can be useful to see if your application, or your X server is leaking resources. While it only accepts the ‘q’ key to exit it does accept a few options.

powertop

Built by Intel to help tune laptops to get the best performance out of your battery. It shows the percentage time spent in each CPU state and lists the programs and devices that caused the most wake ups from idle mode. Its most useful feature though is that it will analyse your system and give a suggestion on action to be taken to save just that little bit more power.