A day in the life of an email
27th Feb 2011 | 12:00
We email every day, but what goes on when we click send?
A day in the life of an email
Email is now so ubiquitous, we no longer even consider how it all works. Billions of emails are sent each day (the majority of which are spam, admittedly), and even with the rise of social networking, we haven't abandoned email yet.
Some good, some bad; some work-related, some personal: it's the communications medium of the 21st century. But what exactly is an email? How does it get from me to you? What processes and servers have to be running in order to ensure all this magic works to the point where we don't need to worry about it?
Back in the very early days, messages could only be sent from computer to computer on the same network. For this to happen, both computers had to be running and online (that is, both endpoints had to have users logged in) since the originating computer made a direct connection to the destination computer in order to transfer the message.
This worked in essentially the same way that phone switches work to route a call: the originating and the destination phone must be connected directly for the length of the phone call. For computers on the same network, this method worked pretty well, but it didn't scale at all once we started to link local networks together.
The birth of email
In 1969, the precursor of the internet, ARPANet, was created by a research team at MIT and at DARPA (Defense Advanced Research Projects Agency). It was the first packet-switched network, so named because all data traffic was split up into packets. The packets were numbered sequentially and put into digital envelopes, with destination addresses encoded into the envelopes.
ARPANet was a collection of servers, each able to receive and pass packets onto other servers on the network. This meant that a large message would be split into different packets, and each packet might be routed a different way through the network to the destination. Each node on the network knew only enough to pass on packets that weren't destined for itself, and it was the receiving computer that was responsible for collecting all the packets that made up a message and checking that none were missing.
This methodology meant that packets from many different messages from many sources could be interleaved and sent on a link, without the need to tie up the link to send a single message.
A couple of years later in 1971, Ray Tomlinson implemented the first system that we would recognise as email. His system was based on a program that copied files across a network and allowed users of different networks to send messages (as files) to each other.
To help with the addressing of the email, he came up with a simple solution: separate the username from the remote network domain name by use of the '@' sign - a convention we still use today.
The earliest emails sent were text files, usually seven-bit ASCII. Although emails are no longer physical files, they remain as text.
An email consists of two main parts: the header and the message section, separated by a single null line (that is, a line that that only comprises a carriage return/line feed).
Nowadays, the message section can - and usually does - have a lot more structure associated with it thanks to the MIME (Multipurpose Internet Mail Extensions) standard. This standard extends the original seven-bit ASCII-only messages to incorporate other character sets including Unicode, attachments (usually encoded with something like Uuencode or base64) and multiple parts (where a message is encoded as pure text, HTML and rich text within the same email).
The header section remains resolutely ASCII (although MIME does allow for addressing with other character sets). It consists of various header information about the email, such as the subject, the recipient address(es), who sent it, a unique message ID, where replies should go to, and so on.
Email clients usually suppress most of this information when displaying an email, although there's usually a way to show them. Figure 1 shows an example header section from a recent email from the Association for Computing Machinery (ACM). Reading this you can see who sent it (and where to send the response to if I wanted to reply) and when it was sent.
The message itself is in a multipart MIME format (the line that defines the boundary between the parts is shown) - as it happens, the message is represented in both straight text format and in HTML within parts of the email and it's up to the email client as to which is actually displayed to the user.
Routing for emails
What also generally happens when an email is sent across the internet is that intermediary servers add extra routing information to the header section. For simplicity, this information is prepended to the header section, so the server doesn't have to hunt for the end of the section to add it.
The routing information generally details which email systems looked after and rerouted the email on its way to the inbox. For example, I've set up my personal email so that all messages are rerouted to Gmail, which means I can access my email easily using a browser or my phone.
The routing information included on the example email from Figure 1 shows (reading from the bottom upwards) the originating server name, the receipt by my email server at my personal domain, its sending on of the email to Google, the receipt by Gmail, and the final delivery to my inbox (see Figure 2).
By tracing the times shown on the routing information, I can see that the email appeared in my domain's inbox in a matter of seconds, whereas the automated Gmail fetch process took about 30 minutes. Although legitimate email servers will provide valid information as they prepend routing information, many others won't.
Spam emails especially tend to contain fake routing information, so you can't rely on this header information until the point when it reaches your email server.
How emails are sent
Having touched on routing for emails, we should take a look at what goes on when you hit 'Send' on an email message until the point when the recipient reads it in their email client.
The vast majority of email uses two types of server to send an email from A to B: the outgoing mail server and the incoming mail server. The outgoing email server is almost certainly an SMTP server (Simple Mail Transfer Protocol), while the incoming server can be a POP3 (Post Office Protocol) or IMAP server (Internet Mail Access Protocol).
When you set up your email client (let's say this is Microsoft Outlook, since that's what I use), you specify for it the address of your SMTP server. You also define to it the user ID and password that has been assigned to you to use the server's facilities (without a properly protected SMTP server, your email could be hijacked for spam broadcast purposes).
You write an email in Outlook, specify the recipient and press 'Send'. Outlook formats the message according the email standards (since 2008 this is defined in the RFC5322 document, which superseded RFC2822 from 2001, which in turn superseded RFC822 from 1982). It then connects to the SMTP server on port 25, passing the user id and password for authentication, and sends the email.
Once the SMTP server gets the email (and adds its routing information), it looks for the address to send it to within the header section. It strips off the username and the @ sign, leaving the domain name that the email must be sent to. The SMTP server queries the Domain Name System (DNS) for the MX (Mail eXchange) records for that domain name.
The DNS entry for a domain name consists of a set of records defining the addresses of servers that process various types of connection (there are A records, AAAA records, CNAME records, and so on), and the MX record defines the server that can receive emails for the domain. For example, with my personal domain, the A record currently points to 126.96.36.199. This is the IP address of the server that hosts my domain and my website.
My highest priority MX record (you can have several MX records and they are ordered according their priority, the order in which SMTP servers try to connect with them) is pointing to smtp. secureserver.net, the GoDaddy server that deals with my email. And, yes, your SMTP server then has to resolve secureserver.net to an IP address in order to continue.
You've got mail
Your SMTP server then sends your email to the recipient's MTP server using the Simple Mail Transfer Protocol. Of course, it may be that, due to unforeseen circumstances, my SMTP server is offline or down.
In this case, your SMTP server will put your email in a queue and try to send it again later. If the server finds that after several tries it can't send the email at all, it wraps the email in a 'cannot deliver' message and posts that to your email inbox. But let's assume that all goes well and my SMTP server receives your email (and adds its routing information). It in turn reads the recipient email address, works out the user name, and puts the email in my inbox.
By 'inbox', I don't mean the inbox in Outlook or whatever email client you use. I mean the inbox on the email server for my email address. In the old days, the inbox was very simple: it was a set of text files, one per email, in a folder named after my email address (or maybe a single text file and new emails were appended).
These days it's more integrated - the inbox is in a database, with the usual failsafe guarantees that provides. Incoming mail servers We now come to the opposite end of the email trail: the incoming mail server. Ignoring the heavy duty corporate email systems such as Microsoft Exchange, Lotus Notes or Blackberry Server, there are two main ones in use today: the POP3 server and the IMAP server.
POP3 is the older and less sophisticated of the two, but they both have roughly the same features. The main difference between them is what happens to the emails. With POP3, although you can leave emails on the server, there's no provision for marking any as read/ unread - the assumption is that emails are downloaded to your client and deleted from the server.
Of course, this presents a problem if you want to use a variety of clients to access your email, because you may find that a particular email that you want to read is on a different PC to the one you're currently using.
With IMAP, the assumption is the opposite: emails are left on the server and can be marked as read/unread. This means that you can access your emails through a variety of email clients (desktop, phone, web) and all clients will agree on the current state of the emails.
With IMAP you can also do things like set up an inbox folder tree on the server or move emails around the tree, and again the clients will all agree on the current state of the inbox.
Let's assume that I'm using POP3. Again, I will have configured Outlook so that my incoming server is at such and such address and has a particular user ID and password (I can't have all and sundry reading my emails after all).
When I ask Outlook to retrieve all my emails, it will log in to the POP3 server with the credentials I gave, ask for a list of emails, and then download and delete them one by one. It will read the header information from each email in order to ascertain how the message is structured, how the constituent parts are encoded, from whom the email came, the delivery date/time, and so on.
Outlook will then decode and display the emails for me to peruse and read, and with that we come to the end of the journey for that email, from your PC to mine.
First published in PC Plus Issue 304
Liked this? Then check out How sand is transformed into silicon chips
Get the oddest tech stories of the week, plus the most popular news and reviews delivered straight to your inbox. Sign up at http://www.techradar.com/register