Exchange Server 2010 : Standards and Protocols - Other Key Technologies Used by Exchange

1. Moving Messages with the Simple Mail Transport Protocol

Although you could argue that the mailbox is the most important part of a mail system — that's where messages are stored, after all — we have always held a different opinion. Without the ability to send and receive messages, a messaging system is worthless. To us, the message transfer agent (MTA) is the living heart and soul of any email system. With modern systems, the key inter-system transport is the Simple Mail Transport Protocol (SMTP), and Exchange 2010 is no different.

The original versions of Exchange were built around the International Telecommunication Union (ITU) X.400 standard as part of the OSI network protocol family. While Exchange used X.400 over TCP/IP as the primary transport to other organizations, internally it used a variant optimized for Exchange-to-Exchange server communications. These first versions of Exchange offered SMTP as a foreign connector service for messaging systems on the Internet, requiring messages to be converted from OSI message formatting to the equivalent Internet formats. These conversions were complicated and often a source of problems, because the various properties didn't map exactly.

With Exchange 2000, Microsoft made crucial changes to the architecture of Exchange organizations, including a switch from the proprietary X.400 variant to SMTP. There are two advantages to using SMTP as the internal transport:

• As the name SMTP implies, SMTP is a simple protocol. It is less complicated, and therefore much easier to implement, than X.400. As organizations started using Exchange 2000, the vast majority of Exchange organizations were transmitting messages with other Internet-connected organizations using SMTP, so in most cases, there was a performance advantage for removing the layers of message conversion.

• SMTP is an open framework that allows additional vendor extensions to be easily added. This was important, because Exchange still required a large amount of additional information and functionality not present in standard SMTP. Exchange 2000 servers, therefore, speak standard SMTP to other mail servers, but a special superset of SMTP to other Exchange servers in the organization.

The X.400-based transport wasn't totally removed at this time, however; it still existed as the X.400 connector, intended to allow connections both to external X.400 organizations as well as legacy Exchange organizations. Since Exchange internally tracked both sets of properties, these connectors no longer required the same degree of message conversion.

In each successive release of Exchange, additional work has been done to implement more of the standard SMTP extensions and remove dependency on the X.400 transport code except for interoperability with previous versions. In Exchange 2007, the legacy X.400 transport code was removed, as were many of the legacy Exchange-specific SMTP extensions. Exchange 2007 and 2010 use standard technologies such as Transport Layer Security (TLS) and Kerberos to secure SMTP sessions.

Table 1 summarizes some of the important mail transport RFCs you should know as an Exchange 2010 administrator or professional.

Part of what makes SMTP so relatively simple is that it isn't based on any complicated transport schemes; at its heart, it is a store-and-forward system. Once an SMTP system accepts responsibility for a message, it attempts to deliver it to the next best hop that it knows about. An SMTP connection, in its simplest form, is easy to understand. It involves just two systems: the client, which opens the connection so that it can submit email, and the server, which accepts inbound connections and determines if it can accept the message that has been submitted.

This is what a typical session, sending a message from [email protected] to [email protected], would look like:

{The client connects to the server}
01 S: 220 smtp.server.tld ESMTP mail system ready
    C: HELO desktop.client.tld
    S: 250 Hello desktop.client.tld, I am glad to meet you
02 C: MAIL FROM:<[email protected]>
    S: 250 Sender ok
03 C: RCPT TO:<[email protected]>
    S: 250 Recipient ok
04 C: DATA
    S: 354 End data with <CR><LF>.<CR><LF>
05 C: From: "Client Sender" <[email protected]>
    C: To: Another User <[email protected]>
    C: Date: Wed, 22 Oct 2008 01:13:22 -0800
    C: Subject: Test message
06 C:
    C: Isn't SMTP easy?
07 C: .
    S: 250 Ok: queued as 918273645
08 C: QUIT
    S: 221 Bye
    {The server closes the connection}

This sample conversation uses a simple subset of SMTP; there is no authentication, no session security — none of the extensions you would see in a real session. Having said that, though, there are still eight key points of note in this conversation:

If the server were not the final destination for this message — perhaps it's just an edge mail system, handling all interaction with the Internet and performing message hygiene functions before routing accepted messages further into the organization — it would pass the system on to the next hop. Note that SMTP only handles message flow from the MUA through the MTA or MSA, and from one MTA to another MTA; it is not used to transfer messages to an MDA or LDA, or from the user's mailbox database back to the MUA or MRA.

2. Accessing Mailboxes with Message Access Protocols

As we just mentioned, SMTP is designed for message transport — getting a message from one system to another. Once the message has been received and delivered to a mailbox store, there's one final hop that needs to happen: to the user's MUA. SMTP would be a poor protocol to handle this function — it has no concept of mailboxes or folders — so alternate protocols must be used.

2.1. The Messaging Application Programming Interface

The Messaging Application Programming Interface (MAPI) is a programming interface that allows a developer to more easily write an application that accesses email or directory functions and services. Though often considered an industry standard, MAPI was developed by Microsoft in the early 1990s and the API set is published so that anyone can use it.

Outlook, for example, uses MAPI to access data on the Exchange server as well as accessing directory information that is stored in the Active Directory. An underlying directory service provider and email service provider allows MAPI to access these systems. Outlook includes an underlying service provider for Exchange Server as well as a service provider that allows access to PST files and IMAP4/POP3 servers. Third parties have developed service providers that will allow Outlook to access other messaging systems such as Lotus Notes, HP OpenMail, Hotmail, and even Zimbra.

A number of variations and versions of MAPI have been released over the years as new messaging functionality has been developed. To communicate with a client/server messaging system such as Exchange Server, MAPI relies on remote procedure calls (RPCs) to transport MAPI requests and data between the client and the server. These RPCs by default travel over a native TCP stream to form the MAPI-RPC protocol and use the User Datagram Protocol (UDP) to provide notifications of new mail. This protocol is used by Outlook when it is on the local network and not configured to use Outlook Anywhere. When Outlook Anywhere is used, MAPI is encapsulated within HTTP/HTTPS and transported from the Outlook client to the Client Access server.

MAPI-RPC over TCP has several characteristics that make it less than suitable for use outside of firewalls, or in networks that are heavily segmented. To understand why, you need to understand a bit about how RPCs work. A single Windows server may have many different services that all use RPCs and each listen on a different. One example of this is the Client Access server's Microsoft Exchange RPC Client Access service. This is the service to which an Outlook client connects in order to access mailbox data. When this service starts, it dynamically allocates an unused TCP port above 1,024 for that function.

If the port is dynamic, how does the Outlook client discover to which port it should connect for mailbox access? Well, the Microsoft Exchange RPC Client Access service also registered a particular application identifier with the RPC End Point Mapper service along with the port that it is using. The Outlook client first contacts the RPC End Point Mapper (always on TCP port 135) and inquires as to which port the information store service is listening. Once the information store port is determined, Outlook can connect to the correct port number. This is, of course, simplified, but it gives you an idea of how Outlook determines the correct port numbers to which it must connect on the Client Access server.

MAPI-RPC over TCP uses the same RPC framework (MS-RPC) as several other Microsoft protocols, including many of the management protocols used by various MMC administration consoles and the SMB file sharing protocol. As a consequence, opening the firewall to allow MAPI-RPC also opens many other protocols for access.

While there are several well-known ports used for initial connections, the actual connections are negotiated on the fly using an upper port range. Without a well-known port for all traffic, firewalls must either understand the protocol and perform application-level proxying or be configured to open the entire dynamic port range.

To securely deploy MAPI-RPC outside of the firewall, you need to tunnel the RPCs over another transport protocol. The Outlook Anywhere protocol uses Secure Hypertext Transport Protocol (HTTPS) as a secure, single-port transport for the RPC connection between Outlook and the Exchange servers. If you would like detailed information on how RPC is encapsulated in HTTP, take a look at this article:

http://msdn.microsoft.com/en-us/library/aa379169(VS.85).aspx

2.2. HTTP and Exchange Server

The HTTP component has gone from being almost an afterthought in early versions of Exchange to a critical component for all Exchange servers. It provides web clients with access to their mailbox and to the directory, as well as a number of additional Exchange Server components. The following is a list of services and functionality that are now provided via HTTP:

• Outlook Web App provides web browser clients with access to mailbox data (email, calendar, contacts, tasks), public folders, and address lists.

• Exchange ActiveSync provides ActiveSync-enabled mobile phones, such as Windows Mobile phones and the iPhone, with access to mailbox data (email, calendar, contacts) and access to the global address list.

• Outlook MAPI clients can encapsulate MAPI requests into HTTP and send them to the Client Access server, where they are de-encapsulated and sent on to Mailbox servers or the Active Directory.

• HTTP provides the PowerShell with the connectivity it needs between Exchange servers to use the Remote PowerShell.

• The Exchange Web Services API is used for services such as Autodiscover and downloading the offline address book as well as being used to access, manage, and manipulate Exchange data.

2.3. POP3 and IMAP

POP3, the third version of the Post Office Protocol, is intended to be a simple, no-frills mailbox retrieval protocol. A POP3 client connects to the Client Access server, provides the user's credentials, gets a list of messages that have arrived in the user's Inbox folder, and downloads one or more of them to the local client. POP3 is simple and lightweight to implement, ensuring that just about every email client application supports it, but it has two main flaws in a traditional business setting:

• POP3 doesn't understand the concept of folders. POP3 will only pull messages from the Inbox; if you have server-side rules that move messages to alternate folders (or have moved them manually, perhaps through a web-based mail client), they will not be visible to POP3. Once messages have been downloaded to the MUA, they can be filed in any local folders.

• Both POP3 and IMAP are supported by Exchange 2010 out of the box, but they're not configured by default. Also, both require the users to have access to an SMTP server to submit any new messages or replies to; neither POP3 nor IMAP handles submitting new messages back from the MUA to the server. They're strictly pull protocols and require a push protocol like SMTP to ensure new messages get delivered.

Ensuring that clients have access to a suitable SMTP port in modern organizations can be a substantial challenge, especially when your users are outside of your network and traveling. Many ISPs block outbound connections on TCP port 25, the port used by SMTP, unless these connections come from their own mail servers. Additionally, because of SPF and Sender ID policies, regulatory compliance policies, or even message hygiene regimes that require all outbound traffic from the domain to be scanned through specific gateways, you may need to ensure that all message clients send their outgoing messages through your Exchange servers only.

There are three main solutions for these problems:

Use clients that don't require SMTP.

We are compelled by union regulations (that's a joke) to point out that Outlook in Exchange mode, Outlook Anywhere (formerly known as RPC over HTTPS), Outlook Web App, and Exchange ActiveSync all receive and submit messages over the same non-SMTP connections and thus avoid these problems. If you can restrict your users to using one of these options, or something that uses a similar transport (such as Exchange Web Services over HTTPS or the latest version of Microsoft Entourage for Mac OS X), then you can sidestep the whole issue. We like this solution because we think that Exchange users, by and large, are going to have the best user experience with one of these clients, especially in Exchange 2010. However, there are times where you have to support other clients.

Use a virtual private network.

By requiring your clients to use a VPN connection to access their messages, you can ensure (and enforce) that they are connecting to your Exchange servers. This approach also has the advantage of allowing you to not publish POP3 or IMAP services outside of your firewall. However, traveling users may be on networks that don't support VPNs, leaving them without reliable email access.

Use the client submission port.

Starting with many Unix messaging systems, administrators began to separate server-to-server mail traffic from client-to-server mail traffic. They did this by reserving TCP 25 for standard server-to-server SMTP sessions as well as opening TCP 587 for clients to submit messages to their mail servers, as shown in Figure 1. By the time Exchange 2007 came about, the submission port was a de facto standard and was supported out of the box. Exchange 2010 continues supporting this standard.

Figure 1. The client submission port

3. Securing Network Sessions with TLS and SSL

The final notable protocol you need to understand at more than a basic level helps keep your network sessions secure through the use of sophisticated cryptography. Transport Level Security (TLS) and its little brother Secure Socket Layer (SSL) are client-server protocols that provide encrypted transport sessions for network connections, as well as X.509 digital certificate-based authentication of the server identity through a trusted certificate authority. Optionally, they also allow for mutual authentication, enabling both the server and client to authenticate each other's identity.

Most of us are familiar with at least the basics of how SSL operates from years of using web browsers with HTTPS. For the most part, TLS and SSL are indistinguishable, certainly from the level of detail you'll need for day-to-day operations deploying, managing, and troubleshooting Exchange issues. Here is the big difference between TLS and SSL:

A SSL session is negotiated and established at the beginning of the connection, before the actual application or service responds to the client.

In order to offer both a secured and non-secured version of a given protocol using SSL, you must expose two ports: the insecure version (HTTP on TCP 80) and the secure version (HTTPS on TCP 443).

A TLS session is negotiated and established after the server has responded to the client, as an in-band option within the protocol being exchanged.

This allows you to offer the secure session as a protocol-level option over a single port, while still using application-level configu-ration options to enforce how that security will be used.

This distinction is illustrated in Figure 2, which shows the difference between SMTP using TLS and SMTPS (Secure SMTP).

Figure 2. SMTP using TLS versus

TLS and SSL are used by default for almost all connections in Exchange 2010.

Out of the box, all Exchange servers use TLS to protect SMTP connections and SSL to protect HTTP connections between systems. They also offer these mechanisms for client connections and in the default configuration require TLS/SSL connections before allowing potentially risky operations (such as the use of Basic plaintext authentication).

TLS and SSL require a working knowledge of (and proficiency with) digital certificates.

Although Exchange 2010 follows the example of Exchange 2007 and includes self-signed certificates, these self-signed certificates are only trusted by other Exchange servers in the organization that have other authentication mechanisms such as Kerberos at their disposal. The principals and mechanisms of request and deploying certificates are fairly simple; the complexity comes only in trying to figure out which names you need.