Troubleshooting Exchange Server 2010 : Troubleshooting Mail Flow (part 1) - Queue Viewer in the EMC

Message delivery is arguably the most important piece of Exchange Server 2010, and it's only fitting that Microsoft has provided a formidable arsenal of troubleshooting weapons to deal with pesky delivery failures. You'll have your pick of tools, from self-serve ones, such as message tracking in the Exchange Control Panel, to several forms of tracing, to the inevitable cmdlets.

However, just because you have an array of choices doesn't mean you have to use them right away. Again, it's important to approach a message delivery problem with clear eyes and ask probing questions about what you're facing. Remember the example earlier in the chapter, with the end user who can't send email? There were a number of plausible explanations for this, some of which didn't involve message delivery at all! So it's still important to gather the essential information:

• Can the user send any emails at all? Is nondelivery restricted to a subset of users?

• Does the user receive a delivery status notification? If so, what is the delivery code?

• Is the recipient in the same Exchange organization or in a different organization (presumably on the Internet)?

• How close do messages get to their destination?

• What is the messaging path between the end user and the recipient?

These questions, though relatively simple, conceal a bewildering list of possible root causes. Consider the impact on message delivery on the following:

DNS Failure

Hub Transport servers can't locate A records and therefore can't reach next-hop servers.

Site Link Failure

No site link exists between sender and recipient.

Transport Failure

All of the Hub Transport servers in the user's site are inaccessible.

Transport Agent

A transport rule prevents this email from reaching the recipient (either because of sender restrictions, content restrictions, or recipient issues).

Mailbox Limits

The recipient's mailbox is full, but nondelivery reports do not reach the sender for whatever reason.

Messages Stuck in Queue

A transient failure has temporarily stopped messages at a back-off location.

Back Pressure

A Hub Transport server is temporarily throttling message delivery due to resource constraints.

This isn't even an exhaustive list, but it includes a wealth of possibilities. Now, there are few listed here that you would probably detect by performing the basic troubleshooting steps we covered earlier in this chapter (like DNS failure or transport failure). We'll begin with a simple cmdlet to check basic mail flow, which is typically the first step in locating undelivered messages, and then move on to message tracking and agent logging.

1. Using Test-Mailflow

Assuming you've done some of the basic checking (is the user's client connected to a database, are Hub Transports available, and so on), you'll probably want to test that mail is flowing in the organization. There's an aptly named cmdlet for just this job: Test-Mailflow. You could argue that Test-Mailflow doesn't belong in this section, since it's ostensibly testing Mailbox servers. But it's listed as a Transport cmdlet in the Exchange Server 2010 help file, and when you get right down to it, mail flow is the Hub Transport's responsibility.

The cmdlet's basic function is simply to send and receive email from the system mailbox of the target server, but it can do so much more. The syntax is extremely simple: Test-Mailflow followed by the source server, then–TargetMailboxServer, -TargetDatabase, or –TargetEmailAddress. The different options mean that you can start with a Mailbox server, and if that test succeeds, focus on the user's database and then the user's email address. If you've deployed multiple databases in a DAG, you should skip the first step and start with –TargetDatabase.

Test-Mailflow hnlmbx05 -TargetEmailAddress [email protected]

RunspaceId         : 0848e0b8-4228-4195-b1ee-c4c967ac9a41
TestMailflowResult : Success
MessageLatencyTime : 00:00:02.5631250
IsRemoteTest       : True
Identity           :
IsValid            : True

The output from Test-Mailflow doesn't need much interpretation. The most important piece is the TestMailflowResult property. If it reads Success, you know that you can reach that server, database, or email address, and you know that email is flowing, at least for some combination of user and database. The next property, MessageLatencyTime, lets you know the time it took for the message to reach the destination server. The IsRemoteTest property simply indicates whether the message left the server (this will also be True if you use the –TargetEmailAddress parameter).

However, if your test fails, the TestMailflowResult property reads *FAILURE*, and that's unfortunately the only indicator you receive—no messages about where the failure might have occurred, or other useful information. That's when you need to start figuring out where the messages are stopped, and for that we need to move into different tools. We'll start with the Queue Viewer and then look into message tracking.

2. Queue Viewer in the EMC

The Queue Viewer is located in the Toolbox of the EMC, alongside a number of other useful tools (some of which we'll cover later in this section). A big believer in truth in advertising, the Queue Viewer allows you to, yes, view the contents of the various delivery queues. Obviously, you need to connect to a Hub Transport server to use this tool, but you can open the Queue Viewer from anywhere and then connect to the appropriate Hub Transport server. The interface for the Queue Viewer is shown in Figure 1.

Figure 1. Using the Queue Viewer interface

It's largely unchanged from Exchange Server 2007, but it didn't need any enhancements; it tells you the status of the queues, how many messages are pending delivery, and where the mail is heading, among other things. The list of queues is a bit thinner than in previous versions of Exchange (particularly when compared to the positively garrulous Exchange Server 2003), but if there's a problem with a particular queue, it'll be listed here.

Most of the columns in the Queue tab are relatively self-explanatory, but here's a brief rundown of what you'll see on this page:

Next Hop Domain

This is where the mail is heading next, whether a Hub Transport in a different site, a Mailbox server in the same site, or an Internet host. You'll also see a Submission queue, which obviously represents incoming mail submission to that Hub Transport.

Delivery Type

This indicates where the messages are heading next in their journey to the recipient.

Status

This simply indicates whether the queue is Active (sending messages at the moment), Suspended (stopped through administrative action), Ready (able to send messages should any arrive), or Retry (unable to send messages). Queues in a Retry state are the most obvious candidates for additional review and analysis, but remember that queues can fail because of the sending server as well as the recipient.

Message Count

This lets you know how many messages are stuck in this queue.

Next Retry Time

This is only applicable to queues in a Retry state, and lets you know the next time Exchange will attempt to "wake up" the queue for delivery.

Last Error

This is the Hub Transport's way of letting you know what might have caused a particular issue: was it an authentication issue, a DNS lookup failure, or simple network connectivity problems?

You can perform a small number of tasks on the queues, including temporarily stopping them (Suspend), forcing them to connect if they've failed (Retry), or deleting all the messages (Remove messages with or without NDR). This can be useful for restarting mail flow after you've resolved a problem somewhere else in the environment, or deleting a quantity of undesired email. The Actions pane at the right of the EMC will display only valid actions for the queue you've selected.

The Messages tab has similar information to the Queues tab, and it's generally most useful when you've clicked a queue and then selected View Messages. You can click the Messages tab right away, but it'll show you every message queued on the server, which might take a little while for a busy system. The columns for the Messages tab are similar to that on the Queues tab:

From Address

This is the address of the sender, taken directly from the SMTP envelope.

Status

This indicates the message status, which is generally the same as the parent queue but is also influenced by administrator action (for example, if the administrator has tried to delete the message while it was being delivered, the message will appear as Pending Remove).

Size (KB)

This is the size of the message, displayed in kilobytes.

SCL

This is the spam confidence level (SCL) rating; the values range from −1 through 9, with −1 representing authenticated email and 9 representing email that is almost certainly unsolicited commercial email (UCE, or spam).

Queue ID

This value indicates the queue in which the message appears. If you chose a specific queue and then selected View Messages, this should be the same for all messages and should reflect the queue you chose on the Queues tab.

Message Source Name

This indicates the Exchange component that delivered the message to this particular queue. Depending on your architecture, this could be a Hub Transport server in another site, a Mailbox server in the same site, or possibly even an application or client submitting a message directly to the Hub Transport via SMTP.

Subject

This is the subject of the email, taken from the SMTP envelope.

Last Error

This indicates the last error experienced when attempting delivery of this message. This typically only appears if the message is in a Suspended, Retry, or Pending state.

There are a couple of good articles on the technical details of the Queue Viewer at http://technet.microsoft.com/en-us/library/bb629568.aspx and http://technet.microsoft.com/en-us/library/bb676413.aspx.

The Queue Viewer is useful for locating a message that hasn't been delivered, but the message-tracking feature is also useful for this in larger environments, and depending on the Last Error field for the queue or message in question, you may be able to figure out what your next move should be. However, the one drawback to the Queue Viewer is that, unless you have a simple topology, you might not necessarily know exactly how a message reached that particular server. For that type of analysis, we need something a little more detailed (like message tracking, which we'll cover in a moment as well).