As mentioned, Exchange has been around
for quite a while, and it has been adapted to the changing landscape of
email. Quite often, especially if they have skipped a few versions of
Exchange, people are shocked at how the storage recommendations for the
latest version have evolved. We find the best way to remedy this
situation is a brief walkthrough of the history of Exchange and how it
has evolved to meet changing demands. This helps us identify trends in
email usage and understand why Exchange has progressed as it has.
Exchange 4.0–5.5
These versions represented the very early days of
Exchange Server outside of Microsoft. Usage increased rapidly with the
release of Exchange Server 5.5, and many IT shops struggled to deal
with Exchange storage for the very first time. The most common approach
to Exchange storage design during this time frame was to fill the
server chassis with hard disk drives and then to create a RAID 5 array
for the large Exchange databases and a RAID 1 mirror to hold the log
files. If a performance bottleneck occurred, more Exchange 5.5 servers
were deployed and the workload was shared among the servers until the
end-user experience became acceptable.
The Exchange database schema in this time
frame was optimized to make extremely efficient use of free space on
hard disk drives because of their small capacity and very high cost.
The primary goal for Exchange versions 2000 and beyond was to improve
the performance of Exchange in order to take
better advantage of hardware and to better utilize the new clustering
technology to provide higher availability with fewer servers.
Exchange 2000–2003
Exchange 2000 allowed more and larger databases
on each server, but it retained the goal of making the best use of
available hard disk space. The enhancements brought with them new
challenges for Exchange storage design, such as having to provide a
backup solution for the additional data that Exchange could now store
and meeting the increased storage Input/Output Operations Per Second (IOPS)
performance requirements that were the result of being able to store
more mailboxes in each database on each Exchange server or cluster.
Microsoft also began to encourage
clustered high availability with Exchange 2000 and Windows 2000. This
brought with it the additional requirement for shared storage that
often resulted in a complex storage area network (SAN).
Exchange 2000 caused design teams and
storage vendors to deal with storage complexity for the very first
time, and many service delivery managers were forced to address the
inevitable consequences of operating a complex infrastructure—even one
that was allegedly highly available and that could never fail. When
things did go wrong, they typically did so with exquisite complexity.
Shared storage clustering required a
precise recipe of a particular Windows version, host bus adapter (HBA)
device drivers, HBA firmware, SAN fabric switch firmware, and SAN
storage firmware to remain even vaguely reliable in the early days.
Embarrassingly, it was not unheard of that customers would attempt to
implement such solutions in order to increase their Exchange service
availability, only to discover that they had actually made things worse
than with their previous Exchange Server 5.5 installations, and they
did so at great cost.
Exchange 2007
Exchange 2007 provided a solution that materially
improved service availability and performance at a much lower cost. It
was also capable of meeting the growing need for much larger mailbox
sizes.
Exchange 2007 brought about a radical change in storage technology. Continuous cluster replication (CCR)
was a new high-availability model that did not require a shared storage
solution. Combined with an I/O reduction of up to 70 percent (when
compared to Exchange Server 2003) and site resiliency technology, this
enhancement produced a paradigm shift in Exchange storage design.
The reduction in I/O and the introduction
of CCR meant that it was no longer necessary to deploy a shared storage
solution to achieve high availability. This allowed Exchange to make
use of less complex and, more crucially, less expensive storage
solutions than in previous Exchange versions. This not only reduced the
capital expenditure required for Exchange 2007 hardware, but it also
reduced deployment and operational costs due to the reduction in
storage complexity.
CCR also brought with it support for up to
200 GB databases. It also provided for two copies of each database plus
a third copy maintained by standby continuous replication (SCR)
in a secondary site. The introduction of multiple database copies
stored on physically independent storage resulted in a measurable
increase in Exchange service availability for most customers plus the
ability to provide end users with much larger mailboxes.
The downside of CCR was that design teams
that stuck with their expensive storage infrastructure often could not
justify the extra costs of deploying a CCR/SCR solution, or they had to
limit the size of mailboxes to keep storage costs down. In many cases,
customers kept using their SAN storage to take advantage of their
enterprise backup infrastructure and to get the ever-growing amount of
Exchange data backed up in an acceptable time frame.
For the first time, storage choice was
significantly affected by the type and success of Exchange
high-availability solutions. Customers staying with SAN storage were
mostly using the legacy single copy cluster (SCC) that was
available in previous versions of Exchange, while customers using
cheaper directly attached storage were generally adopting the new
CCR/SCR technology and benefitting from improved availability and
reduced costs. In extreme cases, customers who had chosen expensive SAN
storage, but who still wanted to use CCR, would store both the active
and passive copies on the same storage array. This added both cost and
complexity, but did not improve service availability.
Even though IOPS had been reduced by 70
percent from Exchange 2003, the IOPS requirements for Exchange Server
2007 were still relatively high and most deployments were still using
10K rpm disk spindles in RAID 10 to meet the combination of performance
and capacity.
Exchange 2010
Exchange Server 2010 made better use of cheap,
directly attached storage in order to entice more enterprise customers
into using the new continuous replication cluster technology. Exchange
Server 2010 matured the CCR/SCR model in database availability groups (DAGs).
A DAG allowed up to 16 copies of each database and enabled each
database to have multiple database copies. IOPS requirements were
reduced an additional 70 percent, and the I/O profile for Exchange was
modified to make much better use of cheaper 7.2K rpm hard disk drives.
With Exchange Server 2010, for the first
time customers could also choose to design Exchange storage to use 7.2K
rpm SAS hard drives with no RAID JBOD (just a bunch of disks).
They also could now rely entirely on the Exchange continuous
replication solution (DAG) to provide database availability.
Being able to use cheaper and slower 7.2K
rpm hard disk drives came at a cost. The database schema and internal
tables within the database were optimized to improve I/O performance by
organizing items in a sequential manner. This change caused
single-instance storage to be lost and Exchange to prefer sequential
positioning of data to making use of every tiny morsel of free space
within the database. The impact of this change was that Exchange
databases were now larger than in previous versions despite the
Exchange product group having introduced database compression that was
intended to compensate for the growth.
COMPRESSION
It is worth noting that in the absence of compression, the database sizes increased by around 25 percent.
Design teams now had more flexibility than
ever before because SAN, DAS, and JBOD were all viable options
depending on the solution requirements. The storage and database copy
layout flexibility provided by Exchange DAGs complicated the Exchange
storage design process. Design teams began to ask the obvious questions:
- Is JBOD viable for enterprise?
- How many copies are needed?
- Are backups necessary?
- Can we use cheap SATA disks?
The increased flexibility provided by the
database availability group model placed increasing importance on a
good design process and requirements definition. With Exchange Server
5.5, storage choices were limited, and thus it was difficult to get it
wrong. With Exchange Server 2010, the available storage choices ranged
from expensive SAN storage arrays with multiple parity, right down to
cheap, locally attached JBOD 7.2K SAS spindles.
For customers who did adopt the simple and
cheap JBOD model, one problem became increasingly apparent. JBOD disk
spindles were increasing in size, with some manufacturers listing 3
TB–4 TB disk spindles in 2012. Yet the maximum recommended Exchange
2010 database size was 2 TB.
The IEEE suggests that mechanical
disk platter areal density will increase 20 percent per year for the
rest of the decade. Both Hitachi and IBM are predicting that 6 TB hard
disks will be commonly available during 2013. In contrast, random IOPS
performance for mechanical hard disks has remained almost static over
the last 10 years. Random IOPS performance is mostly governed by hard
drive rotation speed; however, increases in rotational speed also bring
with them large increases in cost and power. The most common format for
capacity and performance in Exchange 2010 storage is 7.2K rpm 3.5″.
These disk drives vary in performance from 50–65 random IOPS per
spindle depending on the manufacturer. There is no expectation that
IOPS per-spindle performance will increase in the near future, and so
Exchange Server 2013 must be able to make use of these larger capacity
disk spindles.
