Windows Server 2012 : Continuous availability (part 3) - Failover Clustering enhancements - VM priority

Failover Clustering placement policies for Hyper-V

Windows Server Failover Clustering provides a critical piece of Hyper-V infrastructure not just for high availability, but also for mobility. A key concept of a virtualized or private cloud environment is to abstract workloads from their underlying physical resources, and Failover Clustering enables this by allowing the movement and placement of VMs between different physical hosts using live migration with no perceived downtime. There are a few placement best practices that can allow you to optimize the cluster for different Hyper-V scenarios.

Default failover policy

When there is a failure of a node, VMs are distributed across the remaining cluster nodes. In previous versions of Windows Server, any resource would be distributed to the nodes hosting the fewest number of VMs. In Windows Server 2012, enhancements in this logic have been made to redistribute the VMs based on the most commonly constrained resource, host memory. Each VM is placed on the node with the freest memory resources, and the memory requirements are evaluated on a per-VM basis, including checks to see if the VM is NUMA-aware.

If a cluster node hosting several VMs crashes, the Cluster Service will find the highest-priority VM, then look across the remaining nodes to determine which node currently has the freest memory. The VM is then started on that node. This process repeats for all the VMs, from the highest priority to the lowest priority, until all VMs are placed.

VM Priority

In Windows Server 2012, each VM running on a cluster can be assigned a priority: High, Medium, or Low. This can be used to ensure that the high-priority VMs are given preferential treatment for cluster operations. This could be used to ensure that the organization’s most critical services or key infrastructure roles can come online before less important workloads.

If a cluster node hosting several VMs crashes, the high-priority VMs will start first, then the medium-priority VMs, then finally the low-priority ones. This same logic will be applied for other cluster operations, such as multiple live migrations or Node Maintenance Mode, where the high-priority VMs will always be moved first.

Preferred Owners

From earlier versions of Windows Server, it has been possible to configure the preference for node failover order for each VM. This can be helpful in an environment where it is important for certain VMs to stay on certain nodes, such as if there is a primary datacenter where the VMs should usually run (the Preferred Owners), and a backup datacenter available for a disaster recovery for the VMs if the primary site is unavailable.

If a cluster node hosting several VMs crashes, a high-priority VMs will attempt to move to the first node in the list of Preferred Owners. If that node is not available, then the VM will attempt to move to the second node in the Preferred Owners list. If none of those Preferred Owners are available, then it will move to the first node that is on the Possible Owners list.

Possible Owners

The Possible Owners setting for each VM also existed in earlier versions of Windows Server. It enables VMs to move to and start on a cluster node when none of the Preferred Owners are available. This can be used in an environment when VMs should still run on a host, even when none of the Preferred Owners are available. In a multisite cluster, the nodes at the backup site would be assigned as a Possible Owner, but not as a Preferred Owner. In this scenario, the VMs would fail over to the secondary site only when none of the nodes at the primary site (Preferred Owners) are available.

If a cluster node hosting several VMs crashes, a high-priority VMs will attempt to move to the first node in the list of Preferred Owners. If none of those Preferred Owners are available, then it will move to the first node that is on the Possible Owners list. If the first node in the Possible Owners list is not available, then it will move to the next node on the list. If none of the nodes in either the Preferred Owners nor Possible Owners lists are available, then the VM will move to any other node, but remain offline. Depending on Failback policies, the VM can move back to a Preferred Owner or Possible Owner and start as soon as one of those nodes becomes available.

Failback

Another setting for each VM that continues to be important in Windows Server 2012 is the option to move the VM back to Preferred Owners or Possible Owners, starting from the most Preferred Owner. This feature is helpful if you wish to keep certain VMs on the same hosts, and return those VMs to the host once it recovers from a crash.

If a cluster node recovers from a crash and rejoins cluster membership, any VMs that are not running on a Preferred Owner will be notified that this node is now available for placement. Starting with the high-priority VMs that are running on a Possible Owner (or are offline on another node), each VM will determine if this node is a better host, then live-migrate (or start) the VM on that Preferred Owner.

Persistent Mode

One problem that is often seen in highly virtualized environments is a “boot storm,” which happens when simultaneously starting a large number of VMs. Starting a VM requires more host resources than standard running operations, so starting a lot of VMs can sometimes overload the host, affecting its performance, or even causing it to crash (if certain host reserves are not set). As a safety precaution, during failover or when a node is restarted, the number of VMs that will start simultaneously is limited (High priority first), and the rest will be queued up to start on that node. Even when these VMs are simultaneously starting, they are slightly staggered to help spread out the demands on the host. There are still some settings that can be configured to avoid these “boot storms.”

Persistent Mode was introduced in Windows Server 2008 R2 and provides the ability to keep a VM on the last host it was deliberately placed on (either by an administrator or a System Center Virtual Machine Manager placement policy). If an entire cluster crashes, each VM will wait for the node is was previously hosted on to come online before starting up, still honoring high-priority VMs first. This prevents all of the VMs across the cluster from trying to start up on the first node(s) that come online, helping to avoid a “boot storm.” There is a default amount of time the cluster service will wait for the original node to rejoin the cluster. If the node does not join within this period, the VM will be placed on the most Preferred Owner, ensuring that the VM will still come online, while having given that new host an opportunity to start its own VMs.

Auto-Start

There may be cases when there are unimportant VMs that should not be started after a cluster failover or a crash, giving the other VMs an opportunity to fail over and come online quickly. The Auto-Start property has also existed in previous versions of Windows Server, and if it is disabled, the VM will not be automatically started when it is placed on a node.

This can be useful in highly virtualized environments when it is important to keep hosts and critical infrastructure VMs running, while not worrying about constraining resources or “boot storms” caused by VMs that do not need to be continually available, yet are still hosted on the cluster. These VMs can be started later by the administrator or automatically using a script.

Anti-Affinity

The final placement policy has also existed before Windows Server 2012, but looks at other VMs, rather than the hosts. The cluster property, AntiAffinityClassName (AACN), enables custom tagging of a VM so that different VMs may share or have different AACNs. VMs that share the same AACN will distribute themselves across different hosts automatically. This can be useful to separate tenets or VMs with the same infrastructure roles across different nodes in the cluster. For example, having all the virtualized DNS servers or guest cluster nodes on the same host would be a single point of failure if that node crashes, so spreading these VMs out across different hosts helps maintain continual service availability.

If there is a cluster with four nodes and four VMs that have the AntiAffinityClassName of “blue,” then by default, each node would host one of the “blue” VMs. If there are more “blue” VMs with the same AACN than there are nodes in the cluster, then there will be more than 1 “blue” VM on each node, but they will still distribute themselves as evenly as possible.

Conclusion

Using these policies, it is possible to optimize the placement of VMs on a Windows Server 2012 Failover Cluster. Always remember to configure the priority to the VMs so that high-priority VMs are placed first, and consider how VM placement will look when any one of the nodes becomes unavailable.