Deploying Nutanix on Lenovo HX servers with Mellanox SX1012 ethernet switches

rack-rear

First off, I want to say that I’m extremely pleased with the Mellanox switches. They were a real treat to deploy and configure. Simple, with 3 ways to access them out of the box, web interface, ssh and serial console. The SX1012 switches are 1U half width with 12 QFSP ports per switch. Each port is capable of 40Gbps.

The Lenovo HX servers were not what I expected from an HCI solution. Granted, they are high quality 2U rackmount servers, but they are still just that. However, where they lack in space savings, they make up for in connectivity options.

Each server had 2 x intel x520 dual 10G SFP+ adapters, giving them 40Gbps per server for data traffic, 4 x 1G RJ45 connections for legacy connectivity a 1 x IMM connection for out of band management.

The HX nodes were shipped with Nutanix Acropolis Hypervisor pre-installed, so it was very simple to get started by accessing Nutanix Foundation on the CVMs.

[For those that do not know, Nutanix foundation is a web based interface for configuring your out of band management IPs, deploying your hypervisor, deploying the management VMs, creating the clustered storage, assigning IPs and access credentials to the hosts, and configuring all networking. This is done in a 5 minute configuration process, then about 30-45 mins of waiting while everything gets deployed. The CVMs, which are VMs running on the host have an instance of foundation running on them. Another option for deployment is to run foundation on a standalone VM on a laptop.]

The first thing we had to do was to determine our network connectivity. This was being deployed in a brownfield environment, so there was existing servers, switches, etc. We needed to find the best way to connect the new environment to the existing and migrate all the data over.

One problem we had was that there was no 10G SFP+ connection on the legacy HP switches so the QSFP to SFP+ cable would not work in this instance. We had to come up with an alternative method. We were able to use QSFP to SFP adapters which then allowed for the 1G connectivity.

The Mellanox switches are small, but with them you can use what is called a Hybrid cable which is very cool. It takes a single 40G QSFP port and splits it into 4 x 10G ports. So each 12 port half width switch could be a 48 port 10G switch, just by using these cables.

download (2)The cabling was minimal and connectivity is easy. We had 4 x HX nodes, so the cable was perfect for us. Each switch had one QSFP to 4x SFP+ hybrid cable, which connected to each server. This allowed us to have PCIe diversity for the connections, switch diversity and serious bandwidth.

We also decided to create a 2 port QSFP LAG between switches, which gave us 80G inter-switch connectivity.

For connectivity into the legacy storage system, we connected two of the 1G ports from one of the nodes into the pre-existing iSCSI switch. Later on, we will connect to the iSCSI target from the ESXi software initiator.

We opted to use the VM version of Nutanix Foundation for reasons I won’t get into in this post, but the embedded would have worked as well. It is important to note that when deploying the HX nodes, you have to use the HX version of Foundation.

This can be obtained from the Nutanix partner portal, under downloads > foundation.

HX-foundation

You will also need to download the latest AOS version, which is what runs on the CVMs.

This can be obtained from the Nutanix partner portal, under downloads > AOS.

AOS

Lastly, if you are running ESXi, then you will need the Lenovo custom ISO image.

This can be obtained from the my.vmware.com download page. Go to download ESXi, then switch to the “Custom ISOs” tab.

Lenovo_ESXI

 

I got the foundation VM from the partner portal as well.

f-12

Extract it and use VMware workstation or Fusion to run the VM. Power it on and log in.

f-1.pngNow set the IP address.

f-2.png

Now that we have all the required software bits, put them into the proper foundation folders. The easiest way is to scp the files to the folders on the Foundation VM.  For windows use WinSCP, for OSX just use the scp command in the CLI.

The directories are:

For the custom HX foundation package

/home/nutanix/

Then extract the tar.gz file and it will create the foundation folder on the desktop

For AOS:

/home/nutanix/foundation/nos/

For ESXi:

/home/nutanix/foundation/hypervisor/

After the files are in the right directories, then open firefox in the VM and run the setup.

f-3

If you have different networks for out of band management and hypervisor management, and have routing setup, then just enter the top part of the global config. If you have multiple networks but have not configured routing yet, then you can multi-home the VM nic and use a flat switch for the hosts. In this case, configure the multi-home section as well.

It is important to note that the IPMI user and pass are different for Lenovo HX nodes.

Use the following:

Username: USERID

Password: PASSW0RD

The o in password is actually the number zero. It is also case sensitive.

Then click next.

If it automatically detects the available nodes, then great. If not, then manually add the blocks by clicking on add blocks on the top right.

f-4.png

Now add the number of blocks and nodes.

f-5.png

Enter in the MAC addresses of the IPMI nics. If you don’t know these, then you can get them in the BIOS, under “network configuration” then look for burned-in MAC address.

f-13.png

In the top of the foundation page, configure the IPMI, Hypervisor and CVM IP ranges. Just enter in the first IP and it will auto-populate the rest sequentially. The same will go the the hostname. Just specify the prefix in “hypervisor hostname”.

f-6.png

Now in the next page, it will show the AOS package and ESXi ISO that you uploaded via scp earlier.

f-8-1

Now enter the IP that you will use for the cluster. This is a virtual IP and will primarily be used for accessing the prism management web interface. Enter the DNS and NTP servers that will be used, then click on “run installation” on the top of the screen.

f-8

It will first configure the IPMI, or in this case with the HX servers, the IMM IPs. After that it will run through the deployment process.

f-9

f-10

After the deployment is complete, you can log into the IP of the cluster through a web interface to manage the environment. The hosts are running with ESXi and can be added to vCenter.

Some things to note on the Mellanox side. To configure the QSFP port to use the hybrid cables, you need to run this command in the CLI of both switches.

Switch-A# interface etherent 1/1 module-type qsfp-split-4 force

Switch-B# interface etherent 1/1 module-type qsfp-split-4 force

Here is the documentation on the Mellanox site for more info.

https://community.mellanox.com/docs/DOC-1450

Here is a getting started guide for the CLI on the switches

https://community.mellanox.com/docs/DOC-2172

Lastly, this is what the finished cabling looked like from the rear. 1G IMM connections on the left. Two 10G SFP+ connections per node in the center and two 1G connections to the legacy iSCSI switch on one node at the bottom.

IMG_20160513_1539038_rewind.png

To migrate existing VMs over to the new environment, the new ESXi 6.x host on the bottom connected to the existing iSCSI target so it could see the old VMFS3 datastores.

A new VCSA6 vCenter was deployed and the new hosts were added to that. Then the VMs were shutdown and unregistered on vCenter one by one on the old hosts. The new host registered the VMs one at a time and then powered them back on. Once they were back up and running, the maintenance window was over. Total downtime was only the power-off /unregister and re-register / power-on period, which was about 3 mins per VM.

Once the VMs were up and running, then the VMs were sVmotioned to the newly created Nutanix containers.

The legacy servers and storage were decommissioned and everyone was happy with their fancy new hyperconverged infrastructure.

So to summarize, the process for deploying the Lenovo HX nodes with Nutanix AOS on ESXi was very similar to a regular Nutanix deployment, but with a few differences:

1> Custom ESXi ISO required

2> Customer foundation package required

3> IPMI is called IMM and the user/pass is different.

4> Host firmware cannot be managed though Nutanix Prism. It has to be done manually through the IMM web interface. This means that nodes need to be shutdown and updated one at a time. Word on the street is that this fall Lenovo will release a free tool to manage server firmware from a central location, but for now this is hush hush. Mums the word.

 

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s