First off, I want to say that I’m extremely pleased with the Mellanox switches. They were a real treat to deploy and configure. Simple, with 3 ways to access them out of the box, web interface, ssh and serial console. The SX1012 switches are 1U half width with 12 QFSP ports per switch. Each port is capable of 40Gbps.
The Lenovo HX servers were not what I expected from an HCI solution. Granted, they are high quality 2U rackmount servers, but they are still just that. However, where they lack in space savings, they make up for in connectivity options.
Each server had 2 x intel x520 dual 10G SFP+ adapters, giving them 40Gbps per server for data traffic, 4 x 1G RJ45 connections for legacy connectivity a 1 x IMM connection for out of band management.
The HX nodes were shipped with Nutanix Acropolis Hypervisor pre-installed, so it was very simple to get started by accessing Nutanix Foundation on the CVMs.
[For those that do not know, Nutanix foundation is a web based interface for configuring your out of band management IPs, deploying your hypervisor, deploying the management VMs, creating the clustered storage, assigning IPs and access credentials to the hosts, and configuring all networking. This is done in a 5 minute configuration process, then about 30-45 mins of waiting while everything gets deployed. The CVMs, which are VMs running on the host have an instance of foundation running on them. Another option for deployment is to run foundation on a standalone VM on a laptop.]
The first thing we had to do was to determine our network connectivity. This was being deployed in a brownfield environment, so there was existing servers, switches, etc. We needed to find the best way to connect the new environment to the existing and migrate all the data over.
One problem we had was that there was no 10G SFP+ connection on the legacy HP switches so the QSFP to SFP+ cable would not work in this instance. We had to come up with an alternative method. We were able to use QSFP to SFP adapters which then allowed for the 1G connectivity.
The Mellanox switches are small, but with them you can use what is called a Hybrid cable which is very cool. It takes a single 40G QSFP port and splits it into 4 x 10G ports. So each 12 port half width switch could be a 48 port 10G switch, just by using these cables.
The cabling was minimal and connectivity is easy. We had 4 x HX nodes, so the cable was perfect for us. Each switch had one QSFP to 4x SFP+ hybrid cable, which connected to each server. This allowed us to have PCIe diversity for the connections, switch diversity and serious bandwidth.
We also decided to create a 2 port QSFP LAG between switches, which gave us 80G inter-switch connectivity.
For connectivity into the legacy storage system, we connected two of the 1G ports from one of the nodes into the pre-existing iSCSI switch. Later on, we will connect to the iSCSI target from the ESXi software initiator.
We opted to use the VM version of Nutanix Foundation for reasons I won’t get into in this post, but the embedded would have worked as well. It is important to note that when deploying the HX nodes, you have to use the HX version of Foundation.
This can be obtained from the Nutanix partner portal, under downloads > foundation.
You will also need to download the latest AOS version, which is what runs on the CVMs.
This can be obtained from the Nutanix partner portal, under downloads > AOS.
Lastly, if you are running ESXi, then you will need the Lenovo custom ISO image.
This can be obtained from the my.vmware.com download page. Go to download ESXi, then switch to the “Custom ISOs” tab.
I got the foundation VM from the partner portal as well.
Extract it and use VMware workstation or Fusion to run the VM. Power it on and log in.
Now set the IP address.
Now that we have all the required software bits, put them into the proper foundation folders. The easiest way is to scp the files to the folders on the Foundation VM. For windows use WinSCP, for OSX just use the scp command in the CLI.
The directories are:
For the custom HX foundation package
Then extract the tar.gz file and it will create the foundation folder on the desktop
After the files are in the right directories, then open firefox in the VM and run the setup.
If you have different networks for out of band management and hypervisor management, and have routing setup, then just enter the top part of the global config. If you have multiple networks but have not configured routing yet, then you can multi-home the VM nic and use a flat switch for the hosts. In this case, configure the multi-home section as well.
It is important to note that the IPMI user and pass are different for Lenovo HX nodes.
Use the following:
The o in password is actually the number zero. It is also case sensitive.
Then click next.
If it automatically detects the available nodes, then great. If not, then manually add the blocks by clicking on add blocks on the top right.
Now add the number of blocks and nodes.
Enter in the MAC addresses of the IPMI nics. If you don’t know these, then you can get them in the BIOS, under “network configuration” then look for burned-in MAC address.
In the top of the foundation page, configure the IPMI, Hypervisor and CVM IP ranges. Just enter in the first IP and it will auto-populate the rest sequentially. The same will go the the hostname. Just specify the prefix in “hypervisor hostname”.
Now in the next page, it will show the AOS package and ESXi ISO that you uploaded via scp earlier.
Now enter the IP that you will use for the cluster. This is a virtual IP and will primarily be used for accessing the prism management web interface. Enter the DNS and NTP servers that will be used, then click on “run installation” on the top of the screen.
It will first configure the IPMI, or in this case with the HX servers, the IMM IPs. After that it will run through the deployment process.
After the deployment is complete, you can log into the IP of the cluster through a web interface to manage the environment. The hosts are running with ESXi and can be added to vCenter.
Some things to note on the Mellanox side. To configure the QSFP port to use the hybrid cables, you need to run this command in the CLI of both switches.
Switch-A# interface etherent 1/1 module-type qsfp-split-4 force
Switch-B# interface etherent 1/1 module-type qsfp-split-4 force
Here is the documentation on the Mellanox site for more info.
Here is a getting started guide for the CLI on the switches
Lastly, this is what the finished cabling looked like from the rear. 1G IMM connections on the left. Two 10G SFP+ connections per node in the center and two 1G connections to the legacy iSCSI switch on one node at the bottom.
To migrate existing VMs over to the new environment, the new ESXi 6.x host on the bottom connected to the existing iSCSI target so it could see the old VMFS3 datastores.
A new VCSA6 vCenter was deployed and the new hosts were added to that. Then the VMs were shutdown and unregistered on vCenter one by one on the old hosts. The new host registered the VMs one at a time and then powered them back on. Once they were back up and running, the maintenance window was over. Total downtime was only the power-off /unregister and re-register / power-on period, which was about 3 mins per VM.
Once the VMs were up and running, then the VMs were sVmotioned to the newly created Nutanix containers.
The legacy servers and storage were decommissioned and everyone was happy with their fancy new hyperconverged infrastructure.
So to summarize, the process for deploying the Lenovo HX nodes with Nutanix AOS on ESXi was very similar to a regular Nutanix deployment, but with a few differences:
1> Custom ESXi ISO required
2> Customer foundation package required
3> IPMI is called IMM and the user/pass is different.
4> Host firmware cannot be managed though Nutanix Prism. It has to be done manually through the IMM web interface. This means that nodes need to be shutdown and updated one at a time. Word on the street is that this fall Lenovo will release a free tool to manage server firmware from a central location, but for now this is hush hush. Mums the word.