Strange thermal Issue after Update to ESXi 7.0b

Patch build 16324942 for ESXi 7.0 has been released on June 23rd 2020. It will raise ESXi 7.0 GA to ESXi 7.0b. As usual I’m patching my homelab systems ASAP. As all hosts are fully compliant with HCL, I chose a fully automated cluster remediation by vSphere Lifecycle Manager (vLCM).

The specs

ServerSuperMicro SYS-E300-9D-8CN8TP
BIOS1.3
ESXi7.0 GA build 15843807 (before) / 7.0b build 16324942 (after)
HCL compliantyes

During host reboot I realized a temperature warning LED on the chassis. A look into IPMI revealed a critical CPU temperature state. Also the fans responsible for CPU airflow ran at maximum speed.

As you can see, system temperature was moderate and fans usually run at low to medium speed under these conditions. Air intake temperature was 25°C.

My ESXi nodes rebooted with the new build 16324942 and there were no errors in vLCM. But I could hear there’s somethin wrong. A fan running at speed over 8000 RPM will tell you there IS something to look after. Also the boot procedure took much longer than usual.

I quickly shut down the whole cluster in order to avoid a core meltdown.

Continue reading “Strange thermal Issue after Update to ESXi 7.0b”

Veeam Backup v10 on vSAN 7.0

There have been many new releases in the first quarter of 2020. The long anticipated release of Veeam Backup & Recovery version 10, we’ve been waiting for since 2017 and also the latest generation of VMware vSphere. While I had vSAN 7 beta running on my homelab cluster before GA, I’ve worked with Veeam Backup 10 only in customer projects. There’s unfortunately no room for playing with new features unless the customer requests it. One of the new features of Veeam v10 is the ability to use Linux proxies and repositories. With XFS filesystem on the repository you can use the fast clone feature which is similar to ReFS on Windows.

In this tutorial I will show how to:

  • Deploy and size the Veeam server
  • Show base configuration to integrate vCenter
  • Build, configure and deploy a Linux proxy and its integration into backup infrastructure
  • Build, configure and deploy a Linux XFS repository

Using Veeam Backup on a vSAN Cluster has special design requirements. There’s no direct SAN backup on VMware vSAN because there’s neither a SAN, nor a fabric and nor HBAs. There are only two backup methods available: Network Mode (nbd) and Virtual Appliance Mode (hotadd). The latter is recommended for vSAN, but you should deploy one proxy per host to avoid unnecessary traffic on the vSAN interfaces. Hotadd also utilizes Veeam Advanced Data Fetcher (ADF).

Talking about licenses: Having Linux proxies on each host will reduce the cost of Windows licensing. One more reason to play around with this new feature. A Veeam license will be required too, but as a vExpert I can get a NFR (not for resale) license which is valid for one year. Just one of the advantages of being a vExpert. 🙂

Let the games begin. We’ll need a Veeam server that holds the job database and the main application. The proxy and repository role will be kept on individual (Linux) servers.

Continue reading “Veeam Backup v10 on vSAN 7.0”

Upgrade vCenter Server Appliance to vSphere 7

Recently I’ve upgraded my homelab from 6.7U3 to vSphere7. The workflow is straightforward and very easy. The VMware Design team did a very good job with the UI.

First steps

I cannot point that out enough: check the VMware HCL. Just because your system is supported under your current vSphere version, doesn’t mean it’ll be supported under vSphere7 too. On the day I’ve upgraded, vSphere7 was brand new and there were just a few entries in the HCL. But it’s a homelab and if something breaks I don’t care to rebuild it from scratch. Don’t do this in production!

Although my Supermicro E300-9D is not yet certified for version 7.0, it works like a charm. I guess it’s just a matter of time, because the VMware Nano-Edge cluster is based on that hardware.

Before we can start, you need to download the vCenter Server Appliance 7.0 (VCSA) from VMware downloads (Login required). You also need to have new license keys for vCenter, ESXi and vSAN (if yor cluster is hyperconverged).

Continue reading “Upgrade vCenter Server Appliance to vSphere 7”

Replace Intel igbn Driver

There have been issues with VMware network driver igbn which is responsible for Intel 82580, I210, I350, and I354 Gigabit Ethernet Controllers. Under certain conditions this can lead to a PSOD, which makes it a critical issue for all hosts with one of the ethernet controllers mentioned above.

Currently there’s no VMware patch to solve the problem. It is recommended to replace the VMware driver with a newer version (1.4.10) of Intels native driver.

If we start SSH service on the host, we can check the installed igbn version.

esxcfg-module -i igbn 

esxcfg-module module information
input file: /usr/lib/vmware/vmkmod/igbn
License: ThirdParty:Intel Propietary
Version: 0.1.1.0-5vmw.670.3.73.14320388

Updating the driver

First we have to download the driver package from VMware (login required) and extract the archive. It contains a documentation with release notes and update guide, a VMware Installation Bundle (VIB) and an offline bundle (ZIP). While it is possible to install the VIB on a command shell from an ESXi host, it is more convenient to use VMware Update Manager (VUM). The latter is the procedure I will explain here.

Open vSphere-Client and go to Menu > Update Manager. If you’re not running vSphere 6.7 U1 or later, you’ll have to use the infamous Web-Client (Flash-Client). Select Updates and click on “Upload from File”.

Select the extracted ZIP File (Offline Bundle). Just to avoid some confusion: The file you’ve downloaded from VMware is a ZIP-archive. Extract it once. Within that archive there’s another ZIP-archive. Do not extract that one! From the dialogue we select that ‘inner’ ZIP-file for upload to VUM.

Continue reading “Replace Intel igbn Driver”