Host Upgrade fails with “Cannot execute upgrade script on host”

I recently had the pleasue to time-warp a dinosaur upgrade an old ESXi 6.0 host to ESXi 6.7. Right after I triggered remediation with a current ESXi 6.7 iso image, I got an error message:

Cannot execute upgrade script on host

That message isn’t really specific. If you google it you’ll probably find a dozen possible reasons tor the failure. That can be:

None of the issues above did fit my observed problem. A good startpoint should be a look into vua.log on the affected host.

less /var/log/vua.log

Unfortunately that didn’t help either. So we had (again) a closer look at the VMware upgrade path matrix. A direct host upgrade from ESXi 6.0 to ESXi 6.7U3 is supported but while we re-checked the matrix our attention was drawn to a little footnote.

KB 76555 says there’s an issue with expired VIB certificates on hosts below a specific build numer.

  • ESXi 6.0 GA before build 9239799
  • ESXi 6.5 GA before build 8294253

In fact our ESXi host 6.0 had a build level of 7967664 (U3e) which is in the critical range. So we had to install some patches up to July 2018 (ESXi600-201807001). After that the upgrade to ESXI 6.7U3 went flawlessly.

What went wrong?

Of course we did check the matrix during the planning phase in early March 2020. That’s a standard operating procedure. Unfortunately something has changed in the meantime (the footnote was added). KB 76555 was updated in May 2020 and the issue affects upgrades to versions of ESXi 6.7 beyond April 28th 2020.

Take home message: Check your design and matrices again right before the projects starts.

Replace Intel igbn Driver

There have been issues with VMware network driver igbn which is responsible for Intel 82580, I210, I350, and I354 Gigabit Ethernet Controllers. Under certain conditions this can lead to a PSOD, which makes it a critical issue for all hosts with one of the ethernet controllers mentioned above.

Currently there’s no VMware patch to solve the problem. It is recommended to replace the VMware driver with a newer version (1.4.10) of Intels native driver.

If we start SSH service on the host, we can check the installed igbn version.

esxcfg-module -i igbn 

esxcfg-module module information
input file: /usr/lib/vmware/vmkmod/igbn
License: ThirdParty:Intel Propietary
Version: 0.1.1.0-5vmw.670.3.73.14320388

Updating the driver

First we have to download the driver package from VMware (login required) and extract the archive. It contains a documentation with release notes and update guide, a VMware Installation Bundle (VIB) and an offline bundle (ZIP). While it is possible to install the VIB on a command shell from an ESXi host, it is more convenient to use VMware Update Manager (VUM). The latter is the procedure I will explain here.

Open vSphere-Client and go to Menu > Update Manager. If you’re not running vSphere 6.7 U1 or later, you’ll have to use the infamous Web-Client (Flash-Client). Select Updates and click on “Upload from File”.

Select the extracted ZIP File (Offline Bundle). Just to avoid some confusion: The file you’ve downloaded from VMware is a ZIP-archive. Extract it once. Within that archive there’s another ZIP-archive. Do not extract that one! From the dialogue we select that ‘inner’ ZIP-file for upload to VUM.

Continue reading “Replace Intel igbn Driver”

Runecast 3.0 requires elevated privileges for HCL checks

A couple of days ago Runecast Analyzer has been upgraded to version 3.0.0. With that upgrade a very important beta-feature became GA: HW Compatibility and Upgrade Simulator.

I used to run the Runecast service account with readonly privileges. It has been sufficient up to version 2.7.x. Even the hardware compatibility check (beta) did work with readonly privileges. After upgrading my appliance to version 3.0.0 (GA), I found a notification. Missing privileges..

Once you open host details and click on I/O devices tab, there’s further information.

Continue reading “Runecast 3.0 requires elevated privileges for HCL checks”