Manually updating and upgrading the 5.5 vCSA

System: VMware vCSA 5.5

- last updated on -

About: The purpose of this document is to provide a guide on how to update and upgrade the 5.5 vCenter Server Appliance through a hands-on approach.

How-to:

It should be noted that a better way to monitor the update than what is described below might be to connect to the ESXi that's hosting the vCSA, via the VMware fatClient (Windows C# client), and monitor the console of the VM itself.

Preparations

Update process

Revert to snapshot

The below assumes that the update has failed - considering that it is done automatically and that the vCSA should be, more or less, considered a black box, the safest bet is to restore the initial snapshot and get the appliance back to a consistent/working state.

Note: the manual snapshot reverting has not been tested so far, but it should work as intended.

Upgrading a 5.5 vCSA

Note: this section should provide sufficient information for assisting in migrating a 5.5 vCSA to the newer 6.0 variant.

  1. Prepare the source vCSA by taking backups of important configuration details, such as users, licenses, local name resolution, network time synchronization and so on - basically, take a backup of whatever customization was performed on the system. Also, take a memory snapshot of the appliance, set DRS on the cluster(s) to Manual and disable HA.

  2. Download the latest version of the target vCSA in the .iso format to a Windows system that's placed in the same subnet as the source vCSA. Mount the .iso and open vcsa-setup.html from its root directory. Also install the Client Integration Plugin 6.0, available on the .iso at the [driveLetter]:\vcsa\VMware-ClientIntegrationPlugin-6.0.0.exe path, if you use this Windows system to access the webClient and manage vSphere.

  3. Select the Upgrade option and specify the ESXi host, on which the target vCSA will be deployed, via its IP address (known as the target ESXi); provide the root account's password for authenticatio and bypass the ensuing certificate warning, if you don't have a proper PKI set up in the environment. Next provide the name of the new appliance by appending a V6 (make the name distinct from the old one) to the source appliance's name and select the Enable ssh option.

  4. Provide the requested information in the next window, which are the details of the source vCSA and the ESXi hosting it (also know as the source ESXi). NOTE: the wizard needs the FQDN or routable IP address of the old vCSA, because it expects a corresponding DNS PTR record (so the FQDN can be mapped to the IP address and vice versa; this is not possible via the hosts file mechanism, so actual DNS records are needed). Possible problems at this step:

    • if you encounter the Could not find VM with the hostname [vCSA's_FQDN] error message, try using the IP address corresponding to the FQDN instead, or otherwise check the name resolution on the Windows system

    • if, when using the IP address instead of the FQDN, you get the below error message, then consider applying this VMware KB article:

    vCenterServer FQDN [vCSA's_FQDN] does not match DNS servers "hostname#1,hostname#2" and ip addresses "" from VC certificate. Examine the VC certificate and make sure it is valid and point to vCenterServer FQDN.

    • if the wizard returns the error message seen below, check if the hosts file of the system happens to resolve the FQDN (the hosts file has a higher precedence in the name resolution process than the DNS client and it might be used for mapping the private, non-DNS registered, IP addresses the environment's systems; this was the case for the environment which originated the instructions) - if it does, comment out the line in question:

    DNS server maps host name [vCSA's_FQDN] to address(es) ; <<>> DiG 9.9.6-P1 <<>> [vCSA's_FQDN] +short,;; global options: +cmd,;; connection timed out; no servers could be reached which seems to be not local ip address. Please either update appliance /etc/hosts file to properly access DNS servers or update appliance DNS servers with the proper host name mapping. To check if DNS has record about local host name, please execute dig <vCSA's_FQDN> +short locally on the appliance, or nslookup <vCSA's_FQDN> on a windows machine

    • if the wizard complains that it cannot contact the vCenter Server, then the host from which the setup wizard is being run is likely based in a different subnet than the target vCSA, raising concerns with the firewalls/routers that control the network path between the two; if that's the case, consider using a Windows system that's based in the same subnet that the target vCSA is being placed in

  5. Select the appropriate size for the new appliance ('Tiny' should be OK for small environments) and deploy the appliance unto a datastore that has a sufficient amount of free space

  6. Set up a static IP address for the temporary network configuration; this IP address also needs to be registered in DNS, with both an A and a PTR record defined. Notes:

    • the temporary network needs to correspond to the IP address that's defined for the appliance, the gateway and subnet mask information can be taken from the vCSA's VAMI page

    • only one DNS server must be provided, but that particular server has to satisfy the next condition:

    • the DNS server defined MUST be one of those recorded in the vCSA's VAMI page, if any are defined there

  7. Review the settings and Finish the wizard. Note that the upgrade can take some 30-60 minutes to complete.

  8. Once the wizard has completed its work, use the vSphere Client to connect to the ESXi hosting the new vCSA and shut down the appliance's guest OS, then edit the VM's settings and make sure that its vNIC configuration matches that of the old vCSA

  9. Start the vCSA again access its VAMI page (https://[FQDN]:5480) once fully booted up and go to the Networking tab. Edit the Networking Interfaces and provide the vNIC(s) with the IP address(es) of the old vCSA; the network prefix will likely be a 24. Save the changes and exit the vSphere Client session (possible inventory inconsistencies can occur otherwise).

  10. If one is used, connect to the vDPA and disable the backup job that is responsible for the old vCSA appliance, then clone it and assign the new vCSA VM to the cloned job; set the new job's name to reflect the name of the new VM and, once the cloning is done, make sure you enable it. This step is necessary because the vDPA's references to the VMs it protects are GUID based and cannot be modified - once a job is defined, then that job will not accept a different VM-to-GUID pairing than it has recorded initially. So we cannot have the new appliance use the old one's name, but instead use the V6 suffix approach.

Quirks:

The 6.0 vCSA has been found to exhibit a few quirks, which will be listed below:

  1. The new VMware Syslog Service, which the appliance uses in order to export its system logs to a remote syslog collector (via rsyslogd) did a rather silly thing in that its health check was to attempt a telnet connection on the port on which the remote rsyslogd is listening, the problem with which being that this means a connection attempt on TCP/514, while rsyslogd only listens on UDP/514.

  2. The vSphere fatClient experienced timeouts when used to login to the vSphere environment. This behaviour did not occur when using the webClient and the root cause was under investigation with the vendor; I have no idea what the outcome has been.

References:

  1. https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.upgrade.doc/GUID-30485437-B107-42EC-A0A8-A03334CFC825.html
  2. https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.install.doc/GUID-CE128B59-E236-45FF-9976-D134DADC8178.html
  3. http://www.vm-help.com/esx40i/manage_without_VI_client_1.php
  4. http://www.doublecloud.org/2013/11/vmware-esxi-vim-cmd-command-a-quick-tutorial
  5. https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.upgrade.doc/GUID-66836F60-A095-4749-86C9-1DAFB5D21070.html