Today I was creating an upgrade baseline for some old ESX 4.x hosts to be patched up to a newer update level. I ran into an error whilst uploading an ESX ISO with the new update version and subsequently found myself troubleshooting the issue. I thought I would do a quick post on general things to check when troubleshooting VMware Update Manager.
First of all check you are of course uploading the correct file / ISO! Note that ESX upgrade baselines work with ISO files and ESXi deal with .zip files. Ensure you are using the correct file and build of ESX or ESXi depending on which you are planning on using.
Consult the log files! Logs are kept in different locations depending on your OS that VUM is running on.
Windows XP, 2000, and 2003 – C:\Documents and Settings\All Users\Application Data\VMware\VMware Update Manager\Logs
Windows Server 2008 and above – C:\ProgramData\VMware\VMware Update Manager\Logs\
ESX update manager logs are kept in – /var/log/vmware/esxupdate.log
use cat /var/log/vmware/esxupdate.log | more to view the log file in ESX from the shell / PuTTy SSH session.
The log file in Windows should be named something similar to “vmware-vum-server-log4cpp.log”
You should be able to locate an issue that has occurred by noting the time the issue happened in Update Manager. Open up the relevant log file and navigate down to the time it happened in your logs. Hopefully the description / entry will point you in the right direction.
In my case today, I was trying to upload an ESX 4.0 Update 4 ISO (Complete) to create a new Upgrade Baseline for some older ESX 4.0 hosts. I got an error after uploading the ISO using the new baseline upgrade wizard. See below:
Error message after uploading ISO for Host Upgrade Baseline
Although the message above in the GUI is not very descriptive, after looking at the log files on the Update Manager server I found an entry which explained what my problem was:
The problem was that my ISO file was corrupt – the download had seemed to complete just fine when downloading from vmware.com, but there must have been an issue. A quick check of the ISO using my md5sum command line utility in Windows confirmed that the MD5 hash for this ISO file did not match the MD5 hash listed for the ISO on vmware.com (as pointed out in the log files above).
I downloaded a fresh copy of the ISO, checked the MD5 again to ensure it matched this time, and re-uploaded to create a new baseline. Everything worked as expected this time around.
For my 100th post, I thought I would put something totally different up. Introducing PowerCLI bot!
PowerCLI Bot
I recently purchased a Wacom Bamboo tablet and have been doodling around with it – last night I decided to create a little bot for a custom 404 error page I would like to set up for Shogan.tech. I was looking at the PowerCLI logo at the time and thought I quite liked the colour combination. Anyway, things progressed and I thought I would create a little “PowerCLI Bot” as I have recently been thoroughly enjoying working with PowerCLI and PowerShell.
I’ll be changing him a bit for my custom 404 error page so keep an eye open for his distant cousin 🙂
This is a little bit of a fun project I have been working on in bits of spare time I find. It is all written in PowerCLI 5.0 / 5.1 / PowerShell and the GUI was laid out using Primal Forms Community Edition.
Updates (17-02-2013) – version 1.3:
Hosts are retrieved using a new method (for both backup and restore options)
Updates (29-12-2012) – version 1.2:
Added ESX/ESXi host validation into utility – will now test that the host is valid and either connected or in maintenance mode before attempting backup or restore (See the script’s new “Check-VMHost” function for those interested)
Minor UI improvements
Updates (29-10-2012) – version 1.1:
This utility has been tested on ESXi 5.1 Hosts and confirmed to be able to successfully create a backup archive file of these.
Be sure your DNS is working correctly on the system you run the Backup utility on – it relies on DNS to find your ESXi hosts as they are named in the dropdown when you select the host to backup. (See comments for troubleshooting)
Restore to new hardware (force restore) option added
Fixed labels and connection box label description
What it is essentially, is a GUI that allows you to Backup ESXi 5.0 or ESXi 5.1 hosts to a destination of your choice on a local drive. It also allows you to restore ESXi Host configuration bundle backups taken, to other Hosts. I had other plans to integrate cloud storage options in (Amazon S3) as a backup target, but I thought that this doesn’t really offer anything valuable and I would just keep it relatively simple for now – therefore I have disabled this functionality. The PowerCLI script is fairly long – and I know there are plenty improvements I could make to better it and shorten the code, but for now it does the job. I have added various catches for exceptions / errors so you should get visual feedback if you have entered an invalid path, or username for example.
Anyway, hopefully this proves useful to some! As always, take care using the backup / restore functionality of this utility. The restore functionality works by first putting the ESXi host into Maintenance mode (if it isn’t already in this mode) and then applying a backup bundle to the host (Restore). The Host will reboot immediately following this. The Backup and Restore is implemented using Get-VMHostFirmware and Set-VMHostFirmware cmdlets so you can read the help descriptions for an idea on how exactly these work. The Host Backup Bundles are stored in a path of your choice and when restoring from a path the cmdlet works by looking for a bundle filename that matches the name of your ESXi host selected to restore to. Note that the file browser module I have implemented here (used to select paths) is a modification of the work done by Ravikanth Chaganti on his PowerShell Help Browser GUI. I simply adapted his code to list the contents of local drives in a tree view instead of PowerShell Help cmdlets. The only limitation here is that I have not implemented code to browse further than one level into the root of each drive. If you can provide this modification yourself, please feel free to contribute in the comments! I have just not spent the extra time to do this myself yet.
Here are a couple of quick screenshots of the Utility / GUI:
An interesting thread appeared on the VMware Community forums today under the PowerCLI section where someone asked how to find all the svMotions for the past month. LucD answered the post with the solution to get the events listed out for svMotions. I tried this out in my own environment (first of all I datastore migrated (svMotioned a couple of VMs to make sure there would be some events to find). However, I didn’t get any results back. At first I thought this may be because the event names had changed between vSphere 4 and vSphere 5.
After some looking into the last 5 events that had run by using “Get-VIEvent -MaxSamples 5” and looking at the .Info section for the svMotions, I found that their Info.DescriptionId sections were showing as “StorageResourceManager.applyRecommendation” rather than “VirtualMachine.relocate” as LucD’s script had indicated they should be. After digging some more this evening, I realised that maybe the difference was down to the fact that I was svMotioning VMs between Datastore Clusters, rather than between normal Datastores (not part of a DS Cluster). I tried svMotioning a VM to a normal Datastore, and low and behold the event’s Info.DescriptionId now appeared as “VirtualMachine.relocate”!
So the difference is kind of related to the vSphere edition (in the way that you only get SDRS / Datastore Clusters in vSphere 5. More specifically, the task/event difference is seen when using Storage vMotion to move a VM between two Datastores or two Datastore Clusters.
Migrating VMs Between two Datastore Clusters, you see the event “Apply Storage DRS Recommendations“
Migrating VMs Between two normal Datastores, or from a Datastore Cluster to a normal Datastore (or vice-versa) you see the event “Relocate virtual machine“.
Here are some screenshots to illustrate what I mean:
Recent Tasks - differences between the two svMotion scenarios
PowerCLI - listing both svMotion scenario event types
Here is the PowerCLI script used to list both types of events if you are using both (or just one of) the Datastore types – i.e. Datastore Clusters or normal Datastores. It is based off LucD’s script in the first post of this thread on VMware Communities, but just modified to also show the “Between Datastore Cluster” types of svMotions too.
Welcome session attendance - Photo credit - Chris Kranz (@ckranz)
Today’s meetup was the first London VMUG that I have attended. In the past they have unfortunately landed up on days where work commitments took precedence. Running a few minutes late due to a long walk from Bank Underground Station to the venue, I arrived (to my luck) to find that the Introduction had also kicked off a few minutes late, setting most events forward fifteen minutes. I snuck in through a door near the back to listen to the welcome session.
First Sessions of the morning
Attendance was good from what I saw today – all the sessions were quite full and well attended. Symantec did an interesting presentation on ApplicationHA – a talk followed on by a live demo showcasing Application High Availability. The demo entailed bringing down the SQL Server Instance on a VM at first, allowing ApplicationHA to restart the service to sort it out again. This was followed on by another demo – deleting the entire Database and allowing ApplicationHA to pick up the problem and sort out SQL Server by leveraging Backup Exec to restore the Database. Symantec were also kind enough to offer up some NFR licenses for lab/testing use at the end of their presentation. Its a shame I didn’t get a chance to visit their stand during the break, as I was keen on taking a closer look at this in my own home lab environment.
Next up Chris Kranz and Alex Smith did an informative and interesting set of sessions entitled “Would you like fries with your VM?” and “DevOps & Service Management” respectively. They were interesting talks involving some interesting discussion around the traditional “IT Admin” role, compared with the “Virtual Admin” and “Cloud Admin”. Summing up – IT professionals should stay on top of their game and adapt to survive in this ever evolving industry! Alex also shared some interesting experiences and chatted about DevOps and Service Management along with a few other acronyms – determined to drill these into everyone’s head!
During the break I was able to meet up with Gregg Robertson and Jonathan Medd – there was some interesting chat in the short break, after which the next set of sessions began.
Midday Session
This was a set of sessions that conflicted for me – I was really keen on both. I have had a brief look at Auto Deploy before (whilst studying for VCP5), but I also really wanted to see the VMware View session (End User Computing : Today & Tomorrow – Simon Richardson). I ended up attending Alan Renouf and Max Daneri‘s “How to build 1000 hosts in 10 minutes with Auto Deploy” session – there were quite a few slides to go through, but a good overview of the PowerCLI cmdlets used for setting up Image Profiles (working with VIBs), Rule Sets and Auto Deploy in general was given. Max then handled a great demo showcasing Auto Deploy at work.
Post Lunch Sessions
I went to the “Stop the Virtualization Blame Game” session by Xangati (Ben Vaux) next. This was of interest to me, as a couple of weeks ago I deployed the free “one host” Xangati VI monitoring appliance in my lab at home. There were unfortunately a few issues with the projector in our room, but there was still a good talk about how the product works and some interesting questions were answered by the team. Xangati also had a demo set up in the main vendor / lunch area for live demos throughout the day. The product aims to give SysAdmin’s a “single pane of glass” view of the entire VI / VDI environment – where everything can be monitored and looked after. They monitor stats realtime and also offer a handy “record” feature which allows events / issues in environments to be captured, and replayed later on to see what went wrong. Interesting stuff, and I’ll definitely be playing with this product further in my home lab.
The next session I attended was the “Private vCloud Architecture Deep Dive” with Dave Hill and Aidan Dalgleish. This was an interesting and fairly in-depth session discussing the whole VMware ecosystem: vCloud Director 1.5, vShield, Chargeback etc etc. A “reference architecture” was presented on and discussed along with the three network pool methods and their various pros and cons (VLAN-backed, Port group-backed and vCloud Network Isolation Backed (VCNI)). I also wanted to attend Michael Poore’s session on Orchestration, however these two sessions conflicted and I unfortunately had to decide at the last minute as to which one to view!
The final session had me attending the Embotics lab – I had a quick try out of their V-Commander product to see what benefits it offered. I really wanted to see the Cisco UCS presentation so I did unfortunately miss this one. However I will definitely be catching up on this with the slides that will hopefully be made available soon. Gregg Robertson also did his VCP 5 Tips and Tricks presentation, which I hear went down well – I skipped this one as I was lucky enough to fit in an exam and get my VCP 5 done earlier on this month. Whilst on the topic of VCP’s Jonathan Medd surprised everyone as he casually snuck off during lunch to Global Knowledge to write his VCP 5 exam… and passed!
Ending off with vBeers just down the road, I managed to catch up and have some great conversation with a few other guys, including Gregg Robertson, Jonathan Medd, Darren Woollard, Jeremy Bowman, Michael Poore et-al (sorry to those whose names I omitted i.e. fogot!) All in all, a great day was had with some interesting content!
