Archive for the ‘How-tos’ Category

vCenter Server Appliance VM fails to boot with fsck failed message

November 5th, 2015 No comments


I had a storage outage to deal with recently, and after the datastores on this storage were taken down, a vCenter Server Appliance VM on the storage got some corrupted files and would not boot. Upon start up, I was greeted with this message:


The error message reads:

fsck failed.  Please repair manually and reboot.  The root
file system is currently mounted read-only.  To remount it
read-write do

After trying the mount command using the maintenance mode bash shell, I restarted and found that the appliance still did not boot properly. I found a thread on the VMware community forums where someone had the same issue and was able to run e2fsck to fix the disk issues. I tried this and found it fixed a whole heap of disk errors on the /dev/sda3 mount, but on restart I noticed more issues on the /dev/sdb2 mount, so I ran the e2fsck command again for this path, and was able to finally reboot the appliance successfully. The commands I ran to resolve were essentially:

  • mount -n -o remount,rw /
  • e2fsck -y /dev/sda3
  • e2fsck -y /dev/sdb2
  • CTRL-D to reboot after fixing the errors using e2fsck



VMware T10 compliant VAAI integration and HP P2000 G3 FC storage

November 5th, 2015 No comments


I’ve recently been updating firmware on some development and testing storage and found that the HP P2000 storage array firmware update TS251R004 and above enable the HP P2000 G3 FC enable T10 compliance for the hardware.


To quote VMware’s documentation on their VAAI implementation specific to T10 compliance:

The second required component can be referred to as a VAAI plug-in specific to the VAAI filter. It implements vendor-specific VAAI functions such as ATS, XCOPY and WRITE_SAME. There were different implementations of the VAAI block primitives in vSphere 4.1, but all of the primitives in vSphere 5.0 have been ratified by T10, so any array that is T10 compliant should be able to use VAAI.


This means that you no longer need to be running the HP P2000 VAAI plugin software directly on ESXi hosts. In fact, HP recommend you uninstall and remove the plugin software before you upgrade the firmware on these arrays, otherwise you could suffer from performance degradation and possible loss of access to datastores.

My process was to first of all login to all hosts and check for the presence of the VAAI plugin.

  • SSH into host as root, run find / -name hp_vaaip_p2000
  • Ensure that nothing comes up with the find command, if it does (you see something like this output: /usr/lib/vmware/vmkmod/hp_vaaip_p2000), then you should use this HP document to ensure it is removed correctly: – this will involve some setting changes, and removing claim rules as well as removal of the HP P2000 VAAI VIB itself.
  • After verifying nothing came up, check other hosts, and once happy all hosts are clear of the plugin, upgrade the firmware for the P2000 system.
  • Ideally reboot ESXi hosts after the firmware update and ensure access to datastores is still there. Check the hardware acceleration status of datastores – they should show up as “Supported”.

Octopus Deploy Endpoint auto configuration on Azure VM deployment

January 28th, 2015 No comments

I’ve been working on a very cool project that involves the use of Microsoft Azure, TeamCity and Octopus Deploy.

I have created an Azure PowerShell script that deploys VMs into an Azure Subscription (Web machines that run IIS) as a part of a single Azure Cloud Service with load balancing enabled. As such, the endpoint ports that I create for Octopus tentacle communication need to differ for each machine on the public interface.

I wanted to fully automate things from end-to-end, so I wrote a very small console application that uses the Octopus Client library NuGet package in order to be able to communicate with your Octopus Deploy server via the HTTP API.

Octopus Endpoint Configurator on GitHub

The OctopusConfigurator console application should be run in your Azure VM once it is deployed, with 4 x parameters to specify when run.

It will then establish communication with your Octopus Deploy server, and register a new Tentacle endpoint using the details you pass it. The standard port number that gets assigned (10933) will then be replaced if necessary with the correct endpoint port number for that particular VM instance in your cloud service. For example, I usually start the first VM in my cloud service off on 10933, then increment the port number by 1 for every extra VM in the cloud service. As the deployments happen, the console application registers each new machine’s tentacle using the incremented port number back with the Octopus master server.

Once the Azure VM deployment is complete, I tell the VMs in the cloud service to restart with a bit of Azure PowerShell and once this is done, your Octopus environment page should show all newly deployed tentacles as online for your environment. Here is an example of an Invoke-Command scriptblock that I execute remotely on my Azure VMs as soon as they have completed initial deployment. What I do is tell the VM deployment script to wait for Windows boot, so once ready, the WinRM details are fetched for the VM using the Get-AzureWinRMUri cmdlet for Azure, which allows me to use the Invoke-Command to run the below script inside the guest VM.


Invoke-Command -ConnectionUri $connectionString -Credential $creds -ArgumentList $vmname,$externalDNSName,$creds,$InstallTentacleFunction,$OctopusExternalPort,$OctopusEnvironmentName -ScriptBlock {
	$webServerName = $args[0]
    $DNSPassthrough = $args[1]
    $passedCredentials = $args[2]
    $scriptFunction = $args[3]
    $OctoPort = $args[4]
    $OctopusEnvironmentName = $args[5]
	function DownloadFileUrl($url, $destinationPath, $fileNameToSave)
	    $fullPath = "$destinationPath\$fileNameToSave"

	    if (Test-Path -Path $destinationPath)
	        Invoke-WebRequest $url -OutFile $fullPath
	        mkdir $destinationPath
	        Invoke-WebRequest $url -OutFile $fullPath

	    Write-Host "Full path is: $fullPath"
	    return [string]$fullPath
	# Download the Octopus Endpoint Configurator to C:\Temp
	[string]$ConfiguratorPath = DownloadFileUrl "" "C:\Temp" ""
	Write-Host "Unzipping" -ForegroundColor Green
    cd C:\Temp
    $shell_app=new-object -com shell.application
    $filename = ""
    $zip_file = $shell_app.namespace((Get-Location).Path + "\$filename")
    $destination = $shell_app.namespace((Get-Location).Path)
    cd C:\Temp

    if (Test-Path -Path .\OctopusConfigurator.exe)
        & .\OctopusConfigurator.exe http://theoctopusurl.domain API-XXXXXXXXXXXXXXXXXXXXXX $webServerName $OctoPort
        Write-Host "Reconfigured Octopus Machine URI to correct port number" -ForegroundColor Green
        Write-Host "OctopusConfigurator not found!" -ForegroundColor Red

Cloning and running a duplicate vCenter instance on the same network – process and gotchas

October 19th, 2014 2 comments

Recently I needed to clone a vSphere environment (vCenter 5.0.0) for testing purposes. This environment needed to be cloned to have an exact replica of the vCenter server and SQL database server for various tests/upgrades to be performed on it. As for ESXi hosts, a few were being split off the original environment and added to the duplicate vSphere environment.   All the Windows configuration and SQL server configuration needed to be retained, so my high-level plan was as follows:

  • Deploy a new Windows domain (it had to be the same domain functional level and the DC needed to run the same OS as the original)
  • Hot cloned the existing production vCenter and SQL servers
  • Split off the few ESXi physical hosts that were going to be added to the newly cloned environment later on (removed from prod clusters)
  • The new machines needed to run on the same VLAN and IP ranges as the originals too, which made things even more complex, so I made sure to keep the vNICs in the SQL and vCenter cloned VMs disconnected, and disconnected on start up too.
  • Re-IP the cloned VMs after powering them up on one of the split off ESXi hosts (logged in to vSphere client using root credentials) I also re-named their host names and removed from the old domain in, rejoining to the new domain at the same time.
  • Created new service accounts on the newly deployed domain, and reconfigured vCenter services to use these on the cloned machines
  • On the vCenter server there were some changes needed in vCenter config files. This post details most of what needed to be changed:
  • The main change for me though, was I couldn’t see (or didn’t know) how an existing vCenter SQL database would re-act when starting the cloned vCenter on the same VLAN and IP range! There was a strong possibility that this cloned instance could start interfering with the production vCenter and performing operations cross environment. Therefore I decided to create a new SQL vCenter DB. I logged into the cloned vCenter and deleted the old SQL System DSN pointing to the production SQL DB, and created a new SQL database on the cloned SQL box. I then created a new DSN pointing to this, and made sure I searched around for all configuration files on the vCenter server that pointed to the old DSN/SQL server. (I recall there being some references in registry and possibly the vpxd.conf file).
  • Re-creating the right SQL database structure was a bit of a task though. I needed to re-create the structure of the DB without doing an install of course, as I was using the cloned SQL and vCenter machines – with an existing installation on. I followed this KB article, but found a couple of errors/typos in the SQL queries: by looking at the errors returned in SQL Management studio, you can start to determine where any issues come up and fix the typos. Unfortunately I did not document them myself as I sorted through!
  • Once I was finished with the renaming and IP changes for the cloned machines, I re-connected their vNICs to the relevant networks – happy that they were sufficiently changed!
  • My last issue I came to was that the schema I deployed was one version higher than the vCenter Server build version I was using. I found this out by looking at the vpxd.log file when vCenter failed to start up after deploying the new database schema and content. I fixed this by fudging the version in the database – essentially hard coding the version it wanted into the newly created schema – my thought was that the schema wouldn’t have changed anything old, but rather added new features, so it should be OK. I was right – vCenter started up just fine after this, but I wasn’t happy, so I stopped the services again, deleted the DB, and started again, this time using the right scripts (the scripts referenced in the article link above are located on the vSphere vCenter ISO / media). I had used an ISO with a build increment one higher than the vCenter build I was working with on the cloned VC!) So make sure you use the correct media for your vCenter install here. (I had a vCenter 5.0.0 install, but had deployed schema for 5.0.1!)


Hopefully that gives some ideas as to the tasks required when attempting to clone / duplicate an existing vCenter installation, whilst keeping

Fix for VM console error – unable to connect to the mks the operation is not allowed in the current state

September 2nd, 2014 No comments

Bit of a strange one this – I have not dug deeper to find the root cause, but here is a quick fix for anyone with the issue.



I found we could not open VM console sessions in a vCenter 5.5 environment today. Usually one’s first thought is that it is a DNS or port issue when you see the classic MKS console error in a VM, but in this case I knew that DNS and ports were not an issue, as RDPing direct to the vCenter Server itself, logging in with the C# client and opening VM consoles from there were giving the exact same message. This was the case for the web client as well as the C# client.

The issue was either with the host that VMs were running on, or with the VMs themselves – the simple fix:

vMotion the VM to another host. As soon as this was done, I could open the console session. The underlying issue is still out there, but I have not had the time to dig any deeper to find out the root cause. More discussion and info available from this VMware communities thread:

Force ESXi trial license to expire ahead of time

July 4th, 2014 No comments

I recently caught a question on Twitter from Steve Jin, asking if anyone knew how to force an ESXi host to expire it’s trial license for testing purposes.

This got me thinking a bit, and I initially thought the obvious solution would be to set the host’s system clock forward by 60 days for example. I quickly remembered though, that ESXi hosts always seem to count time toward their trial license time based on the number of hours they are powered up for. If you power your host down for a month, and power it back up again, you’ll still have the same amount of time left over on your trial license.

So the next thing I thought, was if I were building a product and protecting it with licensing, surely I would try to prevent people from tampering with the license files. So if someone were to tamper with a license, I could immediately deactivate it, or expire it. This is where I got the idea that worked for Steve’s use case – finding the license.cfg file, and entering some invalid data.

The exact solution, as Steve found, was to find the etc/vmware/license.cfg file on your ESXi host, and tamper with <epoc> the entry, causing the license to become invalid. At this point, any remaining trial license time is invalidated and your license enters an expired state.



Change the string highlighted above to some random entry, save the file, then reboot your host. Once rebooted, your trial period will have expired!

This could be really useful in some circumstances. Perhaps there is no clear documentation on how a host running VMs in your environment would react if a trial license expired, or you wanted to know how your 3rd party backup software would react to unlicensed hosts. Being able to easily test an expired license scenario can be really handy!