Cloning and running a duplicate vCenter instance on the same network – process and gotchas

October 19th, 2014 No comments

Recently I needed to clone a vSphere environment (vCenter 5.0.0) for testing purposes. This environment needed to be cloned to have an exact replica of the vCenter server and SQL database server for various tests/upgrades to be performed on it. As for ESXi hosts, a few were being split off the original environment and added to the duplicate vSphere environment.   All the Windows configuration and SQL server configuration needed to be retained, so my high-level plan was as follows:

  • Deploy a new Windows domain (it had to be the same domain functional level and the DC needed to run the same OS as the original)
  • Hot cloned the existing production vCenter and SQL servers
  • Split off the few ESXi physical hosts that were going to be added to the newly cloned environment later on (removed from prod clusters)
  • The new machines needed to run on the same VLAN and IP ranges as the originals too, which made things even more complex, so I made sure to keep the vNICs in the SQL and vCenter cloned VMs disconnected, and disconnected on start up too.
  • Re-IP the cloned VMs after powering them up on one of the split off ESXi hosts (logged in to vSphere client using root credentials) I also re-named their host names and removed from the old domain in, rejoining to the new domain at the same time.
  • Created new service accounts on the newly deployed domain, and reconfigured vCenter services to use these on the cloned machines
  • On the vCenter server there were some changes needed in vCenter config files. This post details most of what needed to be changed:
  • The main change for me though, was I couldn’t see (or didn’t know) how an existing vCenter SQL database would re-act when starting the cloned vCenter on the same VLAN and IP range! There was a strong possibility that this cloned instance could start interfering with the production vCenter and performing operations cross environment. Therefore I decided to create a new SQL vCenter DB. I logged into the cloned vCenter and deleted the old SQL System DSN pointing to the production SQL DB, and created a new SQL database on the cloned SQL box. I then created a new DSN pointing to this, and made sure I searched around for all configuration files on the vCenter server that pointed to the old DSN/SQL server. (I recall there being some references in registry and possibly the vpxd.conf file).
  • Re-creating the right SQL database structure was a bit of a task though. I needed to re-create the structure of the DB without doing an install of course, as I was using the cloned SQL and vCenter machines – with an existing installation on. I followed this KB article, but found a couple of errors/typos in the SQL queries: https://pubs.vmware.com/vsphere-50/index.jsp#com.vmware.vsphere.install.doc_50/GUID-F953497E-2170-4168-806F-6FF0EA6497A7.html by looking at the errors returned in SQL Management studio, you can start to determine where any issues come up and fix the typos. Unfortunately I did not document them myself as I sorted through!
  • Once I was finished with the renaming and IP changes for the cloned machines, I re-connected their vNICs to the relevant networks – happy that they were sufficiently changed!
  • My last issue I came to was that the schema I deployed was one version higher than the vCenter Server build version I was using. I found this out by looking at the vpxd.log file when vCenter failed to start up after deploying the new database schema and content. I fixed this by fudging the version in the database – essentially hard coding the version it wanted into the newly created schema – my thought was that the schema wouldn’t have changed anything old, but rather added new features, so it should be OK. I was right – vCenter started up just fine after this, but I wasn’t happy, so I stopped the services again, deleted the DB, and started again, this time using the right scripts (the scripts referenced in the article link above are located on the vSphere vCenter ISO / media). I had used an ISO with a build increment one higher than the vCenter build I was working with on the cloned VC!) So make sure you use the correct media for your vCenter install here. (I had a vCenter 5.0.0 install, but had deployed schema for 5.0.1!)

 

Hopefully that gives some ideas as to the tasks required when attempting to clone / duplicate an existing vCenter installation, whilst keeping

Fix for VM console error – unable to connect to the mks the operation is not allowed in the current state

September 2nd, 2014 No comments

Bit of a strange one this – I have not dug deeper to find the root cause, but here is a quick fix for anyone with the issue.

mks-console-error-vm

 

I found we could not open VM console sessions in a vCenter 5.5 environment today. Usually one’s first thought is that it is a DNS or port issue when you see the classic MKS console error in a VM, but in this case I knew that DNS and ports were not an issue, as RDPing direct to the vCenter Server itself, logging in with the C# client and opening VM consoles from there were giving the exact same message. This was the case for the web client as well as the C# client.

The issue was either with the host that VMs were running on, or with the VMs themselves – the simple fix:

vMotion the VM to another host. As soon as this was done, I could open the console session. The underlying issue is still out there, but I have not had the time to dig any deeper to find out the root cause. More discussion and info available from this VMware communities thread: https://communities.vmware.com/thread/450294

Force ESXi trial license to expire ahead of time

July 4th, 2014 No comments

I recently caught a question on Twitter from Steve Jin, asking if anyone knew how to force an ESXi host to expire it’s trial license for testing purposes.

This got me thinking a bit, and I initially thought the obvious solution would be to set the host’s system clock forward by 60 days for example. I quickly remembered though, that ESXi hosts always seem to count time toward their trial license time based on the number of hours they are powered up for. If you power your host down for a month, and power it back up again, you’ll still have the same amount of time left over on your trial license.

So the next thing I thought, was if I were building a product and protecting it with licensing, surely I would try to prevent people from tampering with the license files. So if someone were to tamper with a license, I could immediately deactivate it, or expire it. This is where I got the idea that worked for Steve’s use case – finding the license.cfg file, and entering some invalid data.

The exact solution, as Steve found, was to find the etc/vmware/license.cfg file on your ESXi host, and tamper with <epoc> the entry, causing the license to become invalid. At this point, any remaining trial license time is invalidated and your license enters an expired state.

 

lice-eval-expire-ESXi

Change the string highlighted above to some random entry, save the file, then reboot your host. Once rebooted, your trial period will have expired!

This could be really useful in some circumstances. Perhaps there is no clear documentation on how a host running VMs in your environment would react if a trial license expired, or you wanted to know how your 3rd party backup software would react to unlicensed hosts. Being able to easily test an expired license scenario can be really handy!

Solving VMware Fusion 6 and Windows 7 VM performance issues

May 28th, 2014 5 comments

I have been struggling along with various VM performance issues over the last couple of months using VMware Fusion 5.x, as well as the latest 6.0.3. I just didn’t get the time to dedicate to find a fix for the performance degradation I was seeing until just recently.

I have the following specifications on my Macbook Pro Retina which I use for development purposes:

macspec1

I have a Windows 7 Professional VM running in VMware Fusion, with a spec that I had tried all kinds of different configurations on – mainly 2 vCPUs, and 4GB RAM though. This VM is running on the built-in 256GB SSD.

Nothing seemed to fix the performance issues I was seeing, which was that by at least half way though a typical work day of using Visual Studio and a few tabs of Chrome/IE/Firefox, the VM would slow down to an absolute crawl. I knew it was the VM though, as everything in OSX Mavericks, the host OS was perfectly normal. Most of the time just restarting the Windows VM itself would not help though – I would have to reboot the whole macbook.

The other week I decided enough was enough, and spent a bit of time googling and looking around the VMware Communities forums for a fix. Here is the combination of settings that seems to have resolved my issues now.

  • Settled on a VM spec of 3 x vCPUs (helpful for Visual Studio), and 4GB RAM.
  • Disabled app nap for VMware Fusion (Applications -> Right-click, Get Info on VMware Fusion, and tick the box that says “Prevent App Nap”.
  • Added 3 x new entries into my VM’s configuration file (.vmx file). To edit the .vmx file you’ll need to right-click your VM and select “Show Content”. This will allow you to browse the file content of the VM, and you’ll need to locate your VM’s .vmx file. Right-click this file and open it in your text editor of choice. I added the following lines to the bottom of the file:
MemTrimRate = "0"
sched.mem.pshare.enable = "FALSE"
prefvmx.useRecommendedLockedMemSize = "TRUE"

Don’t forget to disable App Nap for Fusion.

prevent-app-nap

Disable app nap for Fusion

Interacting with VMware vCO and the Rest API using PowerShell – getting a list of workflows

April 2nd, 2014 2 comments

Recently I needed to get a list of VMware vCO workflows from a remote server using PowerShell. A colleague of mine pointed me in the right direction by providing me with a URL to access the vCO Rest API on, as well as letting me know what I needed to send in order to authenticate.

To connect and retrieve content back in the PowerShell example below, we’ll need to:

  • Access the Rest API URL for Orchestrator using a web client object
  • Send basic authentication in the header of our request

Notes:

One thing I did notice is that when you use your web browser to test the URL, the result is returned to you as XML, however when I used a web client object in PowerShell, I got a result returned to me in JSON. The PowerShell script below is therefore tailored to interpret the result as JSON. This being so, you’ll need to make sure you are using PowerShell 3.0 or above, as the ConvertFrom-Json cmdlet is only available using PowerShell 3.0 and above.

When sending your authentication details with the web client object request, make sure your username/password combo are used in this format:

Authorization: Basic username:password

This means that your header you add to your web client option, should be added with the string as per the above, but with the username:password part encoded using base64. The script below takes this all into account, and all you need to do is provide your username and password for vCenter Orchestrator to the PowerShell function, it will handle the base64 encoding and passing of the values to the web client itself.

Anyway, enough of that, let us get onto the actual script itself. This is presented as a PowerShell function. Load it into your session (copy-paste) or add it to your PS profile for future use. Apologies for the formatting – Syntax Highlighter really messes with the formatting and nice clean indentation I normally have in my scripts!

Here is a direct download of the PowerShell script if the script paste below doesn’t work for you:

Download381 downloads

 

Function Get-VcoWorkflow() 
{

<#
.SYNOPSIS
Fetches vCO Workflow information and details from a vCenter Orchestrator server

.DESCRIPTION
Fetches vCO Workflow information and details from a vCenter Orchestrator server

.PARAMETER Username
Username for the vCO server

.PARAMETER Password
Password for the vCO server

.PARAMETER Server
The vCO server hostname or IP address

.PARAMETER PortNumber
The port to connect on

.EXAMPLE
PS F:\> Get-VcoWorkflow -Username Sean -Password mypassword -Server 192.168.60.172 -PortNumber 8281

.LINK

http://www.shogan.co.uk

.NOTES
Created by: Sean Duffy
Date: 30/03/2014
#>

[CmdletBinding()]
param(
[Parameter(Position=0,Mandatory=$true,HelpMessage="Specify your vCO username.",
ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
[String]
$Username,

[Parameter(Position=1,Mandatory=$true,HelpMessage="Specify your vCO password.",
ValueFromPipeline=$false,ValueFromPipelineByPropertyName=$true)]
[String]
$Password,

[Parameter(Position=2,Mandatory=$true,HelpMessage="Specify your vCO Server or hostname.",
ValueFromPipeline=$false,ValueFromPipelineByPropertyName=$true)]
[String]
$Server,

[Parameter(Position=2,Mandatory=$true,HelpMessage="Specify your vCO Port.",
ValueFromPipeline=$false,ValueFromPipelineByPropertyName=$true)]
[ValidateRange(0,65535)] 
[Int]
$PortNumber
)

process 
{
$Report = @() | Out-Null

# Craft our URL and encoded details note we escape the colons with a backtick.
$vCoURL = "https`://$Server`:$PortNumber/vco/api/workflows"
$UserPassCombined = "$Username`:$Password"
$EncodedUsernamePassword = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($UserPassCombined))
$Header = "Authorization: Basic $EncodedUsernamePassword"

# Ignore SSL warning
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}

# Create our web client object, and add header for basic authentication with our encoded details
$wc = New-Object System.Net.WebClient;
$wc.Headers.Add($Header)

# Download the JSON response from the Restful API, and convert it to an object from JSON.
$jsonResult = $wc.downloadString($vCoURL)
$jsonObject = $jsonResult | ConvertFrom-Json

# Create a blank Report object array
$Report = @()

# Iterate over the results and transform the key/value pair like formatting to a proper table.
# Note I use a hashtable to populate a new PSObject with properties, using values found in the original JSON result
foreach ($link in $jsonObject.link)
{
	$kvps = $link.attributes
	$HashTable = @{}
	$kvps | foreach { $HashTable[$_.name] = $_.value } # foreach item in the link object, populate the hashtable
	$NewPSObject = New-Object PSObject -Property $HashTable # Create a new PSObject and populate the properties/values using our hashtable

	# Add our populated object to our Report array
	$Report += $NewPSObject
}

return $Report

}
}

I hope that this script comes in handy, and gives you an idea as to how you can retrieve and convert data from vCO and its RESTful API for use in PowerShell :)

Results example after running the Function:
get-vcoworkflow-results

 

vExpert recognition for 2014

April 2nd, 2014 No comments

I am very honored to have received VMware vExpert recognition for 2014 (the third year running now). Keeping this blog updated with new content has proven a difficult task as of late with many more work and personal commitments on the go, in addition to a second blog I started relating to my hobby of game development (and coding).

Hopefully I can maintain momentum here on Shogan.tech and keep posting content related to IT, Automation, cloud, virtualisation and scripting though!

Congratulations to all the vExperts for 2014 (both new and returning) then – it is great to see such an enthusiastic community out there!

Categories: VMware Tags: ,