How to Manage the “Big Data” Revolution

Big data is quickly becoming a problem for enterprising companies as it presents difficulties in how to best analyze, monetize and capitalize large amounts of information within a business and across the globe. There are large amounts of sources that information and data comes from, leading to companies which are looking for partners that can help with their entire big data spectrum in assessing how the company is doing and all of the information surrounding it, as well as preserving that information for later dates and campaigns.  Over the next decade, these needs will become greater for enterprising level organizations and it is important that they explore the options and solutions to the big data revolution in order to keep on top of their game.

 

The Sources of Data

There are a few solutions to managing the big data revolution, but first it is important to understand where all of this information is coming from. Of course, information across the globe comes from many sources and can be seen in several formats, but the most common places data is found are on social media, in phone and web applications, on customer profiles and in documents. Other places of vital information are found in financial transactions as well as emails, videos, and favorite subscribers. These main sources of data are continually being improved upon and gathering more information by the second. This can seem daunting to enterprising businesses but there are a few easy solutions to understanding the massive amounts of data coming from several different sources at all times.

 

The Solutions

There are three solutions that intertwine together to create one large solution to the big data revolution.  The first is to turn you data center into a virtualized one or to find a data center that can host your company virtually. The second solution is to look into storage options that can hold all of the information being gathered and cope with it. Thirdly, detailed and complete infrastructure management (DCIM) is key to have in place. It is the combination of all three of these solutions that an enterprising business can manage the big data revolution.

Author Bio

Chad Calimpong has been recognized locally and nationally for his photography and video documentaries. He enjoys cooking, baking, and has a passion for technology and computers. He currently resides in Austin, Texas with his wife and two cats.

 

[Disclaimer] I’d like to clarify to readers of this blog, that I’m not affiliated with Dell, and have not been sponsored or paid to publish this article. Information and images in the above blog post have been provided to me from Dell, hence it being a guest blog post. I encourage anyone interested in solutions to the “big data” revolution to also explore other hardware vendor solutions and compare the availabile offerings in detail.

ESXi Host Backup & Restore GUI Utility (PowerCLI based) updated to 1.2

A quick post today to just mention that I have updated my ESXi 5.0 / 5.1 Host Backup & Restore GUI utility to version 1.2.

 

There are a couple of improvements in 1.2 based on feedback received in the comments I have received about the utility. The main improvement introduces a function in the script which backs the GUI to check that ESX hosts are valid before attempting to backup or restore these. You can check the utility out over on it’s page here.

 

Updates (29-12-2012) – version 1.2:

  • Added ESX/ESXi host validation into utility – will now test that the host is valid and either connected or in maintenance mode before attempting backup or restore (See the script’s new “Check-VMHost” function for those interested)
  • Minor UI improvements

 

Writing files to an NTFS volume in OSX using VMware Fusion

This is more of a quick tip than the usual full blog posts I do, but a useful one none the less (at least for me). By default when you plug an NTFS (Windows) volume into an Apple Mac, you are able to read the volume, but not write to it. There are ways to forcefully allow NTFS writes, but for those who do not wish to mess with system settings and are using VMware Fusion, this may come in handy.

  • Make sure your Windows VM is powered on and booted, then your plug external USB drive with the NTFS formatted volume into Mac
  • When prompted, choose to connect the drive to your Windows VM
  • Make sure you have a folder from your Mac shared to your Windows VM (e.g. Documents) through Fusion

 

  • Place any files you would like to write to your NTFS volume in your shared folder on your Mac (e.g. Documents folder)
  • Using your Windows VM, open explorer, navigate to the shared folder presented to Windows via Fusion, find the files you want to copy, and then copy/paste them to your NTFS volume that has been passed through and mounted on your Windows VM
  • As Windows is doing the writing it will of course allow writes to the NTFS volume which has been passed through from the Mac

 

Using plink to modify ESXi host configuration files via SSH from a PowerCLI script

I am a big advocate of automation and saving time with a good script. Whenever I can find a task that is fairly lengthy, and is likely to be repeated in future, I always consider scripting it. There are many way to configure an ESXi host when it comes to writing build or automation scripts. In fact, I often feel we are quite spoilt for choice. Here are just some of the tools we have available to use:

  • PowerCLI
  • esxcli
  • vMA
  • vCLI

I was working on a build configuration script the other day using PowerCLI and found the need to edit some configuration files on the hosts. I wanted to edit the configuration file /etc/vmware/config during the execution of a single PowerCLI script without needing to stop the script or have an additional step to do myself. The following is what I came up with to achieve this:

  • Configure host as normal using PowerCLI
  • Use PowerCLI to start SSH service on host
  • execute plink script to connect to host, run command via SSH, then disconnect
  • Use PowerCLI to stop SSH service on host
  • Continue with rest of PowerCLI script

 

Plink is a command line connection tool – essentially a command line version of PuTTy. You can call it from dos prompt and issue it with a single, (or list) of commands to run once connected to a specified host. You can download Plink over here.

 

So without further ado, let’s take a look at the script as I described above.

# At start of our script we ask for the host's IP or name (this could be automated if you like)
$hostIP = Read-Host "Enter ESX host IP/dns name: "
$vmhost = Get-VMHost $hostIP

# Start the SSH service
$sshService = Get-VmHostService -VMHost $vmhost | Where { $_.Key -eq “TSM-SSH”}
Start-VMHostService -HostService $sshService -Confirm:$false

# Use SSH / plink to configure host with some additional script
cmd /c "plink.exe -ssh -pw HOSTROOTPASSWORD -noagent -m commands.txt root@$hostIP"

# Stop SSH service
Stop-VMHostService -HostService $sshService -Confirm:$false

 

As you can see, we start off by asking for the host IP or name, this is the only bit of manual input, but even this could be automated. The script then finds the SSH service on the host, and starts it. After this, the script calls the plink.exe file via cmd /c and connects using the root user@ the host’s IP as we entered at the beginning of the script over SSH. Plink is pointed to a commands.txt file (previously placed in the script execution folder), which contains the actual lines of bash script to be executed on the ESXi host via SSH.

Here is the content of the commands.txt file that I refer plink.exe to use (as an example, this bit of script enables copy/paste operations on all VMs running on this host in the guest OS’ console, as per VMware KB 1026437), but could contain any other commands you wish to execute on the ESXi host over SSH.

echo 'isolation.tools.copy.disable="FALSE"' >> /etc/vmware/config
echo 'isolation.tools.paste.disable="FALSE"' >> /etc/vmware/config

 

* Note two very useful techniques show by Alan in the comments section below, showing how to automatically download plink.exe if it is not available when the script is run, and also how to accept the SSH fingerprint key request by piping Y to plink.exe via the script – check out Alan’s blog post here for more detail.

vExpert spotlight

 

The vExpert title is given out to those who have provided a significant contribution to the VMware community over the previous year. Individuals lucky enough to be awarded this title may receive it for a number of different reasons. For the 2012 title, there were three paths possible to become a vExpert.

  • Evangelist path
  • Customer path
  • VPN (VMware Partner Network) path

Having created and run Shogan.tech over the last four and half years or so, providing content to the community around VMware, automation, and scripting (as well as a couple of small little utilities), I was lucky enough to receive the title on the evangelist path this year.

Gregg Robertson (from thesaffageek.com) has kindly been helping out with organising and publishing the vExpert spotlight entries. After finally managing to find a bit of time, I managed to get my entry together a few weeks ago and submitted it. It has just been posted up on the VMware VMTN blog: http://blogs.vmware.com/vmtn/2012/12/vexpert-spotlight-sean-duffy.html