How to Manage the “Big Data” Revolution

Big data is quickly becoming a problem for enterprising companies as it presents difficulties in how to best analyze, monetize and capitalize large amounts of information within a business and across the globe. There are large amounts of sources that information and data comes from, leading to companies which are looking for partners that can help with their entire big data spectrum in assessing how the company is doing and all of the information surrounding it, as well as preserving that information for later dates and campaigns.  Over the next decade, these needs will become greater for enterprising level organizations and it is important that they explore the options and solutions to the big data revolution in order to keep on top of their game.

 

The Sources of Data

There are a few solutions to managing the big data revolution, but first it is important to understand where all of this information is coming from. Of course, information across the globe comes from many sources and can be seen in several formats, but the most common places data is found are on social media, in phone and web applications, on customer profiles and in documents. Other places of vital information are found in financial transactions as well as emails, videos, and favorite subscribers. These main sources of data are continually being improved upon and gathering more information by the second. This can seem daunting to enterprising businesses but there are a few easy solutions to understanding the massive amounts of data coming from several different sources at all times.

 

The Solutions

There are three solutions that intertwine together to create one large solution to the big data revolution.  The first is to turn you data center into a virtualized one or to find a data center that can host your company virtually. The second solution is to look into storage options that can hold all of the information being gathered and cope with it. Thirdly, detailed and complete infrastructure management (DCIM) is key to have in place. It is the combination of all three of these solutions that an enterprising business can manage the big data revolution.

Author Bio

Chad Calimpong has been recognized locally and nationally for his photography and video documentaries. He enjoys cooking, baking, and has a passion for technology and computers. He currently resides in Austin, Texas with his wife and two cats.

 

[Disclaimer] I’d like to clarify to readers of this blog, that I’m not affiliated with Dell, and have not been sponsored or paid to publish this article. Information and images in the above blog post have been provided to me from Dell, hence it being a guest blog post. I encourage anyone interested in solutions to the “big data” revolution to also explore other hardware vendor solutions and compare the availabile offerings in detail.

My VMware vSphere Home lab configuration

I have always enjoyed running my own home lab for testing and playing around with the latest software and operating systems / hypervisors. Up until recently, it was all hosted on VMware Workstation 8.0 on my home gaming PC, which has an AMD Phenom II x6 (hex core) CPU and 16GB of DDR3 RAM. This has been great, and I still use it, but there are some bits and pieces I still want to be able to play with that are traditionally difficult to do on a single physical machine, such as working with VLANs and taking advantage of hardware feature sets.

 

To that end, I have been slowly building up a physical home lab environment. Here is what I currently have:

Hosts

  • 2 x HP Proliant N40L Microservers (AMD Turion Dual Core processors @ 1.5GHz)
  • 8GB DDR3 1333MHz RAM (2 x 4GB modules)
  • Onboard Gbit NIC
  • PCI-Express 4x HP NC360T Dual Port Gbit NIC as addon card (modifed to low-profile bracket)
  • 250GB local SATA HDD (just used to host the ESXi installations.

Networking

  • As mentioned above, I am using HP NC360T PCI-Express NICs to give me a total of 3 x vmnics per ESXi host.
  • Dell PowerConnect 5324 switch (24 port Gbit managed switch)
  • 1Gbit Powerline Ethernet home plugs to uplink the Dell PowerConnect switch to the home broadband connection. This allows me to keep the lab in a remote location in the house, which keeps the noise away from the living area.

Storage

  • This is a work in progress at the moment, (currently finding the low end 2 x bay home NAS devices are not sufficient for performance, and the more expensive models are too expensive to justify).
  • Repurposed Micro-ATX custom built PC, housed in a Silverstone SG05 micro-ATX chassis running FreeNAS 8.2 (Original build and pics of the chassis here)
  • Intel Core 2 Duo 2.4 GHz processor
  • 4GB DDR2-800 RAM
  • 1 Gbit NIC
  • 1 x 1TB 7200 RPM SATA II drive
  • 1 x 128GB OCZ Vertex 2E SSD (SATA II)
  • As this is temporary, each drive provides 1 x Datastore to the ESXi hosts. I therefore have one large datastore for general VMs, and one fast SSD based datastore for high priority VMs, or VM disks. I am limited by the fact that the Micro-ATX board only has 2 x onboard SATA ports, so I may consider purchasing an addon card to expand these.
  • Storage is presented as NFS. I am currently testing ZFS vs UFS and the use of the SSD drive as a ZFS and zil log / and or cache drive. To make this more reliable, I will need the above mentioned addon card to build redundancy into the system, as I would not like to lose a drive at this time!

Platform / ghetto rack

  • IKEA Lack rack (black) – cheap and expandable : )

 

To do

Currently, one host only has 4GB RAM, I have an 8GB kit waiting to be added to bring both up to 8GB. I also need to add the HP NC360T dual port NIC to this host too as it is a recent addition to the home lab.

On the storage side of things, I just managed to take delivery of 2 x OCZ Vertex 2 128GB SSD drives which I got at bargain prices the other day (£45 each). Once I have expanded SATA connectivity in my Micro-ATX FreeNAS box I will look into adding these drives for some super fast SSD storage expansion.

 

The 2 x 120GB OCZ SSDs to be used for Shared Host Storage
HP NC360T PCI-Express NIC and 8GB RAM kit for the new Microserver

 

Lastly, the Dell PowerConnect 5324 switch I am using still has the original firmware loaded (from 2005). This needs to be updated to the latest version so that I can enable Link Layer Discovery Protocol (LLDP) – which is newly supported with the VMware vSphere 5.0 release on Distributed Virtual Switches. This can help with the configuration and management of network components in an infrastructure, and will mainly serve to allow me to play with this feature in my home lab. I seem to have lost my USB-to-Serial adapter though, so this firmware upgrade will need to wait until I can source a new one off ebay.

 

Installing VMWare ESX using a Dell DRAC card

Here is a how-to on installing VMWare ESX 3.5 using a DRAC (Dell Remote Access Controller) card to access the server. I was installing a new cluster in a Dell M1000e Blade Centre for work the other day and wrote up this process in order for it to be documented for anyone else doing it in the future.

Just for interests sake the basic specs of the system are:

1 x Dell M1000e Blade Centre
3 x Redundant 2000w+ Power supply units
16 x Dell M600 Blades (Each one has 2 x Quad core Xeon CPUs and 32GB RAM).

1. Connect to the M1000e’s chassis DRAC card.
a. Connect to M1000e chassis DRAC card. (https://x.x.x.x) – use the IP for that particular blade centre’s DRAC card. Login with DRAC credentials.
b. Use the agreed DRAC user credentials, or if this is a new setup, the defaults are username: root password: calvin).

login_drac

2. Select boot order for Blade and power it up
a. Choose the Blade server number that you will be working with from the Servers list on the left side.
b. Click on the Setup tab, and choose Virtual CD/DVD as the first boot device then click Apply.
c. Select the Power Management tab and choose Power on, then click Apply.

configure_boot_order_for_blade

3. Go to iDRAC console of the blade server
a. Click on Launch iDRAC GUI to access the iDRAC for the blade you have just powered on.
b. You will need to login again as this is another DRAC we are connecting to (This time the DRAC is for the actual blade server not the chassis).

launch_idrac_gui

4. Configure Mouse
a. Click on the Console tab near the top of the screen and then click the Configuration button near the top.
b. In the mouse mode drop down, select Linux as the mouse type, then click Apply.

configure_mouse

5. Launch Console viewer
a. From the console tab we can now select the Launch Viewer button.
b. An activeX popup might appear – allow it access and the DRAC console should now appear with the server in its boot process.

6. Mount Virtual ISO media for installation disc (ESX 3.5)
a. Click on Media, and then select Virtual Media Wizard.
b. Select ISO image and then browse to the ISO for ESX 3.5 – this could be on your local drive or a network share.
c. Click the connect CD/DVD button to mount the ISO.
d. Your boot order should be configured correctly to boot off this ISO now. (*Optional* You could always press F11 whilst the server is booting to choose the boot device anyway).

attach_virtual_media_iso

7. Reboot the server if the boot from virtual CD/DVD has already passed
a. Go to Keyboard – Macros – Alt-Ctrl-Del to do this.

8. ESX install should now start.
a. Press enter to go into graphical install mode
b. Select Test to test the media (The ISO should generally be fine).
c. Select OK to start the install.
d. Choose United Kingdom Keyboard layout (or whatever Keyboard layout you use).
e. Leave the mouse on generic 3 button USB.
f. Accept the license terms.

esx_install_start

esx1

9. Partitioning
a. For partition options, leave on “Recommended”. It should now show the Dell virtual disk of 69GB (in this case) or the Dell RAID virtual disk / disk configuration.
b. Say “Yes” to removing all existing partitions on the disk. (That is if you don’t mind formatting and completely clearing out any existing data that may be on this disk).
c. Alter partitions to get the following best practice sizes: (See http://vmetc.com/2008/02/12/best-practices-for-esx-host-partitions/)
d. Note: It doesn’t matter if these sizes are 2-3MB out for some. The installer deviates these sizes slightly. The swap partition should have 1600MB minimum though.
e. Next page is Advanced Options – Leave as is (Book from SCSI drive).

esx_partitions_recommended

10. Network Configuration
a. Setup network configuration
b. IP address (x.x.x.x) – whatever IP you are assigning this particular ESX Host.
c. Subnet mask: 255.255.255.0 for example.
d. Gateway:  Your gateway IP address (x.x.x.x)
e. Primary DNS:  (x.x.x.x)
f. Secondary DNS: (x.x.x.x)
g. Hostname: localhost.localdomain for example : ESXhost01.shogan
h. VLAN ID – Leave this blank if you are not using VLANs. If you are, then specify the VLAN here.
i. Create a default network for virtual machines – Unless you have a specific network configuration in mind leave this ticked on.

11. Time zone
a. Set Location to  your location.
b. System clock uses UTC is left as ticked.

12. Root password
a. Set default root password . (This is your admin password)!

13. Finish installation
a. Next page is “About to Install”
b. Check the information is all correct and click Next if all looks fine.

14. Change boot order back and restart the blade server.
a. Via the iDRAC page, change the boot order back to Hard disk for the blade so that it will reboot using the server’s RAID hard disks instead of the ISO.
b. Reboot the host by pressing the Finish button back in the console.
c. Disconnect the Virtual CD from the Media option in the console menu.
d. Watch the console while the server reboots to ensure no errors are reported on startup.

If all went well, you should now have an ESX Host booted to the console. Press Alt-F1 to access the command line (you will need to login as root or any other user you setup).

You can now access your server via the web browser (https://x.x.x.x). From here you can download the Virtual Infrastructure client to manage the ESX Host with.

This host could now be further configured and added to an ESX cluster for example. SANs could be assigned and vMotion setup so that HA (High Availability) and DRS (Distributed Resource Scheduling) can be put to good use!

E8400 Gaming rig build

This is an old post from my other site. I thought as it was IT relevant I would clone the small write up I did across to this blog…

I recently bought myself a new rig, consisting of a Coolermaster CM-690 and the following hardware:

Asus P5Q P45 Pro motherboard
Intel E8400 overclocked to 3.6GHz 24/7
OCZ 2GB ATI Heatspreader RAM DDR800 4-4-4-12
Sapphire ATI HD 4870 512MB GDDR5 Graphics card
OCZ GameXstream 600w Power supply
Western Digital 750GB SATAII Hard drive
Logitech G15 Keyboard (orange backlight model)
Logitech MX518 (5 year old mouse that has travelled the world with me!)

For display I chose a 24″ Dell LCD with a native resolution of 1920×1200 and 6ms response time.

My ultimate goal was to build a faster, cooler and quieter PC than the previous one I had in S.A.

Right, so in my last rig I had the pre-built CM-690 L-shaped window panel. This came with the chassis when I bought it, so I was pretty lazy and didn’t change anything. I also had a Coolermaster Aquagate watercooling unit that fitted in 2 x optical drive bays, which had the pump, radiator and everything incorporated, cooling my E8200 on the old rig. Temperatures were not much better than the Zalman 9700LED that I used to have on it and it was quite messy. I also didn’t enjoy the tiny tubing that this unit used, hence my custom kit choice with 1/2″ diameter tubing for this project. I had never built myself a custom watercooling system, so this will be my first. It will also be the first batch of modding I have done in about 10 years! (The last mod I did was on an AMD K6-2 333MHz in an AT case many, many years ago)! That is barring some odd LED, and minor case mods here and there.

Anyway, here is an image of the final product (Case cut, window installed, hardware assembled and modded to fit the watercooling gear. Cables neatened and basically everything finished, barring the watercooling of the graphics card.

final-1

night-shot

I cut a rough pattern out of the top with my jigsaw, this is where the radiator is to be fit:

case-cut

I cabled-sleeved most of the loose / visible wiring throughout the chassis:

cable-sleeving

Next to be cut was the side panel – Masked off the area to be cut, and used the jigsaw once again:

perspex

This is the box of goodies (watercooling hardware) I ordered from Specialtech:

goodies

The waterblock for cooling the CPU:

cpu-block

Shortly after finishing the water components, and tubing, I started the system up for leak testing…

test-run

A few weeks later the graphics card was ready to be added to the watercooling system. This is a Sapphire ATI HD 4870 512MB (GDDR4) card. I had to remove the stock air cooler, and re-apply some new thermal compound. I used Zalman STG-1 thermal paste for this.

4870-air

Here the card is naked, with the old thermal compound applied to the GPU. The card still needed to be cleaned with some pure alcohol to remove the old thermal paste.

4870-naked

Everything installed, Feser one non-conductive cooling fluid in the loop with the system up and running :

final-2

A small update on this build.

Since the original work was finished, I have now upgraded the RAM. I added another 2GB OCZ RAM to give a total of 4GB. I also pushed my original overclock a bit further, and now run the FSB at 445MHz with a CPU multiplier of 9x giving me a total of 4.0GHz on the E8400. The RAM is running a multiplier of 2x overclocking the four modules to 890MHz each, with timings of 4-4-4-12. My Vcore setting for the processor is on around about 1.375 volts, and my RAM is sitting at 2.2 volts which is what I consider a safe 24/7 setting for RAM modules cooled by passive heatsinks. The FSB is set to 1.16 volts for the increase FSB speed to hold stable. I also flashed the 4870’s bios with a custom image, that sets the card’s default core speed to 795mhz (from a default of 750mhz) and the memory to 1100mhz (from a default of 900mhz). I then use Catalyst Control Centre to up the core speed to a further 830mhz for gaming. The PC now runs at these speeds 24/7 and has no stability issues.