Building a Home VMware Infrastructure Lab

This is something I consider mandatory if you're going to take VMware as seriously as I think everyone should :-)

There's been numerous good sites on this (here: http://www.vmweekly.com/articles/cheap-esx-server-hardware/1/; and here: http://www.techhead.co.uk/building-a-low-cost-cheap-vmware-esx-test-server), but I started differently than most - rather than starting with something that's a pre-packaged, I like to buy the parts and roll up my sleeves.

So, here are the shots of my 2 labs:

First - this is the view of the entry to the server closet, which is literally a closet with a "raised floor" - not for cabling and airflow, but rather so that when the basement inevitably floods, the gear is spared.   It gets hot, so I put in an industrial thermocouple that acts as a cut-off of the dedicated 20A circuit I pulled in - for a shut-off just in case my sophisticated HVAC system fails.

This is the view of my "production servers" - an Intel Q6600 based cluster (8GB RAM each) and a AMD x2 3800+ based cluster (4GB each) - cookbook instructions below.   I use the two bigger Dell PowerConnect 5324s for iSCSI and LAN/NAS traffic - the little Netgear switch is for VMotion.   I could have used VLANs, but I like the physical topology, and had the switches kicking around (or just say "screw IP best practices", but what's the fun in that).   The little host in beside the Intel cluster is my VC host.   You can save some bucks and run VC as a VM of course (handy in some DR use cases!).   I'm glad that VMware made that a supported use case around the VC 2.0.1 timeframe (can't remember the exact release)

This is the view of my "DR cluster" - an Intel E4300 based cluster (8GB RAM each).   These are cheap as dirt, but kinda sucks that they don't have VT.   I also just use a cross-cabled Cat5e cable for Vmotion.   I use this for when I'm just playing around with Site Recovery Manager (how do I do that without arrays?   READ ON!!!).    The poster has one of my favorite quotes:

"Whatever can be done, will be done.   If not by incumbents, it will be done by emerging players.   If not in a regulated industry, it will be done in a new industry born without regulation.   Tehcnological change and it's effects are inevitable" - Andy Grove

Slightly less inspirational (but only slightly) is the rolled up construction plans for my elaborate MAME-based arcade that I'll get started on any day now.....

This is a quick shot of my sophisticated HVAC system.   This rig generates a LOT of heat, particularly when I spin up the arrays.   I bought a cheap 300CFM bathroom fan, and then tore apart all the drywall and vent it out.   the intake is baffled, so when I close the door, the whole thing is pretty quiet.

If this is making you eager to do the same thing - it's gotten ridiculously cheap, and ridiculously easy.   I'm going outline how to build two ESX servers - one where you're looking for the CHEAPEST thing you can build, and one which is a little more pricey, but a great "bang/buck" ESX lab.

ESX 3.5 makes doing this a LOT cheaper - now that SATA drives/controllers are supported, Nvidia NICs are supported in ESX 3.5 bits, but there is a trick - technically the Nforce professional chipsets are the ones supported, but they use the same MAC as the cheapo consumer stuff - you just need to make sure that the motherboard doesn't use a Realtek NIC (or buy one of the Intel NICs)

General things to make sure you do

• Get a CPU that supports 64-bit guests - this is generally an Intel CPU that starts with the letter "Q" not the letter "E" (or just check the specs and look for VT support). Any Athlon 64 or opteron works.    Can you go cheap - yeah, but that often costs you 64-bit guest support on Intel.   If you're going cheap, I personally go AMD.
• Get a motherboard that supports a minimum of 4 GB of RAM - 8GB is nice (all ESX servers are generally constrained by RAM)
• Get a decent (but still super-cheap) GigE switch - something that supports VLANs so you can create configs that work with less physical NICs - it's crazy, but you can get an 8-port switch with full support for 802.1q, p, and everything else you could possibly need.
• Make sure you have a motherboard that has onboard VGA - you don't need a good graphics card, but you need something for initial config.

UPDATE (Jan 5th, 2009) - one of my colleagues sent me a new "record cheap" dual core 8GB config, and I've done a post on that HERE - you might want to start with that, as technology moves pretty fast - heck, some of the older stuff below you can't even BUY anymore :-)

AMD ESX configuration (as cheap as it gets, but you have everything you need) = $337
This config leverages the fact that ESX 3.5 supports Nvidia NICs - and there will only be one NIC for VMotion, network, and IP storage. Name of the game = how cheap can you go

• Athlon x2 4000 retail - dual core (comes with the heatsink/fan) = $55 (http://www.newegg.com/Product/Product.aspx?Item=N82E16819103774)
• Generic ATX Motherboard - based on the 430, 6100 or 6150 chipsets - just MAKE SURE it has the Nvidia NIC, not a Realtek NIC = $54 (http://www.newegg.com/Product/Product.aspx?Item=N82E16813157108) NOTE - THIS ONE HAS A REALTEK NIC, so you need to by an additional intel NIC.   I use an old ASUS A8N-VM CSM socket 939 motherboard, and it has a Nvidia MAC and works great - but you need to find an older Athlon that fits that socket 939 form factor.....
• cheap as dirt HDD = $49 for a 160GB drive (http://www.newegg.com/Product/Product.aspx?Item=N82E16822136075)
• cheap as dirt ATX case/PS = $23 (http://www.newegg.com/Product/Product.aspx?Item=N82E16811164073)
• big 2GB DDR2 memory sticks (you can start with 2, and add another 2 later) = $72(2 x $36) http://www.newegg.com/Product/Product.aspx?Item=N82E16820141300
• cheap as dirt DVD/CD (to install the ISO) = $29 (http://www.newegg.com/Product/Product.aspx?Item=N82E16827106228)
• If you're not sure what MAC the NIC uses on the motherboard, or just want to be safe - add 1 Intel GbE NICs (these are a trick - you need specific ones for the Intel e1000 driver that comes with ESX 3.5 to work - hard to find, and DON'T buy the server MT versions - find the cheapo desktop GT PCI or PT PCIe versions - hundreds cheaper and work fine) = $42 http://www.allstarshop.com/shop/product.asp?pid=16016&ad=pwatch

Intel ESX configuration (a super cheap quad core, 8GB, lotsa GbE powerhouse) = $695
This config leverages the fact there are ridiculously cheap multi-core CPUs and RAM these days. the NICs on Intel motherboards are usually based on older Intel or Realtek chipsets, (no driver support in VMware) - so you need to find some fancier (but still cheap) NICs. Name of the game here = how cheap can you build a powerhouse that you can run 10 VMs at once without breaking a sweat?

• Lots of CPU Intel Q6600 retail - quad core (comes with the heatsink/fan) = $270 (http://www.newegg.com/Product/Product.aspx?Item=N82E16819115017)
• Intel G33/P35 based motherboard (you want ram slots and PCIe slots) = $54 (http://www.newegg.com/Product/Product.aspx?Item=N82E16813121099)
  • Note - some people have struggled with the SATA controller on this board, others have had it work - one that has been proven to work consistently is the EVGA Nvidia 780i (http://www.newegg.com/Product/Product.aspx?Item=N82E16813188024&Tpk=EVGA%2b780i)
• cheap as dirt HDD = $49 for a 160GB drive (http://www.newegg.com/Product/Product.aspx?Item=N82E16822136075)
• cheap as dirt ATX case/PS = $23 (http://www.newegg.com/Product/Product.aspx?Item=N82E16811164073)
• Lots of RAM - big 2GB DDR2 memory sticks = $144(4 x $36) http://www.newegg.com/Product/Product.aspx?Item=N82E16820141300
• cheap as dirt DVD/CD (to install the ISO) = $29 (http://www.newegg.com/Product/Product.aspx?Item=N82E16827106228)
• 3 Intel GbE NICs (these are a trick - you need specific ones for the Intel e1000 driver that comes with ESX 3.5 to work - hard to find, and DON'T buy the server MT versions - find the cheapo desktop GT PCI or PT PCIe versions - hundreds cheaper and work fine) = $126 http://www.allstarshop.com/shop/product.asp?pid=16016&ad=pwatch

OK - what now?

• You will need to buy two of whatever model you get - for VMotion, VM HA, DRS, Storage VMotion, etc... (so AMD total cost = $674, Intel cost = $1390)
• You will need ESX Server and Virtual Center - within EMC, we have a VMware/EMC ELA (remember - VMware operates independent of EMC as the parent!).  You can, of course, download the software from http://www.vmware.com/ - they have 60 day eval timeouts.
• What the heck to use for shared storage?   Well, I have a CX300i, an EqualLogic PS100E, a Storevault S500, and have an Openfiler box - but I'm a freak, and have a very supportive wife.   There is another simple option (that I actually use more than any of the others): use a Virtual Storage Appliance - these turn the DAS storage in the ESX server into iSCSI LUNs or NAS.   EMC offers a free, unlimited, no time-out Virtual Celerra, which just runs on ESX, and is otherwise a fully functional Celerra - anyone who wants one, head over to this post...

Comments

[Chad Sakac]

@Mike - the ILO/RAC is non-existent. I use a KVM and the 'ol manual method.

That's the biggest downer of the whiteboxes to date - but I've got to say, it's pretty easy, with ESXi of course you configure that mgmt interface, then do everything via the vSphere Client.

[Anon]

"if you're going to take VMware as seriously as I think everyone should"

Then maybe you should take your cabling seriously; those wires look like Lionel Richie with a bad hair day.

[Chad Sakac]

@Anon - LOL - aint that the truth! if you look at the post on the work lab, you can see it has very nice cable routing.

At home - it's bad (and has gotten a LOT worse since I took this shot - have added 4 more hosts, and another pair of GbE switches!

The only thing that you got wrong is tha Lionel Richie has NEVER HAD a bair hair day!

:-)

[Brian Lee]

Hi Chad,

I plan to buy the following network storage from iomega to be used as the shared storage.
Do you think that it will work with ESX 3.5?

iomega 33872 1TB Network Storage
Standards: IEEE 802.3, IEEE 802.3u, IEEE 802.3ab
Port RJ-45: 1 x 10/100/1000M
Supported Protocols: Microsoft (CIFS/SMB) Linux/UNIX (NFS) Apple (AFP) and Apple Bonjour support Internet (HTTP 1.1) FTP
Management: Management via user friendly web interface
RAID: RAID 0, 1 and JBOD

Thanks,

[Michael Chapa]

Hello Chad,

I am trying to build a home lab using ESX4i but all of the links to newegg.com hardware are out of date. Does anyone have a list of intel and AMD HW configs with MB and CPU (quad core) that work well with ESX4i for a reasonable price? I plan on getting an Intel Pro 1000 NIC to add to the system.

Also when is the iomega ix2 going to support iscsi ??

Kind Regards,

Michael

[Josh-coen]

Hi Chad,

Does the Q6600 support FT? Have looked (not very well) via google, but haven't been able to find anything on it. Thanks, and great post.

-Josh

[Chad Sakac]

@Josh - thanks for the comment! The Q6600 worked with FT just fine...

[velmueugan]

Hi,

I purchased new computer.the configuration is i3 530 2.9Ghz processor,DH55TC Motherboard,2GB DDR3 1333Mhz RAM,500GB Seagate Sata Harddisk,Samsun Sata DVD drive.i am trying to install VMware ESX 3.5.But it’s showing error “Unable to find a supported device to write the VMware ESX Server 3i 3.5.0 image to and No driver found. Select Driver/Use a Driver Disk/Back”.How to install VMWare ESX 3.5.also i am install ESX 4.0.it’s showing error”No compatible network adapter found.please consult the products” and “Network-adapters returned critical failure” in vmkernal mode (ALT+F12).

I enabled VT technology in bios,

http://www.intel.com/products/desktop/motherboards/DH55TC/DH55TC-overview.htm

Please help me.Thanks in advance,Velmurugan

[Dedicated Server]

Wow, you don't mess around when it comes to this. I've been toying with the idea of doing something similar, but this looks very time consuming to set up. Any tips for implementing something like this on a much smaller scale?

[seahawksgo]

I had been installed ESX 3.5 and openfiler 2.2 on HP ML110 servers without any problem since nearly 3 years ago without any problem. Now this system has problem on hardware then I intend to upgrade to MSI motherboard and i7 core. First it failed cos NICs and Raid on motherboard did not supported by vmware ESX 3.5. Now I intend to buy intel pro 1000 PT NICs and Smart Array e200 128mb to work with ESX driver with the mother board/i7 system. Servers are ready but NIC and Raid controller parts have not coming yet for testing. Does this sound good to work? Any idea?

velmueugan, you have the same problem with my new system. You should try as my new system. I will let you know soon.

[seahawksgo]

Hello Chad,

My small business system has been using ESX 3.5 and Openfiler 2.2 since 2008 still now. Now system slow down that I intend to upgrade hardware as MSI VT supported motherboard, intel i7 core, 12G ram but ESX 3.5 did not recognize NIC and motherboard RAID driver.

I did some research and bought intel pro PT 1000 NICs, either HP smart array e200 PCI express 128Mb (supports raid 5). Does it sound good to work with ESX 3.5? Pls advise. Thanks

[seahawksgo]

Hi Chad,

I received NIC and array controller e200 already. Here are the result:
A. esx3.5
NIC intel pro 1000 pt are recognized by ESX3.5
HP E200 smart array controller are not recognized by ESX >> any comment to help me solve this problem?

Openfiler 2.2 or 2.3 did not recognized E200 too.

B. ESXi 4.1
Both NIC intel pro 1000pt and E200 work well.

The problem is I have license for ESX 3.5, VCenter for esx 3.5 only. CDo you have any idea?

[seahawksgo]

The ESX 3.5 is working fine now with e200 controller after updating the update 3 of ESX 3.5 drivers

[Chan]

Hi Chad,

Just want to update. I tested successful ESX3.5 and openfiler 2.3 on customized 4 servers: MSI motherboard, Intel i7 core, 12Gram, e200 IO controller (vmware esx update 3 is a must) to run RAID 5 Max 3Gbyte, intel pro/1000 PT nics. Both run very fast which VM machines (2003, 2008, redhat, xp)response very fast too.

I will try to build clustering on those system this weekend.

Chan

[Regina Ali]

Chad - have you updated this for vsphere - can you please share

[Cesar Caminero]

The links to download the new Workstation and ESX versions of the Besser UBER : Celerra VSA UBER v2 are here:

http://nickapedia.com/2010/05/19/besser-uber-celerra-vsa-uber-v2/

Enjoy!

[Cesar Caminero]

Actually, the newest version is here:

http://nickapedia.com/2010/09/30/smoothed-edges-celerra-uber-vsa-update-v3-1/

[Robert]

Hey,
Thanks! Great post you have written on "Building a Home VMware Infrastructure Lab". Really I can say that your post is very informative, I'll come across your blog again when you will update it with new.
Thanks,
Robert
http://www.o-n.com.com/

[dhenver]

I wish i have this in my house