Updated "Homebrew" ESX Hardware List

One of the most popular 2008 articles I've posted was this one on how to build a home-brew ESX server.   Within EMC it's a game now for our nerd elite to come up with creative ways to do it cheaper, faster, smaller, and cooler :-)

Keep reading for the new (at least for now!) record holder, from Adam Jones a Sr. TC, VCP (and more acronyms behind his sig than I've ever seen :-) in Kansas City with a dual core, x86-64 (ergo future ready), 8GB machine....

Do you have your own home-brew story?   I'd love to hear it!

Ok, so... the name of the game is cheap hardware that works with VI3.x and also the current builds of future VMware products.   Bypass the most sensitive part (SATA controller support) by using the ESXi installable and dd to a bootable USB stick (though I can confirm my own success with Intel P35 chipset (ICH9) SATA controllers and VI3.x - will report back on the G/P33 and G/P35 chipset's SATA controllers and future ESX builds in a couple days).  

UPDATE (Jan 5th, 2009) - yup, both the P33 (ICH9) and P35 chipset (ICH9) SATA controllers work out of the box with those future builds - odd, since the P33 doesn't with VI3.x - must use a different PCI ID....

This is a great place if you're looking to build your own home-brew and aren't using the shopping list links below and want to check on other parts and people's experiences.

The one thing I WOULD stay away from (if you are planning on moving beyond ESX 3.5 eventually) are the desktop intel cards - man, they've given me a lot of grief recently..   the Intel server cards (i.e. instead of the GT or PT, use the MT) have worked consistently.   They are all good with ESX/ESXi 3.x, but for now at least, I've only had success with the MT with future builds - though that says nothing of future functionality or timing.

Heck, if you really want to have it easy, get something on the HCL :-)

Use either internal SATA drives in the ESX hosts, or low-cost external storage options and then leverage VSAs (Virtual Storage Appliances - like the Celerra VSA here) which can provide iSCSI and NFS shared storage to create clusters, and also to emulate the high-end array functions (snapshots, replication, SRM).   You CAN run VC as a VM on these avoiding the need for a 3rd physical box.   Do this by install VIC somewhere, point it at the first ESXi box, then create a VM, and install VC there (though I still prefer VC as a physical or a VM on another cluster, personally).  

Just REMEMBER (this has given me grief lately - another post on that coming up), the ESXi build needs to have the scratch space (user world) specified (advanced settings on the host in the screenshot below) and enabled for VM HA to work (clear error message on this if you don't do it right), AND this needs to be a separate datastore location for each host in the cluster (no clear error message on this - it just doesn't enable on the subsequent hosts booting - the bottom checkmark is always not checked).    VMware - a nit - there is a typo in the dialog box "user wold" (this is update 3)

This isn't hard to resolve (just create a bunch of iSCSI VMFS datastores or different NFS exports), but the fact that they MUST be different is not obvious...

Without further ado, Adam's h/w list:

ESX 3i Server with 8 GB RAM and 2 GB IP NICs - $326.93 Before RAM Rebate

• Case - Count: 1 - IN WIN IW-BT611T.300SL Silver Best Performed 0.8mm SECC Japanese Steel metal MicroATX Desktop Computer Case 300W Power Supply - Retail - http://www.newegg.com/Product/Product.aspx?Item=N82E16811108060 - $60.99
• Motherboard - Count: 1 - ASUS M2A-VM AM2 AMD 690G Micro ATX AMD Motherboard (note that this doesn't have ESX/ESXi SATA support, so you will need the ESXi Installable USB technique, and external storage) - Retail - http://www.newegg.com/Product/Product.aspx?Item=N82E16813131172 - $64.99
• Processor - Count: 1 - AMD Athlon 64 X2 5000+ Brisbane 2.6GHz Socket AM2 65W Dual-Core Processor Model ADO5000DOBOX - Retail - http://www.newegg.com/Product/Product.aspx?Item=N82E16819103211 - $56.99
• RAM - Count: 2 - OCZ Gold 4GB (2 x 2GB) 240-Pin DDR2 SDRAM DDR2 800 (PC2 6400) Dual Channel Kit Desktop Memory Model OCZ2G8004GK - Retail - http://www.newegg.com/Product/Product.aspx?Item=N82E16820227199 - $83.98 for 8 GB Before Rebate
• NIC - Count: 1or 2 - Intel PWLA8391GTL 10/ 100/ 1000Mbps PCI PRO/1000 GT Low Profile Desktop Adapter - OEM - http://www.newegg.com/Product/Product.aspx?Item=N82E16833106122 - $29.99 Each
• Optional for GB IP Network

Switch - Count: 1 - NETGEAR GS605 10/100/1000Mbps Desktop Switch - Retail - http://www.newegg.com/Product/Product.aspx?Item=N82E16833122140 - $38.99 Before Rebate

Storage - Adam used the Intel Entry Storage System SS4200-E - $353.55 (before you add your own SATA drives - up to 4 drives)

UPDATE (Feb 2, 2009) - The SS4200E is a fine choice, and so is the ix2.   If you can wait a little bit... I would hold on.   iSCSI work is in the lifeline codebase, but still pushing to make it available in all the platforms - which have varying CPUs, and varying iSCSI performance.   for now, consider the shipping SS4200E and ix2 as NAS-only until I give a firm update.   There are other reasons also to wait for about 1 month....

• http://www.amazon.com/Intel-SS4200E-Storage-Retrospect-Software/dp/B00113U42O/ref=pd_bbs_sr_2?ie=UTF8&s=electronics&qid=1231119129&sr=8-2
• This is one of the storage platforms (in this case Intel OEM) that runs EMC Lifeline (our SMB stack for NAS and increasingly iSCSI).    The iomega StorCenter is using this same codebase in the 2 drive iomega StorCenter ix2 (the ix2 is actually on the VMware HCL here).
• WARNING the Intel unit isn't on the HCL, and whereas I know the iomega HCL roadmap (There are more of those SMB "VMware Ready" EMC iomega storage devices coming in larger drive configs... ), I don't for the Intel HW platform - so don't assume it will be on the HCL.  remember - this is a home-brew!  
• http://www.smallnetbuilder.com/content/view/30482/75/ - Review
• http://www.intel.com/products/server/storage-systems/ss4200-e/ss4200-e-overview.htm - Intel Product Main
• Now, the Intel unit has more CPU and spindle horsepower than the really low-end iomega devices (Adam measured 90MBps write and 50MBps read performance) and is a good buy at that price (could you go even cheaper, yes with things like openfiler), IMHO.

Have fun!!!!

Comments

[vinf.net]

Hey Chad,

my colleague has a good set of posts on doing this on an ML115/ML110 which are cheap enough to be white-boxes (in the UK anyway)!

http://www.techhead.co.uk/building-a-low-cost-cheap-vmware-esx-test-server

I've built mine up on some cheap HP D530 desktops from eBay (write-up below) but will be moving to ML115's to get some x64 goodness in due course.

http://vinf.net/2008/01/14/vmware-esx-v35-on-cheap-pc-hardware/
http://vinf.net/2008/02/10/performance-update-on-cheap-esx-pc/

[Duncan]

That's weird, the "scratchconfig" behaviour must have changed. Cause I'm pretty sure that I've used just 1 LUN for this purpose during a ESXi implementation a while ago. I think it was 3.5 U1. Not 100% sure though.

[Chad Sakac]

Trust me on this one Duncan - I tried this several different ways to validate the behavior... It was making me tear my hair out...

Now - any ideas on that VUM problem I asked about? :-)

[Shawn Eveleigh]

Chad,
I'm pretty sure I remember you mentioning the EMC/Iomega product, Storcenter IX2 as an iSCSI target that can be used with ESX.
Just picked one up and it doesn't have iSCSI support - CIFS and NFS but no iSCSI. I have also seen press releases talking about iSCSI for this box.
What's up with that?
Do you have access to some pre-release code at EMC and can hook a brother up? :)

[Chad Sakac]

Shawn - thanks for asking... Answer - the current 1.x software rev doesn't have the iSCSI support, but that is coming to the general public VERY, VERY soon (so soon, it's right around the corner).

Since you are a brother at Dell - I dropped you a pre-release in your inbox. Have fun!

What do you think about the box and GUI?

[Paul Evans]

Hi Chad,

Can you point me to EMC storage offerings that support dedup and can be used as shared storage by development teams as well as release engineers/ system administrators deploying ESX? I guess this would be mid to low market for you folks, but I would like to provide this clarity to readers of my blog at

http://sharevm.wordpress.com/2009/01/07/who-is-feeling-the-pain-for-shrinking-vms/

Thanks
Paul

[Chet Bliss]

Chad,

I really enjoy your blog. Is there any word on ISCSI support on the IX2 yet. I am dying to get my hands on the update for my test lab.

Thanks
Chet

[Alex K.]

Was wondering the samething about ISCSI support. IX2 looks like a very nice NAS for home network.

[Andi]

As the iomega IX4-100 is out with ESX certification but still no iscsi

Are there chances that iscsi is ever released for the IX2 or IX4?

[Collin MacMillan]

RE: scratch

1. Create a "scratch" sub-directory on a target shared volume (I use NFS)
2. Create a unique sub-directory in "scratch" for each host (HOST_ID)
3. Get the volume ID (VOL_ID) from the console
4. Change the location of scratch to /vmfs/volumes/VOL_ID/scratch/HOST_ID
5. Reboot
6. Enjoy

This works in ESXi 3 update 3 build 143129

[Cliff]

hp 530's are going on ebay for $125. You need to put in a sata hd, but everything else works out of the box. (32 bit only obviously)

[Matt Lachberg]

Chad, looking to enter as cheaply as possible with the hope to support vSphere which is 64bit as I understand. Any thoughts on what these two home grown systems would look like?

[Chad Sakac]

Matt - I can say with 100% confidence (as I'm running vSphere RC and it's a rock) that you can run it happily on the Intel P35/G33/G35 chipset and the Q6600 CPU. The really important thing is to check for the VT support from Intel. Doesn't boot from ATA, but just use the installable ESXi build on USB (which is cake, and handy).

Haven't tested FT on these systems (FT support is a little more finicky) - but should be able to run the new validation tool pretty quickly - will do and will post back.

My team is also having great success with the new i7 chipset/CPU combination - but they are considerably more expensive.

[Paul Wegiel]

Hi there, I am very happy and excited (woohoo!! hello to sleeping in my basement again after playing with vSphere into wee hours.. Sorry dear wife:-) to report that today I received the pieces for my third "whitebox" ESX server.

After an hour or so of putting the stuff together, I was able to finally start playing around with the ESX 4.0 bits. I was relieved (and impressed) to find out that vSphere had no problems at all installing and running on my new whitebox. You can see the details on my blog. I used the following components:
CPU: Intel Core 2 Quad Q6600 Kentsfield 2.4GHz LGA 775
Board: ASUS P5BV-M LGA 775 Intel 3200 Micro ATX Intel
Drive: Seagate ST3640323AS 640GB 7200 RPM SATA
Memory: CORSAIR 4GB (2 x 2GB) 240-Pin DDR2 SDRAM DDR2
CD-ROM: ASUS 22X DVD Burner Black SATA Model DRW-22B1S

vSphere looks AWESOME!!!

[spartango]

Btw, ESX can be hacked to run on any [x86*] hardware, rather easily. Really all you need to do is unpack the contents of the install disk and look for a .dd file, which can be dd'd onto a USB stick and used :D

[venom]

I think most people want an inexpensive, good, out of the box experience. So far I have read confusing reports from sites like this of people who used this motherboard but this or that had to be altered yada yada. Most of us know nothing of Linux.

Therefore, I present the perfect Out of the Box Motherboard that works with ESX 3.5 update 3 (or earlier) and ESX 3i

The Supermicro X7DAL-e-o (There is a +0 for 5.1 sound and $30 more)

You can use the onboard Intel Raid
Its dual LGA 775, meaning two quad xeons can work in this
16 gigs ram (I currently use 4x2gig sticks, ECC for 8 gigs)
Dual Intel Nics, works fine

So the above, while being $320, is by far the best inexpensive, no BS, no hassle MB I have found thus far without having to buy other nics or a RAID controller. Performance is great and i would recommend getting fast drives and if you go further get a better RAID card with on board memory.

But to screw around, this is it.

[David W]

Hi Chad,

Any chance you've been able to run the FT validation tool yet on your Q6600 server(s)? I should probably dig around for the tool myself but I haven't been able to dig it up yet and hadn't seen a follow up post from you...

Thanks,
David

[Michael Arnold]

Hey Chad,

How are things? Long time no talk as I left EMC a couple of months ago. Now with Vizioncore and definately give you some thanks for getting me moving on VMware technologies. The training you gave over a year ago was very useful.

Any thoughts on timings to update the "Homebrew" ESX Hardware list to support vSphere?

Thanks,
Michael

[Denis]

Hi Chad,

Will Lifeline 2.0 be released for SS4200-E platform ?
The last version available today is still 1.11...

Thank's,
Denis

[velmueugan]

Hi,

I purchased new computer.the configuration is i3 530 2.9Ghz processor,DH55TC Motherboard,2GB DDR3 1333Mhz RAM,500GB Seagate Sata Harddisk,Samsun Sata DVD drive.i am trying to install VMware ESX 3.5.But it’s showing error “Unable to find a supported device to write the VMware ESX Server 3i 3.5.0 image to and No driver found. Select Driver/Use a Driver Disk/Back”.How to install VMWare ESX 3.5.also i am install ESX 4.0.it’s showing error”No compatible network adapter found.please consult the products” and “Network-adapters returned critical failure” in vmkernal mode (ALT+F12).

I enabled VT technology in bios,

http://www.intel.com/products/desktop/motherboards/DH55TC/DH55TC-overview.htm

Please help me.Thanks in advance,Velmurugan

[Virtualization Talk]

Good write up, Chad.

I am also in the process of finishing up my home virtualization lab:
http://www.virtualizationtalk.net/58-building-home-virtualization-lab-on-a-budget/

There are number of other storage options available in addition to the intel entry storage. I am looking at QNAP and Thecus NAS devices.

[RonTom]

Will there ever be an iSCSI target for the Iomega IX2? Mine has been waiting for a year now :(

[Chad Sakac]

@RonTom - NO, there will not be an public build of the LifeLine code that adds an iSCSI target to the IX2 (at least the currently shipping version). The CPU on the box and the interface just can't take it. The internal builds just never got robust enough.

The IX4 and IX12 have been shipping with iSCSI support for some time now, though, and there's nothings saying that iSCSI on future IX2 class hardware won't happen.

Apologies.

[Chris Stable]

For the motherboard enthusiast, there is an interesting configurator / search engine at http://www.interloper.com/mbsearch.php Try it. I am sure you'll like it.