Updated 2011 vSphere ESXi 4.1 (and future) Homebrew/Whitebox parts list….

UPDATED Feb 22nd 2011, 9:40am – note that this build uses a MB that has a Realtek NIC (or more accurately MAC/PHY).   In my build notes, I purchased/repurposed Intel Pro GT/PT NICs from older whiteboxes.  If vSphere boots, but can’t load any network drivers, it will report an error like this: “vmkctl.HostCtlException Unable to load module /usr/lib/vmware/vkmod/vmfs3: Failure”.   There are two easy solutions: 1) customize your vSphere install ISO to support the Realtek NIC (great instructions here); 2) just buy the Intel NICs.   I **personally** recommend option 2 – not because option 1 doesn’t work (it does, I tried it), but because it means you have one more strange thing on an already Frankenstein whitebox :-)

So, as I’ve said before, when things get political, and get all messy – I like going into the lab and spending a solid day just playing with tech.   It’s how I get back into a zen state.   That home lab powers a ton of VMs, and let’s me play, learn and stay fresh – not just on EMC stuff (using Virtual Appliances) and VMware itself, but a lot of folks across the industry.   The home lab has 5 “mainstream” ESX hosts – and some other loosey-goosey ones I use for various purposes.

Staying technical…  Ooommm.

So – I’ve been playing with the latest ESXi 4.1 bits, and also with “future VMware software”, and noticed that my old whiteboxes – one of the most popular blogs I’ve done had the “parts list” – (basically Intel Q6600, Intel Q/P35 chipsets, 8GB of DDR2 mem, boot from USB,Intel PCI PRO/1000 GT and PCIe PT NICs) where starting to have some problems.

Specifically, with future VMware software, there were various UEFI chipset things – but I can’t blame VMware, we’re talking about very old, very non HCL hardware.

There are other great “building a home lab” articles on the internet.   I like this one, this one, and this one.

So – I did a refresh – and was stunned by how much you can get for how little.  With a disclaimer that I can’t vouch for anything other than my current experience if you want the shopping list, and the results – read on….

So – I wasn’t running into too many CPU issues with the Q6600, and the process of going to Nehalem or Sandybridge would have upped the cost substantively.   I was running into forward compatibility issues, and you can always use more RAM.

I do plan on doing a Sandybridge-based Home Lab build in a bit (when Intel fixes the P65 chipset issues), and will also post findings.

So, with out further ado – here are the details of my shopping list.   All prices are in CDN, and I bought them all at www.Newegg.ca

Step 1) I was looking for a new motherboard, ideally with LGA775 support, support for DDR3 and 16GB max memory configs, all at lowest cost.   Ideally, it would be ATX (lots of PCIe/PCI slots for NICs).   I settled on the MSI P43-C51 LGA 775 Intel P43 ATX Intel Motherboard – for $84.99

Upside:

• Low cost
• 16GB max mem capacity
• Support for my existing Q6600 CPUs
• Right form factor (ATX, lots of space, 4 DIMM slots)

Downside:

• No onboard video – necessitating an additional VGA card.

http://www.newegg.ca/Product/Product.aspx?Item=N82E16813130252

Step 2) It’s a bummer that there are so few low-cost, ATX MBs with onboard VGA.   If you go into mini-ATX, they are a dime a dozen (but you get less IO slots) and at the other end of the extreme, the Sandybridge P65 chipsets have onboard video, but come at a price point that doesn’t meet my “el cheapo” target.  

Since the MB doesn’t thave video – I threw in a cheap $29 VGA card (MSI N8400GS-D256H GeForce 8400 GS 256MB 64-bit DDR2 PCI Express 2.0 x16 HDCP Ready Low Profile Ready Video Card) – the only key thing with an el-cheapo VGA card (since you really don’t need anything fancy for this function) is to make sure it’s low profile, and that the heatsink/fan doesn’t stop you from putting a PCIe/PCI card into the adjacent slot.   Personally, I wanted something that used passive cooling as well (just a little less noise)

 

Step 3) The BIG upside of the new MB is that you can use 4GB DDR3 DIMMS.  It’s crazy, but I picked up 16GB (2 x 4GB kit for $94.99 x 2) for less than $200.   Think about that for a second.   Man.  It wasn’t a year ago that 6GB was “x-treme!”.   I picked the Patriot Signature 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) Desktop Memory with heatshield.   I went with DDR3 1600 because while it is way faster than my older Q6600 procs support, eventually when I jump the whole set of systems up to Sandybridge, the RAM will still be fast enough.  I might be crazy, but I tend towards kingston, corsair, crucial – rather than go to the rock-bottom prices.   Advice, run memtest on your systems before you go to far into the build.  I have a little USB utility flash thumbdrive I boot the systems from and run a few tests after the physical build.   Memory issues will make you pull your hair out later, so check up front.

I reused my previous heatsinks, HDDs and CD-R/DVD-R.   The latter two are insanely low-cost these days – you can get a 1TB HDD for around $50, and a Lite-On CD/DVD-R for $15.

For a case, this may seem a bit of overkill (but in the immortal words of Corporal John T Hannibal from the A-Team “overkill is underrated”), but I use rack-mount cases even in the home-lab.    You can even get surprisingly good rackmount cases pretty darn cheap.   I picked the iStarUSA D-300-PFS Black Steel 3U Rackmount Server Case for $89.99.  They are pretty solid.   A little bit of a weird layout IMO, but not bad.

 

NICs… This is another item that gets VMware whiteboxes in trouble.   The onboard NIC on the MB worked fine (and technically you could get by without adding any more – NOTE THE COMMENT BELOW, but I like having more interfaces on my whiteboxes.   I’ve found that if you stick with Intel GT (PCI ~ $40 each) or Intel PT (PCIe ~$80 each) you can count on them working well with ESX.

Note that this build uses a MB that has a Realtek NIC (or more accurately MAC/PHY).  If vSphere boots, but can’t load any network drivers, it will report an error like this: “vmkctl.HostCtlException Unable to load module /usr/lib/vmware/vkmod/vmfs3: Failure”.   There are two easy solutions: 1) customize your vSphere install ISO to support the Realtek NIC (great instructions here); 2) just buy the Intel NICs.   I **personally** recommend option 2 – not because option 1 doesn’t work (it does, I tried it), but because it means you have one more strange thing on an already Frankenstein whitebox :-)

So, tallying it all up:

• Motherboard: 84.99: http://www.newegg.ca/Product/Product.aspx?Item=N82E16813130252
• Video Card: $29.99: http://www.newegg.ca/Product/Product.aspx?Item=N82E16814127473
• 16GB Memory $94.99 x 2:  http://www.newegg.ca/Product/Product.aspx?Item=N82E16820220570
• Case: $89.99   http://www.newegg.ca/Product/Product.aspx?Item=N82E16811165083
• I re-used my power-supplies, but if you didn’t, you can get a decent powersupply for about $70.   Don’t scrimp on powersupplies.   You don’t actually need huge wattage for this use case, so buy a better 500W PS rather than a cheap 700W or 800W PS.
• I had the USB flash (for USB-based ESXi install), HDD (for HDD-based install) and the CD/DVD-R – but if you didn’t, you would need to spend another $80.
• I re-used my old Intel Q6600 CPUs, but if you didn’t have CPUs to reuse, the key with this config is you would need the older LGA775 socket type (remember, I picked the MB to house my older CPUs).   You can’t get the Q6600 anymore, but a wofldale-series Intel proc should work.  Here’s one as an example (dual core costs $89.00) http://www.newegg.ca/Product/Product.aspx?Item=N82E16819116093.   It’s worth pointing out two things:  1)  MAKE SURE the proc supports Intel VT. 2) you might be able to find another CPU/MB pairing that gives you more juice for less $$, but in my experience, the key for a happy ESX whitebox is the chipset (HDD controller, bios, onboard network), so go off at your own risk.

Grand total – for a great rack-mount quad core ESXi host with 16GB of RAM?: $632.96 CDN.

I ordered several of these (5 in total) to update the home-lab completely, and here’s the kit all at home.   It’s like Christmas :-)

So – after assembling and testing, I’ve found this config to work excellently not only with the current vSphere 4.1(both before and after update 1) ESXi, but also with future vSphere releases.   For me, that latter part is important, as I know I won’t need to update the homelab again for a while.

For shared storage, you have a ton of options.  You can go Iomega IX4, you can homebrew a Openfiler box, you can use VSAs like the EMC Celerra UBER VSA.   There is a great updated “using the EMC Celerra UBER VSA guide here, and demo video series here.

Enjoy – and remember, no self-respecting VMware fanboi should go without a home vSphere lab :-)

Comments

[Jason Boche]

Thank you for sharing Chad. I like a good lab post and people are interested in where they can find vSphere compatible gear at affordable prices.

[Derek]

Chad,

Good article! I put a whitebox together one a few months ago and posted the details here:

http://derek858.blogspot.com/2010/10/build-home-esx-4x-server-for-1000.html

[Andrew Fidel]

Chad, any chance for a VNX VSA or is the experience close enough to the Celerra to make it unwarranted?

[Steven Johnston]

Wow great post, nice price for all that. I currently have 2x DL380 G5 servers but can't run them for more than a weekend at a time because it costs far too much £££ and it makes my third bedroom/home office into a sauna. This seems like a great alternative.

[Martin G]

If anyone wants a vSphere lab on the very cheap; can I recommend the HP Microservers...okay, they don't have a lot of oomph but after the £100 cashback offer in the UK; they come in at £119.95 and everything works in ESX4.1. And they are very low-power, very quiet and very small....

You won't want to run lots of production workloads but for experimenting with..pretty much unbeatable value.

[Simon Gallagher]

Do you do much nesting ESX? Ala http://vinf.net/vTARDIS

Works scarily well, even if understanding all that virtual inside virtual does occasionally cause a black hole to open up inside your brain :)

[Kiel]

Hello, I had purchased almost the exact hardware you have for an at home ESXi lab. same motherboard, 16GB RAM, and a 2TB HDD. the only difference is that I am using an nvidia 8600gt i had laying around and am getting this error:

vmkctl.HostCtlException Unable to load module /usr/lib/vmware/vkmod/vmfs3: Failure

I have googled the error and gound people are saying this is due to unsupported hardware however you seem to have gotten it to work. So
Before i go buy a new GPU i was wondering if you had any install issue or if it just worked. Thanks.

[Chad Sakac]

@Kiel - I've updated the post with details on the root of the issue, and the workaround. Short version, it's due to the Realtek NIC, and there a way to make it work. There's also the easy way (add an Intel NIC).

[Chris Trotter]

For those who want to do a Core i5 system:

Mobo: Intel BOXDH55HC (ATX, 4xRAM slots, onboard video, Intel NIC) ($94.99)
RAM: Kingston KVR1066D3N7K2/8G ($83.49)
CPU: Core i5-760 Retail ($209.99)
Case: Same as above - iStarUSA D-300-PFS ($99.99)
PSU: Seasonic S12II 380 ($69.99)

I think this is what I'll be purchasing for my new lab. Will be using my Synology NAS for iSCSI with some PCI-X dual-port Intel NICs (have used them in the past with success). Install will be on a USB flash drive.

Not quite as cheap as the S775 system, but I do not have any CPUs, and the only quad-core S775 available is $159...only $50 more to step up to a Core i5, and even a little cheaper to get the Intel motherboard with onboard video and a NIC that might actually work with ESXi.

Anybody see anything wrong with that spec?

[Chad Sakac]

@Chris - thanks for the comment. Indeed - if the design point was "net new" from the get go, there are better ways. In my case, I needed something I could reuse a bunch of older Q6600 Socket 775 procs (and had a ton of Intel NICs sitting around).

As soon as fixed P67 chipset based MBs are out, I'm going to do a "net new whitebox".

[Chris Trotter]

@Chad - No, thank you! Your post was inspiration for me to take a different approach to setting up my lab. Previously I had set up a Core i7/12GB box to use with Workstation, but I've found that it's not the same thing as having an actual bare metal box.

Before I saw your post, I was going to get a matching i7 box and upgrade to 24GB/host, but I'd then have to replace my desktop...doing two 16GB hosts is a better choice from a cost perspective, and 16GB should be plenty to run a test environment.

I look forward to your P67 article!

[Delano]

@Chad,
I'm building a net new whitebox, what motherboard would you recommend to keep with the vsphere future releases while being el cheapo?

Thanks

[Coach Bags]

It's nice to your blog.

[Chanel J12]

We can believe we are being self-reliant and independent, and yet there is still clearly an overarching destiny, a Great Maker.

[Coach Purses]

You may believe these four proverbs:
While there is life, there is hope.
From small beginnings comes great things.
Money spent on the brain is never spent in vain.
Wise men learn by other men's mistakes; fools by their own.

[Chris Trotter]

Chad, ordered my new lab today, will let you know how it works out.

Core i5-760, 16GB, Intel BOXDH55HC. Ended up going with three i5 hosts versus two i7 hosts, a little more flexibility.

Detailed it here: http://practech.blogspot.com/2011/02/new-lab-setup.html

[Chris Trotter]

I've looked around quite a bit for a newer alternative to the Iomega IX4-200d. Have not found anything.

Only reason why I am hesitant about that unit is that it was released mid-2009, and is surely due for a refresh, yet the manufacturer's site has nothing about any refresh.

Can anyone comment about how long it will be before that unit is EOL?

[Chris Trotter]

One more comment on this Chad...how are you racking these? Two-post? Four-post? Cabinet?

Any pictures of the cases racked?

[Nat]

Can you please elaborate on future UEFI needs? How did your new motherboards cover this issue?

[Kiran]

Thanks, this has been helpful.

I am also trying to build an ESXi 4.1 box. I have an Intel q8300 and need to purchase a motherboard and 16gb of RAM.

Will the q8300 work with the MSI P43-C51 motherboard and the Patriot Signature 8GB (4 x 4GB) 240-Pin DDR3

Physically I know it will work but I have tested the CPU on an Asus motherboard and installed ESXi and it has not registered all 4 cores.

Please let me know if this setup will be fine?

Thanks

[Brian Santucci]

I am also trying to put together a whitebox for my home lab and was wondering if someone could leave comments on whether or not this hardware will work with ESX & ESXi 3.5, 4.0 and 4.1.

Intel BOXDX58SO2 LGA 1366 Intel X58 SATA 6Gb/s USB 3.0 ATX Intel Motherboard
Model #:BOXDX58SO2
Item #:N82E16813121470

PC Power & Cooling Silencer PPCS910 910W ATX12V 2.2 / EPS12V 2.91 SLI Certified CrossFire Ready 80 PLUS SILVER Certified ...
Model #:PPCS910
Item #:N82E16817703022

Intel Core i7-950 Bloomfield 3.06GHz LGA 1366 130W Quad-Core Processor BX80601950
Model #:BX80601950
Item #:N82E16819115211

Qty (2) G.SKILL Ripjaws Series 16GB (4 x 4GB) 240-Pin DDR3 SDRAM DDR3 1066 (PC3 8500) Desktop Memory Model F3-8500CL7Q-16GBRL
Model #:F3-8500CL7Q-16GBRL
Item #:N82E16820231309

Qty (2) SAMSUNG Spinpoint F3R HE103SJ 1TB 7200 RPM SATA 3.0Gb/s 3.5" Internal Hard Drive -Bare Drive
Model #:HE103SJ
Item #:N82E16822152238

Qty (2) LITE-ON 24X DVD Writer Black SATA Model iHAS424-98 LightScribe Support
Model #:iHAS424-98
Item #:N82E16827106335

CORSAIR CWCH50-1 High Performance CPU Cooler
Model #:CWCH50-1
Item #:N82E16835181010

comments soon would be awesome because I want to order the parts tomorrow and just want to make sure that this equipment will indeed work with ESX/ESXi.

Thanks Guys/Gals

[Kiran]

Just setup my ESXi server with the following spec:

MSI P43-C51
Intel Q8300 CPU
Kingston 2 x KVR1333D3N9K2/8G (total = 16GB)
2 x Intel Pro/1000 MT Desktop Adapter
1 x 2gb pen drive
4 x 1TB drives ( non RAID config)
1 x ati pci-e graphics card
All worked fine except for the installation which I needed an External USB cd-rom drive to install on to the Pen drive. Once this was done, all worked fine.

I did not want to setup raid since this is only my lab server and am looking to purchase a QNAP nas box..

Thanks for recommending the Mobo Chad.

[Kiran]

I have now built my ESXi 4.1 box. ( thanks for the recommendations)

1 x MSI P43-C51
1 x Intel Q8300
2 x Kingston KVR1333D3N9K2/8G (total 16GB RAM)
2 x Intel PRO/1000 MT Desktop Adaptor
1 x pci-e ATI graphics card
4 x 1TB Samsung Drives (non Raid)
1 x 2GB pen drive (installation)

Everything is working perfectly except for installation. I needed an external USB cdrom drive to install on OS on to the Pen drive as it would error out when using IDE or SATA Drive.

Nice and cheap solution for a homebrew ESXi 4.1 Box.

[Jerel]

I just built one and works great on 4.1 for months without a hiccup(no passthrough though)if anyone is interested.
Running Exchange 2010, 2008R2, centos, 2008 core install and an XP backup machine using Veeam.

SAMSUNG EcoGreen F4 HD204UI 2TB SATA 3.0Gb/s 3.5" Internal Hard Drive -Bare Drive
Item #: N82E16822152245 $159.98 ($79.99 each)

ASUS M4A88T-V EVO/USB3 AM3 AMD 880G USB 3.0 HDMI ATX AMD Motherboard
Item #: N82E16813131668 $94.99

G.SKILL Ripjaws Series 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Desktop Memory Model F3-10666CL9D-8GBRL
Item #: N82E16820231311 $49.99

AMD Athlon II X3 450 Rana 3.2GHz Socket AM3 95W Triple-Core Desktop Processor ADX450WFGMBOX
Item #: N82E16819103886 $77.99

Subtotal: $382.95

Add $70 for a perc 5i from Ebay and a (you'll need special cables and modify the heatsink and airflow. Even with a huge heatsink upgrade mine got hot until I removed a couple of slot blockoff plates to allow incoming air pulled from the main case fan to flow over the heatsink, then cool to the touch rather than burning hot). If you go this route make sure and get the actual 5i or 6i with memory and battery, there are some visual clues as to what is what here :
http://www.overclockers.com/dell-perc-5i-throughput-benchmarks/

Also added and Intel Dual pro 100/1000 adapter from Ebay for $10 (didn't even bother with the onbboard)

around $460 (already had Case and PS) with 4 TB of Storage (mine is mirrored so only 2TB) But the drives are 5400 RPM. Its great for me though this system is whisper silent, low heat output and low power usage. It replaced a Proliant dl380 G5 that was none of the above.
Running 4.1 off a Verbatim thumb drive.

[Yann Laliberte]

Hi Chad, great post. I'm wondering if your hardwares list is still good, because I want to build my own vmware servers at home, by exemple vmware 5.

Or if you have any good recommendations, you're welcome. I want to order asasp.

Thanks

Yann from Montreal.

[Tom]

Yann,

I'm looking to build a lab using VMware 5. How did your's turn out?

Tom

[Brian N]

Hanibal was a Colonel, not Corporal