« Introduction! | Main | Oh, and what does a Corporate VMware Infrastructure Lab look like? »

June 06, 2008

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00e552e53bd2883300e553067c0b8834

Listed below are links to weblogs that reference Building a Home VMware Infrastructure Lab:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Susan

WOW!!! I'm impressed and perplexed.... what do you use it for? You discussed all the "whats" and "hows" but not the "whys." Is it just for practice? Buy your supportive wife something special, "just for practice."

Chad Sakac

Thanks Susan - and I try to be as supportive for my wife as she is for me :-)

So what's the "Why" - answer - it's important (at least for me) to REALLY know what you're talking about. I manage large team in a large business, but fundamentally it's about expertise in the applied intersection of EMC and VMware technology. Having a home lab lets me do things that I wouldn't be able to otherwise.

For example - Storage Vmotion right now is only offically supported in Fiber Channel configurations, but it didn't make sense to me (it is after all essentially: 1) ESX snapshot; 2) filecopy ; 3) reparent snapshot (filesystem operation), so I wanted to see how it worked on iSCSI based VMFS volumes and NFS datastores. I could ask someone to do it for me - but what's the fun in that! Using the home lab (which I can access anywhere in the world), I can quickly see how it works.

Another key thing is it enables me to be hands on with EMC technologies and stay current - a hard task. Most software (and even a couple of arrays now - the Celerra, LifeLine, and arrays of the future) are available internally as VMs. I can just use VMware converter and import them to the cluster, and play with just about everything EMC produces, from backup tools, next-generation management tools, and arrays.

That's the why....

Oh - and one more thing - occassionally (it happens every few months), I start "jonesing" to build a new rig, and this is an outlet. I find something very satisfying and relaxing about building the stuff.

John Bergin

Love the setup. I have the initial components for a similar build out but on a smaller scale. Using 2 4U Chenbro cases I have configured a similar setup and I am working on getting a purpose built chenbro to through disks at and then run FreeNAS in a VM to act as my home NAS/Share.

Going with the home desktop cases is cheaper but I am not a fan for a long term deployment. If I could convince the guys at Dell to send me an R900 like I run in my infrastructure I would be a very happy kid. Without that kind of pull I have to make my own.

Glad to see other folks are doing this at home although your visitation stats must have gone off the chart when Twomey gave you a whole post. More than a little bit jealous.

Tomas Mieres

Hi Chad,
Good Article!!
I am starting to build my ESX servers in home. I would download virtual Celerra so I can play around with my ESX 3.5, but just wondering if it is also worth it to spend the time in getting Openfiler as another virtual iSCSI flavor? Or should I make the efforts to get an external array as well?
Thanks

Rich

Chad,

Thanks for putting this together, and thanks for letting me finally pay proper tribute to your work on my blog at http://vmetc.com!

Chad Sakac

Rich, my pleasure - http://vmetc.com is on my watchlist, along with http://www.virtualization.info, the always useful Scott Lowe's blog at blog.scottlowe.org and many others...

Drew

Hi Chad,

Great article! I am in the process of building a lab at home, so this article prove very timely for me. I was very interested in your mention of a Virtual Celerra. I would very much like to tinker with one of those. How do I go about getting my hands on one? Thanks!

Arne Fokkema

Nice lab! i'am also interested in the Virtual Celerra.

Keep up the great postings.

Ed Grigson

I'm interested in the virtual Celerra too. Where can I get the VM? Thanks.

Jsmith

Chad,
Excellent article. Glad to finally find the source of the shopping list. I'm very close to setting up my ESX3.5 lab. Eventually I'll build two servers so I'd like to go the AMD route because of cost. However, in reading around I get the feeling, compared to the Intel system, it's a greater risk of configuration gotchas. Can you recommend a current, under $100 motherboard from newegg.com that can still be purchased? The posted recommendation has been deactivated from newegg. It needs to be have its SATA, NICs, and Video to be ESX 3.5 compatible. Of the three, it seems that getting SATA to work off of the motherboard is the trickest. I need it to be capable of RAID0 across two or three disks. ESX & VMFS would need to be installed on the same set of disks. Also, the motherboard should be capable of going to at least 8GB. Thanks ahead for any suggestions.

Duncan

just moved into a new house, hope i can build a home-lab like this. although the neighborhood will probably notice the power-dip when I switch on 4 SAN's at the same time :-) Looks amazing.

Ben

I'd be interested in seeing the Virtual Celerra app. I've played with Xtravirt's san app, and openfiler but it would be nice to see the EMC app since we have been looking at the celerra line as a replacement for our current SAN (older dell/EMC branded FC box).

I setup a 2 box HP D530 "cluster" using the Xtravirt SAN VMs and its pretty impressive for what its running on. The funny part was ESX 3.5 installed on those boxes quicker and easier then Windows XP will!


Thanks!

Ole André Schistad

Hi Chad,

Just discovered your blog and am enjoying it a lot. You mentioned the Celerra Virtual Appliance - is the offer for a copy still on? Of so, can I please have one? :-)

ab

Hi!

Thanks for your post. One more interest for Virtual Celerra, can I also have one? :-)

Thank you very much.

Paul Shannon

Hi There.

Fantastic article. I can't believe two things, that you had the time to build it all and that you have an understanding enough family to let you get so deeply involved with a home rig.

Nothing by halves!!

Please count me in for a copy of Virtual Celerra please.

Thanks in advance

Paul Shannon

Daniel Eason

Great blog!

I am waiting for the housing market in the UK and then i will be actively seeking for my own datacentre (box room) :)

Shout me in for a cellera if possible

Stuart Mycock

Hi Chad,

Excellent work, this is exactly the inspiration/excuse I need to setup my ESX lab without the Mrs frowning at me :)

I've used OpenFiler in the past but I'm trying to keep the box count down so the Virtual Celerra app sounds perfect - is it still available? If so, please could you fire a copy over?

Many thanks,
Stuart.

Grega

Good setup.
I built a rig on the weekend ;
used a Gigayte GA-P35-DS3R board. $139 from MSY
Once u get a case, vid card, 4gb ram, CPU ect you'll be all up $499
u could do it a bit cheaper but that works
plus u need an intel NIC as the on board is not recognised (disable in bios)
the vi3.5 build goes thru OK but you have to boot off a USB DVD/CD rom and not the onboard, thats also not supported
i used port 1 (orange connector) on the mobo for my SATA drive.

all boots up fine and works great

for storage, on my core machine (not vi3 box) i have vmware workstation (thanks VCP) which i downloaded freenas and setup a couple of NFS shares. this worked just fine.

i then built a dedicated gbe connected NAS box using freenas again with a spare 80gb SATA drive formatted up as NFS - the physical box is HEAPS better performance but nothing like FC SAN - a CX3-40 is not in the budget and my missus is not understanding.

Grega

Oh FYi.........the EMC Celerra appliance is available on Powerlink for all those ppl who have access...

Cheers

Chad Sakac

Thanks to all those posting comments. The Celerra sim is available now - see this post here:

http://virtualgeek.typepad.com/virtual_geek/2008/06/get-yer-celerra.html

Also - for those looking for a new motherboard (the original ones were a bit dated - I'm going to do a new shopping list eventually, but already some stuff is updated.

For Intel, I've gone from their G33 based motherboard to a newer P35 - no special reason, just that you can't find ATX form factor G33 boards anymore. Here's the one I used:

http://www.newegg.com/Product/Product.aspx?Item=N82E16813121314R

This is on sale right now for $65 USD. Nice, 8GB support, newest CPU support, stable, fast. Confirmed the SATA to work out of the box with ESX 3.5.0.64067 and later. Onboard NIC doesn't work, so you need to get the intel NIC I posted in the shopping list. the other bummer is that it doesn't have onboard video, but any cheap video card will work fine.

UPDATE: Chad here... The P35 Intel motherboard worked GREAT. No video is a bummer, and it has no PS/2 keyboard/mouse, which sucks in the sense that my KVM doesn't work, but that's irrelevant anyway :-)

Re AMD motherboards, I can't say from personal experience (my old ASUS A8N-VM CSM continue to plug away (and you've GOT to love that they have "VM" in the product name :-) ). I would bet you $80 (the price of the board) that this board would work like a charm: http://www.newegg.com/Product/Product.aspx?Item=N82E16813138117


It has onboard video, so you're good there. Note that it has a realtek MAC/PHY, so the LAN won't work, you need the Intel card I spec'ed earlier. What's crazy is it supports 16GB of RAM!!!! It would be madness to do that at this point (will cost more than $1000 just for the RAM) - but you've GOT to know that the ram prices are going to drop.

It's amazing how cheap the AMD configs are.

UPDATE: Chad here... We haven't been able to get this BIOSTAR TForce TF720 motherboard's SATA controller recognized, so for now, take it off the list, use the older 680 motherboards we've gotten to work.. I'm going to keep poking at it, but for now - stick with the older boards in the original config.

jsmith

Chad,

After reading this article http://it.anandtech.com/IT/showdoc.aspx?i=3263 I figured it was best to go with a newer cpu. It seems every year the virtualization extensions get more efficient. So, I opted for your Intel Quad Core recommendation + EVGA nVidia 780i. After downloading the ESX 3.5 Update 1 82663, installing it, reboot....the system reports "Mounting root failed". I've tried it with and without RAID, twiddled numerous BIOS settings, upgraded the bios but all to no avail. You mentioned that people have been successful with this motherboard and sata drives. Would it be possible post their configuration settings and setup tips? I would like to avoid buying any more hardware (i.e scsi/sata controllers) or booting of a memory stick. Any help would be much appreciated!

Chad Sakac

Jsmith - sometimes SATA controllers need a little extra love when you find yourself far, far from the HCL :-)

From an EMC'er using a config much like yours:

"1 ESX on an EVGA 780i, dual-booting Vista (which runs on a RAID5), running on a SATA drive native, 8G ram, q6600 quad core. This board has 2 nvidia nics.
1 ESX on a XFX 680i LT board, 8G ram, q6600 quad core, running on a SATA drive native. This board has one nvidia nic, and I added a PCIE Intel Gig nic.

I ran into the "mounting root failed" issue after install. If this happens, follow these instructions (stolen from http://www.vm-help.com/esx/esx3.5/SATA_mounting_root_failed.html):

1) Setup a single SATA with no RAID.
2) Install ESX
4) On the first reboot, select the Service Console only (troubleshooting mode) boot option.
5) Let the host boot up, then log in.
6) Edit the file /etc/vmware/pciid/sata_nv.xml - change the last entry to use the device ID of your SATA controller (mine was 037f) and update the device name appropriately (mine was MCP55 SATA Controller, see note below)
7) After saving the file, run esxcfg-pciid
8) Reboot, and voila!

To find your device ID, us lspci. You'll have to use your best judgment on which one it is, but both of my motherboards (780i and 680i) were the same. Once you figure that out, visit this site to determine the device name:

http://pci-ids.ucw.cz/iii/?i=10de"

RichardB

Chad,

Any updates regarding the new shopping list that you mentioned in your 06.24.2008 post? I'd also like to know if you have received any feedback on the replacement AMD motherboard (http://www.newegg.com/Product/Product.aspx?Item=N82E16813138117) that you referenced? Any problems with this motherboard's SATA controller?

I look forward to your reply. Thanks for all of your contributions to the virtualization community. Best regards.


UPDATE: Chad here... We haven't been able to get this BIOSTAR TForce TF720 motherboard's SATA controller recognized, so for now, take it off the list, use the older 680 motherboards we've gotten to work.. I'm going to keep poking at it

Chad Sakac

Richard, sorry it's taken a while to respond - no one has told me if it has worked, but here's a proposal:

1) you buy it.
2) I will help you if you run into trouble
3) if we can't get it to work, I paypal you the $$, you ship me the motherboard :-)

I'm THAT confident it will work :-)

BLKJAK

Hey Chad,

If your proposal to Richard is universal, I will likely take you up on the offer this week. I sold my old ESX lab to a coworker who is preparing for his VCP exam, so I need to build 2 more servers. I like the motherboard you spec'd as it has enough slots for me to put 4 NICS in it for more flexibility. I would be putting 8GB of RAM in per server to start off.

Thanks for being a great resource. Are you going to be at the EMC booth during VMworld 2008?

Brad

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

  • BlogWithIntegrity.com

Disclaimer

  • The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC. This is my blog, it is not an EMC blog.

Enter your email address:

Delivered by FeedBurner