« Introduction! | Main | Oh, and what does a Corporate VMware Infrastructure Lab look like? »

June 06, 2008

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00e552e53bd2883300e553067c0b8834

Listed below are links to weblogs that reference Building a Home VMware Infrastructure Lab:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Susan

WOW!!! I'm impressed and perplexed.... what do you use it for? You discussed all the "whats" and "hows" but not the "whys." Is it just for practice? Buy your supportive wife something special, "just for practice."

Chad Sakac

Thanks Susan - and I try to be as supportive for my wife as she is for me :-)

So what's the "Why" - answer - it's important (at least for me) to REALLY know what you're talking about. I manage large team in a large business, but fundamentally it's about expertise in the applied intersection of EMC and VMware technology. Having a home lab lets me do things that I wouldn't be able to otherwise.

For example - Storage Vmotion right now is only offically supported in Fiber Channel configurations, but it didn't make sense to me (it is after all essentially: 1) ESX snapshot; 2) filecopy ; 3) reparent snapshot (filesystem operation), so I wanted to see how it worked on iSCSI based VMFS volumes and NFS datastores. I could ask someone to do it for me - but what's the fun in that! Using the home lab (which I can access anywhere in the world), I can quickly see how it works.

Another key thing is it enables me to be hands on with EMC technologies and stay current - a hard task. Most software (and even a couple of arrays now - the Celerra, LifeLine, and arrays of the future) are available internally as VMs. I can just use VMware converter and import them to the cluster, and play with just about everything EMC produces, from backup tools, next-generation management tools, and arrays.

That's the why....

Oh - and one more thing - occassionally (it happens every few months), I start "jonesing" to build a new rig, and this is an outlet. I find something very satisfying and relaxing about building the stuff.

John Bergin

Love the setup. I have the initial components for a similar build out but on a smaller scale. Using 2 4U Chenbro cases I have configured a similar setup and I am working on getting a purpose built chenbro to through disks at and then run FreeNAS in a VM to act as my home NAS/Share.

Going with the home desktop cases is cheaper but I am not a fan for a long term deployment. If I could convince the guys at Dell to send me an R900 like I run in my infrastructure I would be a very happy kid. Without that kind of pull I have to make my own.

Glad to see other folks are doing this at home although your visitation stats must have gone off the chart when Twomey gave you a whole post. More than a little bit jealous.

Tomas Mieres

Hi Chad,
Good Article!!
I am starting to build my ESX servers in home. I would download virtual Celerra so I can play around with my ESX 3.5, but just wondering if it is also worth it to spend the time in getting Openfiler as another virtual iSCSI flavor? Or should I make the efforts to get an external array as well?
Thanks

Rich

Chad,

Thanks for putting this together, and thanks for letting me finally pay proper tribute to your work on my blog at http://vmetc.com!

Chad Sakac

Rich, my pleasure - http://vmetc.com is on my watchlist, along with http://www.virtualization.info, the always useful Scott Lowe's blog at blog.scottlowe.org and many others...

Drew

Hi Chad,

Great article! I am in the process of building a lab at home, so this article prove very timely for me. I was very interested in your mention of a Virtual Celerra. I would very much like to tinker with one of those. How do I go about getting my hands on one? Thanks!

Arne Fokkema

Nice lab! i'am also interested in the Virtual Celerra.

Keep up the great postings.

Ed Grigson

I'm interested in the virtual Celerra too. Where can I get the VM? Thanks.

Jsmith

Chad,
Excellent article. Glad to finally find the source of the shopping list. I'm very close to setting up my ESX3.5 lab. Eventually I'll build two servers so I'd like to go the AMD route because of cost. However, in reading around I get the feeling, compared to the Intel system, it's a greater risk of configuration gotchas. Can you recommend a current, under $100 motherboard from newegg.com that can still be purchased? The posted recommendation has been deactivated from newegg. It needs to be have its SATA, NICs, and Video to be ESX 3.5 compatible. Of the three, it seems that getting SATA to work off of the motherboard is the trickest. I need it to be capable of RAID0 across two or three disks. ESX & VMFS would need to be installed on the same set of disks. Also, the motherboard should be capable of going to at least 8GB. Thanks ahead for any suggestions.

Duncan

just moved into a new house, hope i can build a home-lab like this. although the neighborhood will probably notice the power-dip when I switch on 4 SAN's at the same time :-) Looks amazing.

Ben

I'd be interested in seeing the Virtual Celerra app. I've played with Xtravirt's san app, and openfiler but it would be nice to see the EMC app since we have been looking at the celerra line as a replacement for our current SAN (older dell/EMC branded FC box).

I setup a 2 box HP D530 "cluster" using the Xtravirt SAN VMs and its pretty impressive for what its running on. The funny part was ESX 3.5 installed on those boxes quicker and easier then Windows XP will!


Thanks!

Ole André Schistad

Hi Chad,

Just discovered your blog and am enjoying it a lot. You mentioned the Celerra Virtual Appliance - is the offer for a copy still on? Of so, can I please have one? :-)

ab

Hi!

Thanks for your post. One more interest for Virtual Celerra, can I also have one? :-)

Thank you very much.

Paul Shannon

Hi There.

Fantastic article. I can't believe two things, that you had the time to build it all and that you have an understanding enough family to let you get so deeply involved with a home rig.

Nothing by halves!!

Please count me in for a copy of Virtual Celerra please.

Thanks in advance

Paul Shannon

Daniel Eason

Great blog!

I am waiting for the housing market in the UK and then i will be actively seeking for my own datacentre (box room) :)

Shout me in for a cellera if possible

Stuart Mycock

Hi Chad,

Excellent work, this is exactly the inspiration/excuse I need to setup my ESX lab without the Mrs frowning at me :)

I've used OpenFiler in the past but I'm trying to keep the box count down so the Virtual Celerra app sounds perfect - is it still available? If so, please could you fire a copy over?

Many thanks,
Stuart.

Grega

Good setup.
I built a rig on the weekend ;
used a Gigayte GA-P35-DS3R board. $139 from MSY
Once u get a case, vid card, 4gb ram, CPU ect you'll be all up $499
u could do it a bit cheaper but that works
plus u need an intel NIC as the on board is not recognised (disable in bios)
the vi3.5 build goes thru OK but you have to boot off a USB DVD/CD rom and not the onboard, thats also not supported
i used port 1 (orange connector) on the mobo for my SATA drive.

all boots up fine and works great

for storage, on my core machine (not vi3 box) i have vmware workstation (thanks VCP) which i downloaded freenas and setup a couple of NFS shares. this worked just fine.

i then built a dedicated gbe connected NAS box using freenas again with a spare 80gb SATA drive formatted up as NFS - the physical box is HEAPS better performance but nothing like FC SAN - a CX3-40 is not in the budget and my missus is not understanding.

Grega

Oh FYi.........the EMC Celerra appliance is available on Powerlink for all those ppl who have access...

Cheers

Chad Sakac

Thanks to all those posting comments. The Celerra sim is available now - see this post here:

http://virtualgeek.typepad.com/virtual_geek/2008/06/get-yer-celerra.html

Also - for those looking for a new motherboard (the original ones were a bit dated - I'm going to do a new shopping list eventually, but already some stuff is updated.

For Intel, I've gone from their G33 based motherboard to a newer P35 - no special reason, just that you can't find ATX form factor G33 boards anymore. Here's the one I used:

http://www.newegg.com/Product/Product.aspx?Item=N82E16813121314R

This is on sale right now for $65 USD. Nice, 8GB support, newest CPU support, stable, fast. Confirmed the SATA to work out of the box with ESX 3.5.0.64067 and later. Onboard NIC doesn't work, so you need to get the intel NIC I posted in the shopping list. the other bummer is that it doesn't have onboard video, but any cheap video card will work fine.

UPDATE: Chad here... The P35 Intel motherboard worked GREAT. No video is a bummer, and it has no PS/2 keyboard/mouse, which sucks in the sense that my KVM doesn't work, but that's irrelevant anyway :-)

Re AMD motherboards, I can't say from personal experience (my old ASUS A8N-VM CSM continue to plug away (and you've GOT to love that they have "VM" in the product name :-) ). I would bet you $80 (the price of the board) that this board would work like a charm: http://www.newegg.com/Product/Product.aspx?Item=N82E16813138117


It has onboard video, so you're good there. Note that it has a realtek MAC/PHY, so the LAN won't work, you need the Intel card I spec'ed earlier. What's crazy is it supports 16GB of RAM!!!! It would be madness to do that at this point (will cost more than $1000 just for the RAM) - but you've GOT to know that the ram prices are going to drop.

It's amazing how cheap the AMD configs are.

UPDATE: Chad here... We haven't been able to get this BIOSTAR TForce TF720 motherboard's SATA controller recognized, so for now, take it off the list, use the older 680 motherboards we've gotten to work.. I'm going to keep poking at it, but for now - stick with the older boards in the original config.

jsmith

Chad,

After reading this article http://it.anandtech.com/IT/showdoc.aspx?i=3263 I figured it was best to go with a newer cpu. It seems every year the virtualization extensions get more efficient. So, I opted for your Intel Quad Core recommendation + EVGA nVidia 780i. After downloading the ESX 3.5 Update 1 82663, installing it, reboot....the system reports "Mounting root failed". I've tried it with and without RAID, twiddled numerous BIOS settings, upgraded the bios but all to no avail. You mentioned that people have been successful with this motherboard and sata drives. Would it be possible post their configuration settings and setup tips? I would like to avoid buying any more hardware (i.e scsi/sata controllers) or booting of a memory stick. Any help would be much appreciated!

Chad Sakac

Jsmith - sometimes SATA controllers need a little extra love when you find yourself far, far from the HCL :-)

From an EMC'er using a config much like yours:

"1 ESX on an EVGA 780i, dual-booting Vista (which runs on a RAID5), running on a SATA drive native, 8G ram, q6600 quad core. This board has 2 nvidia nics.
1 ESX on a XFX 680i LT board, 8G ram, q6600 quad core, running on a SATA drive native. This board has one nvidia nic, and I added a PCIE Intel Gig nic.

I ran into the "mounting root failed" issue after install. If this happens, follow these instructions (stolen from http://www.vm-help.com/esx/esx3.5/SATA_mounting_root_failed.html):

1) Setup a single SATA with no RAID.
2) Install ESX
4) On the first reboot, select the Service Console only (troubleshooting mode) boot option.
5) Let the host boot up, then log in.
6) Edit the file /etc/vmware/pciid/sata_nv.xml - change the last entry to use the device ID of your SATA controller (mine was 037f) and update the device name appropriately (mine was MCP55 SATA Controller, see note below)
7) After saving the file, run esxcfg-pciid
8) Reboot, and voila!

To find your device ID, us lspci. You'll have to use your best judgment on which one it is, but both of my motherboards (780i and 680i) were the same. Once you figure that out, visit this site to determine the device name:

http://pci-ids.ucw.cz/iii/?i=10de"

RichardB

Chad,

Any updates regarding the new shopping list that you mentioned in your 06.24.2008 post? I'd also like to know if you have received any feedback on the replacement AMD motherboard (http://www.newegg.com/Product/Product.aspx?Item=N82E16813138117) that you referenced? Any problems with this motherboard's SATA controller?

I look forward to your reply. Thanks for all of your contributions to the virtualization community. Best regards.


UPDATE: Chad here... We haven't been able to get this BIOSTAR TForce TF720 motherboard's SATA controller recognized, so for now, take it off the list, use the older 680 motherboards we've gotten to work.. I'm going to keep poking at it

Chad Sakac

Richard, sorry it's taken a while to respond - no one has told me if it has worked, but here's a proposal:

1) you buy it.
2) I will help you if you run into trouble
3) if we can't get it to work, I paypal you the $$, you ship me the motherboard :-)

I'm THAT confident it will work :-)

BLKJAK

Hey Chad,

If your proposal to Richard is universal, I will likely take you up on the offer this week. I sold my old ESX lab to a coworker who is preparing for his VCP exam, so I need to build 2 more servers. I like the motherboard you spec'd as it has enough slots for me to put 4 NICS in it for more flexibility. I would be putting 8GB of RAM in per server to start off.

Thanks for being a great resource. Are you going to be at the EMC booth during VMworld 2008?

Brad

BLKJAK

One question - is there an big performance difference between AMD and Intel dual/quad CPUs for a test environment? I plan on testing Exchange 2007 and System Center on my new ESX lab. Should I lean toward AMD or Intel?

Thanks,

Brad

Pete

Hi Chad,

I agree completely with your comment;

"I find something very satisfying and relaxing about building the stuff."

Putting together a home lab where you can fiddle around with technology is a must, even if your job is more business focused.

The ESX lab in my basement should be nominated for "the cheapest lab" award, but it works all the same.

1) ESX Server
Dell PowerEdge 400SC - single 2.7-P4 (about 5 years old)
1.5GB RAM

2) Open-E SAN
Dell Inspiron 8200 Laptop (about 6 years old) It only has 2x40GB drives and can't boot from USB. I had to turn one drive into the Open-E boot drive.

BTW - built in a couple days with stuff laying around on the floor.

Are you willing to sell your MAME arcade plans? ;)

BLKJAK

Chad,

I purchased components to make two ESX rigs using the Biostar TF720 you suggested. I have run into a few snags. First issue is that the ESX install hangs at "running /sbin/loader". I am able to get around this by installing using the "esx noapic" command. From there it doesn't see my SATA drive. I am currently trying to get around that one. I am trying to download the latest ESX build but VMware is doing maintenance today and I can't get anywhere.

Brad

BLKJAK

Update:

System board used: BIOSTAR TF720 A2+, 8GB of DDR2 800 Dual Channel RAM, AMD Phenon 9600 Quad core CPU, 3 Intel GB NICs (1 PCI-E and 2 PCI)

I have gotten the system to boot up and install the latest build of ESX to an IDE hard drive. No SATA so far. I had flash the BIOS and then disable APIC in the BIOS. I also had to use the NOAPIC switch to load ESX otherwise the install hangs when the Intel NICs load e1000.o. I am getting red messeges on the console saying that the "AMD Family 10h stepping B2 is not supported" and "Using PIC, make sure that if 'noapic' is used, it is on purpose".

I don't think either are hindering performance much. Both may be repaired with future builds of ESX that could support both. That may also resolve the SATA issues as well, unless Chad can help me out.

Brad

Chad Sakac

BLKJAK - I bet you just need to follow the "extra SATA lovin'" steps in the earlier comments:

"From an EMC'er using a config much like yours:

"1 ESX on an EVGA 780i, dual-booting Vista (which runs on a RAID5), running on a SATA drive native, 8G ram, q6600 quad core. This board has 2 nvidia nics.
1 ESX on a XFX 680i LT board, 8G ram, q6600 quad core, running on a SATA drive native. This board has one nvidia nic, and I added a PCIE Intel Gig nic.

I ran into the "mounting root failed" issue after install. If this happens, follow these instructions (stolen from http://www.vm-help.com/esx/esx3.5/SATA_mounting_root_failed.html):

1) Setup a single SATA with no RAID.
2) Install ESX
4) On the first reboot, select the Service Console only (troubleshooting mode) boot option.
5) Let the host boot up, then log in.
6) Edit the file /etc/vmware/pciid/sata_nv.xml - change the last entry to use the device ID of your SATA controller (mine was 037f) and update the device name appropriately (mine was MCP55 SATA Controller, see note below)
7) After saving the file, run esxcfg-pciid
8) Reboot, and voila!

To find your device ID, us lspci. You'll have to use your best judgment on which one it is, but both of my motherboards (780i and 680i) were the same. Once you figure that out, visit this site to determine the device name: http://pci-ids.ucw.cz/iii/?i=10de

Let me know if that works. You shouldn't need a newer ESX build.

BLKJAK

Thanks for the tips Chad. Here is the issue...maybe it is my misunderstanding. If I attach an 80GB SATA drive to the controller and boot to the ESX cdrom for setup, ESX doesn't see a hard disk to install to. I can install to an IDE drive, but there isn't a partition for VMFS then. After an installation to IDE, the storage tab in the VI client doesn't show the sata controller or drive. I think this is a different scenario to what you suggest. Or am I wrong?

Brad

BLKJAK

Chad,

From the ESX console I issued the lspci command. It does not list a SATA controller anywhere in the list. Thoughts?

Brad

Chad Sakac

Brad - you are safe, my friend, my commits when stated publicly, are a commit - I'll cover your cost if we can't make this work :-)

Ok, that said - can you try the MCP55 setting (037e and 037f)? Not sure if Nvidia updated the SATA controller from the 680 to the 720, but I doubt it.

I will be at VMworld - if we don't have this working by then, bring it if you can! Meet me in my booth or in my session - it's a joint VMware/Cisco/EMC session(KN EMC)

BLKJAK

One new piece of information...I changed the SATA controller to use AHCI in the BIOS. Now I see 0ad4 showing up which is a MCP78S GeForce 8200 AHCI controller, which is SATA I believe. I am going to follow the instructions above to see what happens. :)

Chad Sakac

Brad, you're almost there!

BLKJAK

Chad, what is this?

"Class 0106: nVidia Corporation: Unknown device 0ad4 (rev a2)"

Brad

BLKJAK

Here's another discovery. If I set the BIOS to RAID, the lspci shows this:

"00:09.0 RAID bus controller: nVidia Corporation: Unknown device 0ad8 (rev a2)"

0ad8 isn't in the list.

BLKJAK

More of the ongoing saga of the BIOSTAR TF720...
I was able to successfully install Kubuntu on box #1 on SATA. I opened a terminal session and ran "lspic" and printed out the results. The print out showed two interesting lines:
00:06.0 IDE interface: nVidia Corporation Unknown device 0759 (rev a1)
00:09.0 IDE interface: nVidia Corporation Unknown device 0ad0 (rev a2)
I then disabled the onboard IDE controller and rebooted and ran lspci again. This time I only see:
00:09.0 IDE interface: nVidia Corporation Unknown device 0ad0 (rev a2)
I am assuming this is my SATA controller. I edited the sata_nv.xml and changed the last entry to be MCP55 with the address of 0ad0 and then ran esxcfg-pciid, then rebooted. ESX saw the controller, but no targets were seen. I then changed the name to MCP785 but there was no love.

My concern is that ESX doesn't see the SATA controller on install...or it is not seeing my drive. I am not sure which one. I am happy that Kubuntu sees the SATA components fine, which leads me to believe that the ESX kernel isn't new enough to see my hardware.

BLKJAK

I should have noted in the above post that I built my second ESX server whitebox and ran the tests on ESX installed to an IDE hard drive. Thanks.

Chad Sakac

Brad - are you going to be at VMworld? I'll take that board of your hands at your cost - I'm a man of my word! Then I'll plug away at it - I remain convinced there is a way....

Also, I do have future ESX builds....

BLKJAK

I will indeed be at VMworld. I am coming in on Sunday and heading to a Vizioncore welcome event :) I appreciate your offer, but these are two screaming machines for what I paid and I am happy with them. I can run IDE for now.

I could test those ESX builds for you :)

Let me know when you will be at the EMC booth and I will stop by with my thumb drive!

Brad

Keith

Very nice rig I have a simialr setup also. Also, can't beat Wonder Boy, Rygar or Bubble Bobble on Mame;)

Just to note www.etherboot.org has roms which can be flashed onto nics (works a treat for the Intel Pro/1000's $30-$50 on ebay) and enable boot from san (like a HBA). EMBoot have similar but costs $100 plus, I believe. I setup an OpenSolaris x86 box with loads of 500gb Sata drives to behave like a SAN. ZFS and RaidZ are interesting to play with especially when trying to snapshot vmdk's (and raw os images also). Have you any PXE setups? Would appreciate some feedback when you have time.

Thank you,

Keith

Mike Foley

Hey Chad,

Fellow EMC (RSA) geek here with my new home config.

. Intel BOXDQ35JOE LGA 775 Intel Q35 Micro ATX motherboard
(bought because it supports vPro so I can do remote power on/off if needed)
. Intel Q6600 Quad-core Kentsfield 2.4GHz CPU
. ZEROtherm ZEN FZ120 120mm CPU Cooler
. 4GB Corsair memory
. 4 Seagate Barracuda 500GB 7200.11 32MB cache SATA drives
. 1 LSI Logic SATA hardware RAID card.
. 1 Corsair 520W power supply
. Cheap case (Gonna change that in a few months)

So far, so great. I converted my old Linux server to a VM. Moved my VPN and ESVA spam filter VM's over. Installed a FreeNAS VM.

The system boots ESXi off a USB stick. VirtualCenter runs in a VM on a laptop that runs VMware Server 1.x.

I've retired two systems and am down to the laptop and the ESXi server. I need to put a Kill-A-Watt on the line to see how it's doing. :)

mike
RSA/EMC employee/technical marketing guy
Resident VMware geek at RSA
Collector of hardware
I used to debug/QA Scott Davis' code when we worked in the VMS group at DEC.

VMKing

It would like much simpler to use, vmware workstation and openfiler to set up vmware esx infrustructure. Loads of tutorial online, like xtravirt.

VMKing.

jhoefker

Was there ever any successful resolution to using the Biostar TF720, or is this simply a victim of esxi not having the proper SATA drivers for the MCP78S controller?

Sorry for dredging up an old post, but google isn't very forthcoming on esxi biostar tf720 information!

mr mel

Unbelievable, other sites never went into detail on the http://pci-ids.ucw.cz/ which was crucial, they farted around on the sata_nv without explaining how to add the correct label for the sata controller (ok, I'm new to ESX and I can figure 2 an 2, but still give abit more detail right guys:). You made my day virtual geek, whoever you are. Everyone, follow this guys advice to the T, I did and it worked for me. Just so I'm not a hypocrit, it looks kind of like this at the end of the file, I'll let you tab yourself. Mine is a ECS GeForce6100PM-M2, so my PCIID on that link is:

http://pci-ids.ucw.cz/v2.2/pci.ids

03f6 MCP61 SATA Controller
1849 03f6 939NF6G-VSTA Board

Good luck.

sata_nv.xml should look like this at the end:

sata_nv

MCP61 SATA Controller

mr mel

oops, wrote too much, was cut off by formatting as well, just add the slash and arrow signs appropriately.

device id="03f6"
vmware label="scsi"
driver sata_nv
driver
vmware
name MCP61 SATA Controller name
device
vendor
pcitable

Jim Tomiser

You'd posted an updated "record cheap" dual core 8GB config on Jan. 5, 09. That link isn't working anymore. I'd love to see the hardware list if you have it.

Thanks! Jim

Mike Laverick

Chad, what success have you had with these "servers" running vSphere4. Also if you want to remotely rebuild the ESX hosts - what are using in lieu of ILO/RAC board - an IP KVM?

Paul Wegiel

Hi Mike: Today I successfully installed and confirmed working vSphere on my newest whitebox. No issues at all on the following HW:
CPU: Intel Core 2 Quad Q6600 Kentsfield 2.4GHz LGA 775
Board: ASUS P5BV-M LGA 775 Intel 3200 Micro ATX Intel
Drive: Seagate ST3640323AS 640GB 7200 RPM SATA
Memory: CORSAIR 4GB (2 x 2GB) 240-Pin DDR2 SDRAM DDR2
CD-ROM: ASUS 22X DVD Burner Black SATA Model DRW-22B1S

install went w/o a glitch.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

  • BlogWithIntegrity.com

Disclaimer

  • The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC. This is my blog, it is not an EMC blog.

Enter your email address:

Delivered by FeedBurner