Virtual Geek

Big Open Source Week for EMC - Part 2: Greenplum is now an open MPP project

Earlier in the year - we announced that we were committing all the elements of the Pivotal Big Data Suite as Open Source projects.   That includes:

• Greenplum (which an Massively Parallel Processing data warehouse stack - ideal for massive data sets)
• HAWQ (is now Pivotal HDB which is a Hadoop native SQL engine - ideal for transactional SQL that is fully  and machine-learning algos on SQL).  Expect a lot more on this shortly.  It’s incubating, and needs things like Ambari integration - but it is AMAZING.   Hadoop Native SQL, and incredible capability, incredible performance.   People often use HIVE, or Impala for this - consider Pivotal HDB:

• MADLib (which is a machine-learning on SQL library).   Works great on Pivotal HDB. 
• Gemfire (which is a in-memory distributed data grid - unbelievable, stratospheric transactional performance - different than the idea Apache Spark use cases which tend to be iterative/ML type use caseS).   The Apache Geode project is based on this contribution.

… These are all Apache Software Foundation projects.   For our Apache Hadoop core, Pivotal is committed to the Open Data Platform initiative (ODPi) together with HortonWorks.   The ODPi effort aims to maintain as much of the ecosystem interoperable.   

Little sidebar - just like my OpenStack post here, this is an open ecosystem - so of COURSE EMC partners (in fact also resells!) not only Pivotal Big Data Suite, but also HortonWorks (also ODPi) and Cloudera.

The Pivotal Big Data Suite is a huge part of Pivotal’s business - and growing… So why open-source?   Answer - this is market that needs disruption, innovation as closed projects is simply too slow.   Many of the customers wanted features/functions/fixes faster than Pivotal could do alone, and saw open-source projects (even if they were not quite as good) as a way forward.

Why is this specific news about Greenplum important?  

• How about disrupting a $16B Data Warehouse market that is dominated by legacy players (Oracle, Teradata) who are unwilling/unable to disrupt their existing business models and technology stacks?
• How about 10+ years of R&D now as an open project?  
• How about the only material Open Source Software project which is a SQL-compliant data warehouse that scales to many, many PBs.

Any customer that believes in Open Source, is tired of massive bills to legacy players - it’s a good time to take a look at Greenplum :-)

 

Pivotal Cloud Foundry (a Linux Foundation project) has been a shining light for the rest of the EMC Federation - recognizing that for many efforts, open is the way to go.   It’s arguable that we should have done it earlier with Greenplum, HAWQ, Gemfire - but it’s not arguable that they are open now.

 

You can get the bits below, and please - contribute!

Get your SDS here (form to access ScaleIO)

Lots of cool new tech to play with when it comes to software-defined storage models.

Recall - SD* is about:

• Decoupled control and data planes (for compute/network/storage) - and control planes (which control policy) that are pure software.    The control plane abstraction enables abstraction, pooling, automation.  It's what ESX does for compute, NSX does for network, ViPR does for storage.
• Data planes (in storage land - these are the things that persist data and offer data services) that increasingly - wherever they can - are implemented as pure software (and therefore offer different deployment models - software + COTS or Software/Hardware appliances).
• Everything being driven programmatically via APIs.

Things like VSAN from VMware (great for all-VMware use cases), or EMC ScaleIO (targets larger scales - up to 100, even 1000's of nodes - and multiple different use cases - including VMware) are examples of #2.  

I'm seeing more, and more customers (including some big ones - see AT&T's view here - which is focused on SDN, but trust me, they are applying the same pressure on the storage domain) that want software + bring your own hardware models - which is exciting!

• EMC Scale IO is currently GA (v1.0) - and the 1.2 beta (more data services, specifically snapshots, native windows support on top of linux) is open!  You get access via this application form.    APPLY!!!!
• VSAN is also in a public beta, and VMware has stated a "1H 2014" target.  You get access via this application form. 

…   And, we are counting down the hours to the ViPR GA date (which will be the industry's first software-only, open, accessible, extensible storage control plane).

Updated "Homebrew" ESX Hardware List

One of the most popular 2008 articles I've posted was this one on how to build a home-brew ESX server.   Within EMC it's a game now for our nerd elite to come up with creative ways to do it cheaper, faster, smaller, and cooler :-)

Keep reading for the new (at least for now!) record holder, from Adam Jones a Sr. TC, VCP (and more acronyms behind his sig than I've ever seen :-) in Kansas City with a dual core, x86-64 (ergo future ready), 8GB machine....

Do you have your own home-brew story?   I'd love to hear it!

"Site Recovery Manager in a Can" - Celerra Virtual Appliance HOWTO 401

UPDATED Jan 19th, 2009 - the updated doc (v1.01) reflects SRM update 1, general minor corrections, and the removal of the ONLY step (Solutions Enabler) that required an EMC powerlink login.

Folks, here it is - the "all-in" step-by-step guide for setting up Site Recovery Manager with nothing more than two ESX servers (which you can even do with home-brew servers, as noted here) and the Celerra VSA.

If this is your first read, you can either just click on the document below or start the 101, 201, 301, 401 journey here.

This guide isn't the first one of these out there.  I think my respected friend Adam Carter at Lefthand did it first (Adam, hope you're enjoying HP!).  The NetApp one is pretty good too.   What I like about ours is the extreme completeness.   It's also nice that the Celerra VSA has no timeouts, no limits (except that you'e not allowed to use it in production).

A quick shout out...   This was absolutely yeoman's work by one of the newest members of the Global EMC VMware Specialist squad - Bernie Baker, with strong assists from one of the other newbies - Stephen Spellicy.   Both of these guys started relatively recently, and this work is (IMHO) FANTASTIC - it highlights that this stuff can be EASY (and that they have what it takes to have a early impact).  Bernie - you've set the bar high for the team and yourself - GREAT WORK!

Until the next major revision of the Celerra VSA (which will be linked to major feature releases), expect to see only one more "301"-level Celerra VSA post (downloading and using EMC Replication Manager for VMware for point-in-time, VM-consistent instant VM backup/restore) - we're warming up in the batter's box a series on Avamar Virtual Edition....

Celerra Virtual Appliance HOWTO 301 - Replicating Between Two Celerra VSAs

Glad to see that people are having downloading and success with the Celerra VM on VMware Workstation based on this original post.  I wanted to provide a quick "HOWTO" to help customers Replicate between two VSAs, for many reasons - one of the most fun being VMware Site Recovery Manager.

Celerra Replicator (I hope the Celerra Product team doesn't read this) is a RIDICULOUSLY inexpensive (particularly when compared with the competition) but very sophisticated remote replication capability.   It can support cool stuff like:

• 1:N and also N:1 replication fan-in/fan-out replication relationships (like with any technology there are operating envelopes and parameters)
• It can support cascading topologies (i.e. site one replicates to site 2 with a given frequency, and then site 2 replicates the data from site 1 to site 3 at a different frequency
• It has pretty sophisticated QoS mechanisms (i.e. you can setup different bandwidth use for different parts of the day
• You can replicate all sorts of configs - CIFS/NFS/iSCSI, and fully supports thin provisioning at the source or the target
• And of course - most interestingly of all - it's fully integrated with VMware Site Recovery Manager (so you can use this to build your own SRM playground.

Of course - SRM doesn't support NFS yet, so I focused this HOWTO on iSCSI - but it's very easy to see how you would do CIFS/NFS (just hit the other radio button in step 3)

Before you do this - make sure you have a working VSA - just follow the Celerra VSA HOWTO 101 here...

I've updated the OVF - tightened it up a bit.   You can get the new one here

Ok - have you got it?   Have you followed the 101?    Want to add more storage - just follow the Celerra VSA HOWTO 201 here.  

Ok... then read on...

Quickly then here - in this HOWTO:

1. Configuring NTP
2. Correcting the Replication Database (post Clone or deploy from OVF the Celerra VM's serial number is changed) - BIG thanks to Himabindu Tummala and Santosh PasulaReddy in Celerra Engineering for helping me figure this step out
3. Configuring an iSCSI LUN that will be used as a replication target
4. Configuring Celerra Replicator

One of our elite-delta squad VMware Specialists is putting the finishing touches on an "all in" doc and pair of Celerra VSAs to make setting up SRM with two Celerra VSAs a breeze, but you have every ingredient you need now with these HOWTOs (and SRM itself is a walk in the park).   

One quick note - remember that if you are looking for support - post on http://forums.emc.com, in the Celerra, Celerra Simulator forum. 

Next up to plate is the HOWTO on Avamar Virtual Edition, and I'm also going to do a HOWTO post on how to use the super-cool Replication Manager with the VSA.   Rep Manager - aka "RM" can actually be used with all EMC's primary storage platforms, for all sorts of replication/test and dev use cases - for Exchange, SQL Server, Oracle, and also VMware VMs and VMFS datastores - NFS datastores being supported shortly.

Read on......

Version 4 of the PowerVDI tool

This tool continues to get updated (thank you Jeff Purcell who did SO much on this and the previous release, Dan Baskette and the Celerra team!).   You can get the latest version here:

http://powervdi.vcloud9.net/

This version is much faster - at least 2-3 times faster than the old one.   In terms of speed, I think it would be hard to get faster than this - in our "playing around", it's at least 10x times faster than the upcoming View Composer approach.

So - does that mean we think the array is the place to do this rather than the ESX server (aka VMware View Composer) long term or in all cases? NO. 

There are so many things that can be done re: image preparation and handling and management better at the VMware layer than the storage array layer - which are both as important as speed.   All the approaches have similar cost savings.

We're embracing the upcoming stuff (stay tuned - another blog post coming on this in a couple weeks).   Over time, we're going to work to merge them as much as possible (i.e. use the VMware View tools to manage, but where speed of mass distribution is the key - you can choose to do it at the array layer rather than at the VMware layer (but it's managed at the VMware layer).

Celerra Virtual Appliance HOWTO 201 - Adding More Physical Storage

Glad to see that people are having downloading and success with the Celerra VM on VMware Workstation based on this original post.  I wanted to provide a quick "HOWTO" to help customers add more physical storage, so they have more to play with!

Before you do this - make sure you have a working VSA - just follow the Celerra VSA HOWTO 101 here...

I've updated the OVF - tightened it up a bit.   You can get the new one here

Ok - have you got it?   Have you followed the 101?    Ok... then read on...

Quickly then here - in this HOWTO:

1. Adding physical storage to the VM
2. Configuring the additional storage to the Celerra VSA (and how to remove storage)

Next up to plate are the 301 (setting up Celerra Replicator and SRM), and the Avamar Virtual Edition.

Read on......

HOWTO - Benchmark an iSCSI target (and any block target, including within VMs living in NFS datastores)

Someone asked me out of band  (thanks for asking, MattL) what I would do to quickly get a envelope of an iSCSI target.  

In all the post-VMworld followup, I don't want to do a long dissertation here, but quick off-the-cuff comments may be useful.   FYI, these are the quick set of tests we do during the application validation testing to quickly see if we're on the mark before we start with app workloads.   Read on for the details!

Celerra Virtual Appliance HOWTO 101 - Basic Setup and Configuration

Glad to see that people are having downloading and success with the Celerra VM on VMware Workstation based on this original post.  I wanted to provide a quick "HOWTO" to help, and will publish a 201 (setting up Replication and a 301 - configuring SRM).  

Most folks are doing well if they just RTFM, but I thought a little "hand-crafted walkthrough" might help others.   The standard documentation is geared towards VMware Workstation, which is the officially supported target (BTW, you don't need to ask, we're making ESX OVF a standard package :-), so some people stumbled a bit when trying to use it with ESX.

I've updated the OVF (some of the problems people were having were a result of the actual previous OVF I posted - SORRY - sheepishly) .  Make sure you download the new one if you're going to follow along (again, you can get that here)

Ok - have you got it?   Ok... then read on...

Quickly then here - in this HOWTO:

1. Importing the Virtual Appliance
2. Get the Celerra Sim up on ESX, and configuring the VMX
3. Configuring network interfaces (and what they all mean)
4. Making the Celerra Sim unique after cloning
5. Licensing Celerra features
6. Configuring Datamover interfaces and iSCSI targets
7. Getting iSCSI LUNs to the ESX cluster
8. getting NFS exports to the ESX cluster

I will post followups shortly:

• 201 series (adding storage to the simulator, configuring snapshots, configuring remote replication, etc)
• 301 series (configuring Site Recovery Manager, VDI mass replicas, etc).

In a little bit, I'm going to start posting bit by bit other EMC VMs (they are legion - ADM, Control Center, Avamar, Replication Manager, Networker 7.4.x, IT Compliance Analyzer, EMC Backup Advisor, and that's just getting started) with similar guides, so let me know if this is useful - BTW - they are also going through the formal release and VM Appliance programs, but I'm notoriously impatient, and think many of you out there are smart enough to get some fun out of these without too much hand-holding :-)

Read on......

UPDATED (old VSA is retired!!) Get Yer Celerra Virtual Machine here!

UPDATED (2:30pm 4/1/2009) – there is a new Celerra VSA here.   I’m keeping this old post for the threads.