Oracle, x86, VMware and update on support

Yup – it’s that topic again!  Everywhere I go, customers are successfully virtualizing Oracle on VMware.   Yet, the conversation about Oracle’s support stance continues unabated as well.

Well, I wasn’t just tilting at windmills in this post, and then in this post, and finally in this post.

Between EMC and VMware – we haven’t been sitting idle.   See this PR from today.

What have we done?

1. We’ve updated our support mutual positions – for the first time explicitly calling out on the VMware HCL and the EMC eLab Matrix that Oracle 11g (single instance for now, RAC via RPQ) is officially supported.   Also, we’ve setup a specific Virtual Escalation Team process for Oracle on vSphere on EMC.  
2. We’ve completed and documented a TON of solutions validation work to show how, across all EMC platforms and protocols (ASM and dNFS use cases), a simple solution can be designed to enable instant VtoP if absolutely needed – think of this as a “get out of jail free” card.

Oh – we also showed how using Fully Automated Storage Tiering, Solid State storage, and deduplication using Data Domain we could lower the acquisition cost by 30%, and the operating costs of Oracle 11g by 45% – while delivering equal or better performance.

Want to hear why virtualized Oracle is important from someone other than me?  Read the results of the Independent Oracle User’s Group survey of 400 individuals here.

My favorite quote:

“Sixty two per cent with Oracle on x86 use virtualization compared to 43 per cent for non x86-based Oracle sites. Cost savings associated with server hardware virtualization and an overall reduction in hardware costs were given as the leading reasons for companies adopting virtualization - 76 per cent and 59 per cent, respectively.”

So why have we done even more?  We’ve done this because our customers have asked us to.

So, trying to be customer-focused, we’ve put our money (all this work – well underway when I posted the earlier articles – is to tune of several million dollars work of expense) where our mouth is.  

If you just want to get on to business, virtualize Oracle with VMware vSphere and EMC Storage, snapshots and Replication Manager with confidence. For those looking for more details, read on…

Ok, the scoop:

Oracle’s policy on virtualization with VMware is governed by Metalink Document ID #249212.1.  While this document doesn’t explicitly say it’s NOT supported, the language is squishy enough to through FUD into the minds of Oracle folks.

EMC runs our Oracle 11i apps on VMware (one of the largest 11i apps deployments around, and a reference story for Oracle) and is in the process of migrating our 10g and 11g RAC environments over to vSphere 4.

Other large customers have also had success virtualizing Oracle, including during case resolution.   Others have simply refused to accept this support stance, and have made explicit support part of their negotiations with Oracle.

Remember, however, no one – certainly not VMware, and not EMC – can dictate, represent, or change Oracle’s support stance.   You, as the customer, are in the best position to influence what happens in your case.   Remember that each vendor (EMC and VMware included) are here to support you the best we can (and I suggest every customer remind Oracle of that).

Sometimes, this means giving customers answers they don’t like.  

An example is that I was on a customer call with a large EMC and VMware customer in NYC in the finance vertical two weeks ago.  They were using boot-from SAN on Symmetrix and replicating all their datastores and had matching HW at their DR site using SRDF.  They literally had 1:1 matching server hardware.  In VI3.5, this worked as a simple DR solution and they liked it.  

Well, this method is not officially supported by VMware (boot from ESX is and continues to be supported, I’m talking about the DR part), and in vSphere 4, the fact that this used to work broke (moving the service console to a VDMK as opposed to a partition on the VMFS volume meant that the volume gets resignatured, and then a reference to the COS location gets busted).  

The customer wanted to hear us say “we will support the workaround”.  The RIGHT answer was “look, this is not way we see customers wanting to do DR going fwd, and is not in the testing/support model.   While the workaround would work today, it could break tomorrow.  That’s why we’re recommending a different, and in our view better way to do DR”.  BTW the “better way” to do DR is to not focus on the physical hosts, but on the VMs – using SRM.   We also offered to help the customer at no charge with internal scripting, tools, and resources if they didn’t want to/couldn’t use SRM. 

It’s a fine line – and it’s possible to view the Oracle on VMware question in the same light as the example above.  Personally, I think they are materially different. In the case noted above, we could clearly point to a “technical something” (a complicated topic perhaps worth another post) where the customer was putting themselves at risk, and we had to stay away from the “easy answer” of “sure, just do the workaround”.   In the Oracle on VMware use case, there hasn’t been a technical reason (exception was back in the Linux 2.4 and ESX 2.x days). 

While every vendor clearly has agendas and priorities, when they don’t align with the customer’s, that often ends up with the vendor suffering long term.

Also – don’t get me wrong.  I think I know how strong Oracle’s technology and products are, and I understand how important they are in our customers IT stack.   It just seems out of whack (uncompetitive?) to suggest that a vendor in the stack could explicitly exclude things that are decoupled in the stack without any technical justification.  

Remember – the analogy isn’t: “Oracle explicitly calls out OS support, and VMware is an OS”, the analogy would be Oracle deciding that they don’t support a server platform – for example deciding one day that you can’t deploy on HP or Dell x86 servers, but only on Sun x86 (would be a shame if they killed those – they were fine servers).

But all that said – and trying to set expectations of what we can and can’t do… to increase confidence, we decided we could do more.

eLab has spent billions (yes, with a “b”) over the years on doing comprehensive end-to-end testing of infrastructure, and is widely regarded as one of the high watermarks of interoperability testing in our industry.  

Likewise, the EMC Proven Solutions program does integrated solution testing including enterprise apps on infrastructure at a very large scale.  

Both focus on a “push it until you break it” rather than a “prove that it works” model.   I don’t claim that these are perfect (what is?), but it’s a pretty heavy-duty set of work.

We worked with the Oracle-focused team at VMware and built a comprehensive test harness for Oracle on vSphere on EMC, and for the last couple of months have been putting it through it’s official paces.

Results?

• If you look here, you can see VMware’s updated support page for Oracle, and note that EMC is called out specifically.   I expect of course that other vendors will follow, and that’s a good thing.   
• Loads on the EMC side here.  If you look at the EMC support matrix, you can see EMC’s april updated calls this out: https://elabnavigator.emc.com.  For the first time, Oracle 11g Single Instance on VMware vSphere 4 is listed as explicitly supported on the EMC elab Support Matrix (previously we supported OSes only on top of vSphere – this is the first case where we call out a specific app use case, normally the ISV does this, but here the ISV is a bit truculent).  Obviously EMC cannot “support Oracle” (after all, Oracle provides fixes to their own products) – so what does thsi mean?   This means EMC cannot close an EMC support case until the customer is satisfied in a case where the configuration complies with the support matrix.   We will use every tool at our disposal (the tools that we use to support more than 75,000 joint Oracle/EMC customers today) to ensure that customer satisfaction.
• EMC is handling Oracle 11g RAC via RPQ (an EMC process) – not because we ran into problems, but it didn’t finish the same degree of testing that the single instance use case.
• EMC and VMware have created a specific support escalation procedure for customers using vSphere 4 with Oracle on EMC storage (Symmetrix, CLARiiON and Celerra NFS and dNFS).   This additional support model is based on extending the existing Site Recovery Manager Virtual Escalation Team people and processes.
• EMC’s Oracle Solutions Practice has developed a best practices guide (EMC doc: h6922) to assist with V2P migration if required by Oracle to perform root cause analysis. The document is posted on EMC PowerLink at the link below for EMC and EMC partners, and the link below goes directly to the article.

Here’s one of the 3 use cases covered in the doc, in this case using dNFS, Celerra snapshots enable instant VtoP.   Midrange block and Symmetrix block storage use cases are also covered and documented.  This uses a simple easy to use tool (EMC Replication Manager) that can be used directly by the Oracle DBA, no storage team needed (with full access control to make sure they don’t muck with other stuff on the array).

Follow the best practices (which are detailed) in this doc, and you eliminate the risk of someone at Oracle pushing back.   Since that metalink article (and I’m paraphrasing here) says: “we reserve the right to ask you to repro on a physical”, I can imagine the phone call now…

Oracle support: “Umm, I hate to ask, but is it running on VMware?”

Customer: “why yes, better than it did on the physical!  Until we ran into this issue – but I don’t think it’s vSphere”

Oracle support: “Umm, I have to tell you that I need you to reproduce this on a physical….”

Customer: “No problem.   Sit tight…. and there, it’s now on a physical.   Let me see… Problem is still occurring”.

Oracle support: “What?   Ok, let’s continue”…

What we have provided is a a ZERO RISK way to move forward (remember – EMC Replication Manager is a single pane of glass across all EMC replication products and integrates with vCenter/ESX APIs for VMware integration).   It is possible to create integrated snapshot replicas for VtoP use on demand, continuously, or point-in-time images – tens to hundreds depending on the configuration.

Now, if only we could do something about the archaic Oracle licensing model…. Oh I can’t, but YOU can.  Remember, you have choices.

I sincerely hope that customers view this as a positive development, and something that de-risks moving forward with Oracle on VMware.  

Special thanks to specific people for efforts on this program:

VMware:

• Tim Harris
• Aditya Vasudevan
• Chris Rimer
• Bill Wiley
• Tamao Nakahara
• Anand D'Souza

EMC:

• Oracle Solutions Team: Rich Wells, David Clarke, Jeff Browning
• Alliance Sales: Dan Thacker
• EMC Customer Support: Roy Backstrom
• E-Lab: Sowmya Narayanan, Gary Teleshevsky, Paul Bessa, Praphul Krottapalli, Shawn Murphy, Carolyn Muise
• EMC Public Relations: Jennifer Dreyer
• Technical Alliances: Jason Kotsaftis

Would love to hear your comments – are you more/less likely to move forward based on this?   Are you moving forward with Oracle on VMware?   Don’t bother with “will you change Oracle’s support stance?” comments – answer is: “We can’t.   Only you as Oracle customers can force that.”   What we have done is everything we can: proven that there is no technical issue, provided a cushion of enhanced support, and an easy “get out of jail free” card to work around any support question.

Comments

[Sketch]

The link doesn't work (within PowerLink), and I can't find the document "Oracle 11g virtual to physical replication..." PowerLink comes up with nothing. Can you re-post?

Thanks!

[Itzik Reich]

Hi,
Itzik Reich (EMC) here.
A financial customer of ours used or services to design and implement vCenter SRM, the 1st stage of the project was to virtualize ALL of his Physical Servers..everything went fine untill we reached the Oracle Servers, not because we had a technical issue but becasue Oracle told the customer that they will not fully support the solution, the customer simply told them that the next SAP module (HR) will be deployed on MS-SQL instead..two days later he received an official email from Oracle stating that they will give support (but still reserve the right to ask the customer to replicate the issue on a physical server, we have used to same techique highlighted in the pdf (RM with RDM formatted luns) so in a case of support issue (hasn't happen even once!) we can reverse to a physical solution..this resulted in a fully DataCenter virtualization project and a Wroking/Automated DR solution using vCenter SRM.

[RL]

We were working with an "IBM AIX LPAR-EMC Storage" combination as our main platform for oracle for many years.

Last year we did some test with "VM Linux-VMware running on Intel x5570 proccesors-EMC Storage" combinantion and the results were amazing. So that, we started to move all our Oracle databases from IBM SystemP servers to Intel&VMware severs.

It's good to know that we could have more support over this new platform.

We did that because we saved a lot (A LOT) of money and from my experince VMware virtualization technology is more flexible and better than IBM power virtualization arquitecture (Vmotion, HA, Cloning, VCB, DRS, etc).

[Keir]

Hey Chad,
Great post! Seems many steps towards a common goal. We are an EMC, VMware, Cisco and Oracle shop.
I'm lucky enough to have responsibility or involvement in all disciplines except for being an actual DBA. We are currently in the middle of migrating to an Oracle RAC implementation. Due to our inexperience with RAC we solicited an Oracle only consulting group for help. In the design phase I suggested we consider virtualizing this infrastructure due to the investment in it's redundancy, performance, etc. This was out of the question (sez the project team), considering "O"'s stance on virtualize that isn't their own. Well, we also run Oracle Unbreakable Linux for our Oracle App Server dev environment on top of their virtualization (OVM) offering. Yikes, what a mess OVM is. My group is pretty saavy, but administrating this environment was not your standard challenge. The only thing more difficult was finding good documentation, anywhere. As the project progressed, I tried again to avail. A short time later, we experienced a random OVM failure thus leaving a few OVMs inoperable. Luckily we happen to get it working. I took the opportunity to "strike while the iron hot" and pleaded my case, again. I took a concession in that it was decided to set up our dev environment on ESXi as a RAC-in-a-box rather than OVM. We still are rolling with physicals for the actual production RAC.
Yes, my fight is/was with the big '"O". But, I also had to win over my teammates, superiors and the consultants. They still fear having to call support. I had the whole dev environment built and running in almost half a day. That would have taken just shy of a week with OVM. In the end, we have a solid dev environment that I hope to use to build a case for rolling it into an ESX farm. Oracle is trying to strong arm their way into an arena they sorely lack in. It stinks, especially since when is no technical reason not to support the VM environment. You're right in that people have just shrugged it off as "that is the way it is". You are correct in your efforts and as a smaller player in the scheme of things, I thank you. Sorry for letting a vein on this one......

[Ianhf]

Chad,

Quick one, above you mention :-
"Oh – we also showed how using Fully Automated Storage Tiering, Solid State storage, and deduplication using Data Domain we could lower the acquisition cost by 30%, and the operating costs of Oracle 11g by 45% – while delivering equal or better performance."

Can you please point me towards the document and models that show this as none of the PDFs referenced in the PR actually show any form of numbers, TCO or ROI models? Or any comparison baselines...

Cheers
Ian

[Jay Weinshenker]

For customers who run Oracle E-Business (aka Oracle Apps) there simply isn't much to use to encourage Oracle to change their support policies. The ability to switch from say, Oracle E-Business to say, SAP, isn't a small endeavor.

Having said that, I strongly encourage any Oracle customer to strongly consider running Oracle virtualized on VMware. My company made the switch for our Tier 1 Oracle software (Oracle E-Business and Oracle Hyperion, and currently in process of moving Oracle Agile and Oracle Universal Content Manager) to running on VMware and our experience has been excellent.

Never once has Oracle denied us support due to VMware virtualization and the flexibility VMware provides with storage vMotion, SRM, etc allowed us recently to move from a CX3 to a CX4 with NO downtime. The ability to revert back in minutes when patching / upgrading a whole Oracle E-Business / Hyperion environments is a level of security we never had without VMware. I simply cannot recommend Oracle on VMware enough.

This comes from a guy (me) who has been an Oracle Certified Professional for 10+ years and a RedHat Certified Engineer / Technician for about 4 years.

[Ricky]

Great post! Seems many steps towards a common goal.I'm lucky enough to have responsibility or involvement in all disciplines except for being an actual DBA.
For more information about our products, click here

[Storage Geek]

Hi Chad - Good and interesting post. Just to echo Ian's comments around TCO - Do you have any documentation / in depth financial analysis that shows these cost saves (Acquisition costs down by 30% and Opex Costs down by 45%) - if this is the case - tht is great - but need to see the meat on the bones to understand more.

Thks

Stuart.

[Cnidus]

I'm trying to get to the EMC document, but it says its been moved or not accessible.... anyone got a copy they can send me, or know how I can access it.

douglas.youd_AT_uwa.edu.au

Cheers,
Doug

[Jason]

Hi, Chad!

THis document is not available and I got the following message:

Access Violation

The content you are trying to access is unavailable. This content may have expired, the link may be incorrect or you may not be entitled to access it with your account. Click here to return to the Powerlink home page.

We apologize for any inconvenience.