Fun with completely not supported, pre-alpha EMC tools: Part 2

One of the things that makes me most happy about my job is meetings, planning, and budgeting… NOT.

In all seriousness – those are necessary evils. What DOES make me happy is working with my team, my fellow EMCers and EMC partners, with customers and with engineering. I’m really lucky to get exposed to a lot of ideas, a lot of technologies, and am surrounded by a lot of people who are constantly coming up with cool – and frankly FUN – ideas.

This is the second of a 3 part series on this topic. Each one will “out” a killer tool. Each of them will grow over time. Each of them is free. For now, all of them are not supported (and in one case not even in circulation).

I will be providing a pile of “Santa Chad” awards to people who innovate on top of the two of these ideas. Think iPads, Iomega PX4s, and VNXe 3300s. Remember, it COULD be throwaway work – since they are not supported products per se. But those of you that know me – there’s always methods to my madness :-)

The first one was “Hyperic/vCenter Operations/EMC VNX NFS Uber Mashup” here.

This is the second one: “Beta of Analyzer Helper”:

The majority of EMC customers either use EMC Unified/VNX platforms, or VMAX primary storage platforms (with legions of Avamar, Data Domain, Isilon, Centera, etc, etc).   For midrange requirements – EMC’s primary "go to” platform is the VNX.  It’s an example of of the “swiss army knife” storage category – does everything under the sun relatively well in all ways.

So – in a VNX, filesystem functions are handled by file blades running VNX Operating Environment (OE) File code, and the block functions are handled by storage processor blades running VNX OE block code.

In the previous post in the series, the tools for providing very rich performance data on the VNX OE File layer of the stack were shown.   BTW – if you don’t like it in Hyperic/vCOPS, Clint has shown how you can just throw any data gathered via powershell into a general vCenter plugin here. (powershell is the basis of a lot of really powerful stuff).   Remember – start with his seminal post on Powershell here.

But – what if you want an insane drill-down into the VNX OE block components?  What if you need to know how each IO ends up on the backend?  What if you need to know about the exact utilization of every single component from the IO ports on the storage processors all the way down to the bottom?

That’s what the Analyzer Helper (AH) tool really shines.  I’ve kind of broken the title of the post – as this is actually a tool in a formal beta stage now.  

Read on to understand more, to find out where/how to get it, and to see the results it produced on the data from one of the VNX 7500’s that supported the VMworld HoL (a real-world case).

Analyzer Helper is a formal tool developed by the EMC Global Solutions tools team. These folks develop, maintain and support the tools used by the EMC field – so it tends to be more programmatic than the mashups that a vSpecialist whips up. 

• Analyzer Helper is designed for these use cases:
• VNX arrays running up to VNX OE R31 and older Unified NS and CLARiiON arrays running up to FLARE 30
• Provide advice and healthchecks to customers with VNX, Unified NS, or CLARiiON arrays and to size new VNX solutions.
• Detailed VNX  analysis for:
• Migration/refresh planning
• Thin Pool sizing
• EFD candidates
• Performance analysis and troubleshooting

As I mentioned, Analyzer Helper is in formal Beta for version 2.0.0 now, and is available at the link below for EMC employees.

http://gstools.corp.emc.com/gers/http://gstools.corp.emc.com/gers/http://gstools.corp.emc.com/gers/

Remember – this is a beta tool:

• All beta testing for tools should never be conducted on EMC customer production systems.
• Beta Software should never be used for active engagements.
• Beta Software should only be used for the duration of the beta test.
• Beta Software should be removed once beta testing has been completed.

Answering the obvious question – it is targeted to be available to EMC partners 1 month after the GA target.

You don’t need Unisphere Analyzer installed to use it.  You just need to collect the core NAR data (which can be done on any VNX, NS, or CX array), and then dump it into the tool.  It churns for a bit – and spits out the analysis – which is detailed and awesome.

Here are some screenshots from the analysis done on the VNX 7500 at the VMworld HoL – put it together with this here, and you have an incredibly detailed view of the storage subsystem.   I’m going to pull out some of the charts I found the most interesting, and highlight what they show.   Again, it’s worth calling out HOW EASY THIS IS…  Step 1: Capture NAR data; Step 2: Import into Analyzer Helper; Step 3: enjoy analysis heaven.

The “Heat Map” is modeled on the heat map from the Symm, and it’s it’s pretty awesome in it’s visual summary.   You can see the ports on the top, the storage processors in the middle, the write cache underneath, the back-end busses, and then the back-end disks themselves.     Here you can see that each SP had 4 front-end ports supporting the VNX file blades.  Remember that this was a 3+1 file blade configuration on the VNX 7500 – a neat architectural advantage – to be able to scale up more file blades as needed – though not a single global namespace, very, very handy when you need more NAS performance.

Note here (useful later on too) – this was a workload where it was all about cache – both in the SPs and in the FAST Cache (and they were the only stressed components).

Nothing much here – SPs weren’t breaking too much of a sweat – below 40% utilization (and remember the top-level load as discussed at the original VMworld HoL post here: VMworld 2011 Hands-on-Lab- 10 Billion IOs served.)

Some neat Write Cache stats.

You can see that we’re driving around 600MBps on a sustained basis here (again, not too much – but that’s because the IO size is relatively small – as you’ll see below) – remember, this is the same platform that supported 10GBps (that’s gigabytes, not gigabits per second) in the “World Record high bandwidth vSphere 5” test here.

This is interesting.   The I/O sizes skew to the 8KB and up because that’s the UxFS allocation (VNX OE filesystem) size – but note how in many cases, it is larger?  That’s due to internal coalescing of IOs that’s going on – that’s efficient.

Skipping forward through the report – you then get to a whackload of detail on devices and RGs/pools.   This is a fascinating histogram of IO distribution amongst the array devices – you can see how the IOps are skewed towards the EFD (which was configured as read/write FAST Cache in this config) and the higher performing magnetic media.

For every device – there are a ton of histograms – here’s one – it nicely shows the IOps (and MBps and internal queues) in max/min/95th percentile/75th percentile values.

…and looking at any individual device – you get down to the crazy level of detail – and can see the effect of FAST Cache and cache on reads and writes.

The tool actually creates much more than that – these are just the highlights from my perspective – and like the other tool example I’ve pointed out (the Hyperic/vCenter Operations VNX integration) – it’s really, really easy.

Feedback welcome!  If you’re an EMCer, I would recommend adding this to your toolkit.  If you’re a partner, I would be clamoring to get it.   I’ve told the tools team that we should open it up completely (including customers) – would be curious for feedback.   What if it wasn’t supported, would you still want it?

Comments

[Jake Roczniak]

Partners can already get some of these charts by submitting a NAR file to the Data Profiling Assessments page. I cannot wait for it to be expanded in functionality!

[Domenico Viggiani]

I read the great series of articles about performance analysis by TheStorageSavy blog:
http://storagesavvy.com/2011/03/30/performance-analysis-for-clariion-and-vnx-part-1/
but the procedure to analyze raw data is not immediate.
A tool like this one could be very useful.

[Wilson]

This is amazing Chad. As a customer I would love to be able to run something like this and get the depth of reporting you've shown without spending money on bespoke monitoring solutions. Can I bribe you for a copy!?
If this is offline and using fairly easily extracted NAR then i see no detriment to opening this to customer use, with the caveat that conclusions drawn may require expert analysis.

[Nicholas York]

Good to see my tool lives on without me. ;)

[Nicholas York]

And for the record, I was always in favor of opening it up to everybody, partners, customers, etc. I was even willing to release the source code so people could make their own edits. ;) The only problem was that I wrote this in a completely night and weekend manner, and then went and had a baby so my free time evaporated, so I wouldn't have had time to respond to people that had issues. But after I left, the Tools team stepped up and volunteered to take over, so now the decision making is in their capable hands. ;)

[Chad Sakac]

@Nick - should have known it was you! Great to hear from you - and hope things are going well for you - drop me an email....

It will be open to EMCers and EMC Partners, and am pushing the tools team (which are really stepping up) to make it wide open....

[JasonC]

Wow thought I had found something really useful then, but as just a poor customer can't have access. Gutted!! We have navisphere analyzer, but this helper makes the data into something I could actually present to management, especially useful when looking to purchase more storage. Probably not the place to ask questions but is there a simple tool for joining nar files together to analyse a days data at a time?

[Mark Burgess]

Hi Chad,

Any news on a release date for partners?

Really keen to try it out.

Many thanks
Mark

[Daniel Pfuhl]

Hi Chad,

would be awesome to be able use this tools for creating the heatmap ourselves. We have serious performance problems since months from starting working with FAST II and need to escalate the SR again and again to get a heatmap for finding out the causes. Every time we lose time waiting for the analysis and we have been told that it takes ~8h work to produce the heatmap. So maybe there are circumstances where you can give the tool to customers if they are instructed how to use it.
We are heading for an DSE to get closer in touch with the support team but such a tool would be a powerful self help tool.

[JS Labonte]

This tool would be so useful in my shop... Sadly I'm only a customer. Hopefully EMC will realize there is a need for powerful tools like this one for some customers.

[Jason Oliver]

As a former EMCer, and now back as an EMC customer, I can say that this type of information is invaluable for help in understanding what is going on in an expedited fashion. It also help in creating the graphs for management.

If you are ever looking for a alpha/beta customer tester, PLEASE count me in!!

Jason Oliver

[Andrew Williamson]

Hi

Is this available to the end-customer yet, and how do we get hold of it if not? Not seeing it available anywhere on PowerLink...

Thanks

AW

[Felicien Mathieu]

Guys,

If you are an EMC customer and would like insight on your data please contact your sales rep or TC. They will help you. I am an EMC TC myself and I do run analysis using this tool on a regular basis at a high level(Heat map, worst performing LUNs) this is free of charge, we are here to help!

If you are experiencing performance challanges also ping your TC, if you do not have a TC raise a call to support or contact your account manager. Someone will direct you in the correct direction.

Fe