One of the things that makes me most happy about my job is meetings, planning, and budgeting… NOT.
In all seriousness – those are necessary evils. What DOES make me happy is working with my team, my fellow EMCers and EMC partners, with customers and with engineering. I’m really lucky to get exposed to a lot of ideas, a lot of technologies, and am surrounded by a lot of people who are constantly coming up with cool – and frankly FUN – ideas.
This is the second of a 3 part series on this topic. Each one will “out” a killer tool. Each of them will grow over time. Each of them is free. For now, all of them are not supported (and in one case not even in circulation).
I will be providing a pile of “Santa Chad” awards to people who innovate on top of the two of these ideas. Think iPads, Iomega PX4s, and VNXe 3300s. Remember, it COULD be throwaway work – since they are not supported products per se. But those of you that know me – there’s always methods to my madness :-)
The first one was “Hyperic/vCenter Operations/EMC VNX NFS Uber Mashup” here.
This is the second one: “Beta of Analyzer Helper”:
The majority of EMC customers either use EMC Unified/VNX platforms, or VMAX primary storage platforms (with legions of Avamar, Data Domain, Isilon, Centera, etc, etc). For midrange requirements – EMC’s primary "go to” platform is the VNX. It’s an example of of the “swiss army knife” storage category – does everything under the sun relatively well in all ways.
So – in a VNX, filesystem functions are handled by file blades running VNX Operating Environment (OE) File code, and the block functions are handled by storage processor blades running VNX OE block code.
In the previous post in the series, the tools for providing very rich performance data on the VNX OE File layer of the stack were shown. BTW – if you don’t like it in Hyperic/vCOPS, Clint has shown how you can just throw any data gathered via powershell into a general vCenter plugin here. (powershell is the basis of a lot of really powerful stuff). Remember – start with his seminal post on Powershell here.
But – what if you want an insane drill-down into the VNX OE block components? What if you need to know how each IO ends up on the backend? What if you need to know about the exact utilization of every single component from the IO ports on the storage processors all the way down to the bottom?
That’s what the Analyzer Helper (AH) tool really shines. I’ve kind of broken the title of the post – as this is actually a tool in a formal beta stage now.
Read on to understand more, to find out where/how to get it, and to see the results it produced on the data from one of the VNX 7500’s that supported the VMworld HoL (a real-world case).
Analyzer Helper is a formal tool developed by the EMC Global Solutions tools team. These folks develop, maintain and support the tools used by the EMC field – so it tends to be more programmatic than the mashups that a vSpecialist whips up.
- Analyzer Helper is designed for these use cases:
- VNX arrays running up to VNX OE R31 and older Unified NS and CLARiiON arrays running up to FLARE 30
- Provide advice and healthchecks to customers with VNX, Unified NS, or CLARiiON arrays and to size new VNX solutions.
- Detailed VNX analysis for:
- Migration/refresh planning
- Thin Pool sizing
- EFD candidates
- Performance analysis and troubleshooting
As I mentioned, Analyzer Helper is in formal Beta for version 2.0.0 now, and is available at the link below for EMC employees.
Remember – this is a beta tool:
- All beta testing for tools should never be conducted on EMC customer production systems.
- Beta Software should never be used for active engagements.
- Beta Software should only be used for the duration of the beta test.
- Beta Software should be removed once beta testing has been completed.
Answering the obvious question – it is targeted to be available to EMC partners 1 month after the GA target.
You don’t need Unisphere Analyzer installed to use it. You just need to collect the core NAR data (which can be done on any VNX, NS, or CX array), and then dump it into the tool. It churns for a bit – and spits out the analysis – which is detailed and awesome.
Here are some screenshots from the analysis done on the VNX 7500 at the VMworld HoL – put it together with this here, and you have an incredibly detailed view of the storage subsystem. I’m going to pull out some of the charts I found the most interesting, and highlight what they show. Again, it’s worth calling out HOW EASY THIS IS… Step 1: Capture NAR data; Step 2: Import into Analyzer Helper; Step 3: enjoy analysis heaven.
The “Heat Map” is modeled on the heat map from the Symm, and it’s it’s pretty awesome in it’s visual summary. You can see the ports on the top, the storage processors in the middle, the write cache underneath, the back-end busses, and then the back-end disks themselves. Here you can see that each SP had 4 front-end ports supporting the VNX file blades. Remember that this was a 3+1 file blade configuration on the VNX 7500 – a neat architectural advantage – to be able to scale up more file blades as needed – though not a single global namespace, very, very handy when you need more NAS performance.
Note here (useful later on too) – this was a workload where it was all about cache – both in the SPs and in the FAST Cache (and they were the only stressed components).
Nothing much here – SPs weren’t breaking too much of a sweat – below 40% utilization (and remember the top-level load as discussed at the original VMworld HoL post here: VMworld 2011 Hands-on-Lab- 10 Billion IOs served.)
Some neat Write Cache stats.
You can see that we’re driving around 600MBps on a sustained basis here (again, not too much – but that’s because the IO size is relatively small – as you’ll see below) – remember, this is the same platform that supported 10GBps (that’s gigabytes, not gigabits per second) in the “World Record high bandwidth vSphere 5” test here.
This is interesting. The I/O sizes skew to the 8KB and up because that’s the UxFS allocation (VNX OE filesystem) size – but note how in many cases, it is larger? That’s due to internal coalescing of IOs that’s going on – that’s efficient.
Skipping forward through the report – you then get to a whackload of detail on devices and RGs/pools. This is a fascinating histogram of IO distribution amongst the array devices – you can see how the IOps are skewed towards the EFD (which was configured as read/write FAST Cache in this config) and the higher performing magnetic media.
For every device – there are a ton of histograms – here’s one – it nicely shows the IOps (and MBps and internal queues) in max/min/95th percentile/75th percentile values.
…and looking at any individual device – you get down to the crazy level of detail – and can see the effect of FAST Cache and cache on reads and writes.
The tool actually creates much more than that – these are just the highlights from my perspective – and like the other tool example I’ve pointed out (the Hyperic/vCenter Operations VNX integration) – it’s really, really easy.
Feedback welcome! If you’re an EMCer, I would recommend adding this to your toolkit. If you’re a partner, I would be clamoring to get it. I’ve told the tools team that we should open it up completely (including customers) – would be curious for feedback. What if it wasn’t supported, would you still want it?