A couple of internal EMC+VMware threads worth sharing…

As we get down to the late hours of Q1, there’s the usual hum and activity.   It’s an energizing time!  Met with some GREAT customers today.

Beyond that – our internal presales transformation continues (we’re holding off broadcasting updates until the beginning of Q2).  A small (but important) part of that change is cultural.  

One thing that is fascinating to me is some of the absolutely killer internal dialog around technical issues.   Nothing EMC confidential in them.

These are fascinating their own right, but also make me ask the meta-question in my mind… read on, and see if you can figure out where I’m going with this…

The first was around a customer experience performance issues around storage during a benchmarking exercise:

It starts with a question from a customer:

“My question to EMC, with perhaps some consultation from a VMware expert familiar with using ESXi with Symmetrix storage, is are these dd tests a valid and reliable way to measure write throughput to a LUN on a Symm?  If not, what would be?”

The customer was using DD to do a test from the service console.

The internal vSpecialist thread ended with this answer (right on the money) from the always awesome Matt Cowger:

“It's not silly, but it's not indicative of the performance that a real, streaming write application would see.  Why not?  Because 'dd' only leaves one outstanding IO at a time, which no real write intensive application would do (because it leads to suboptimal performance).

Additionally, ESXi explicitly throttles the performance of the TSM console to prevent a console app from killing VM performance. 

The right way to test this is to spin up a VM with a disk on that LUN (vmfs or ram), then use a real IObenchmark tool to run the test.   On windows that might be IOmeter (make sure to run it with 32 outstanding IOs) or iometer or vdbench on unix. 

Let me know if you have more specific questions.  I see this all the time(people trying to use dd to do benchmarks) and its a simple way, but very inaccurate way to do it." 

A great answer – and like Matt, I see this often (likewise looking at CPU/Mem perf tools like Prime95 and Memtest which are fine, but not at all like a real-world workload – benchmarking is a lot harder than it looks at the surface).

The next question was around the changes in vSphere 5.0 update 1 and unmap from an awesome dude in Germany.

“…could someone from this list provide a current status of VAAI UNMAP usage with EMC arrays?

I’m aware of VMware KB article 2007427, but it doesn’t contain any schedule when we could expect a final fix.

And I got a call from a colleague who informs me that after multiple svMotion operations on a VNX array the thin LUN’s still allocate storage from the pool even when no VM is stored on the datastore.”

The answer came from Jeff Purcell (the excellent author behind the VNX vSphere Techbook – which you can download here).

“Per the release notes, the unmap functionality in U1 is not an active process.

It must be initiated via the vmkfstools command using a new option (-y) along with a value for the percentage of the volume you want to reclaim.

See attached slides for a basic example using a thin LUN from a VNX storage pool.

In the example a VM has been provisioned with a 120GB eager zeroed thick VMDK. When the VMDK is removed the vmkfstools command is run to reclaim that space within the Thin LUN and storage pool.

Your storage VMotion example would have resulted in space reclaimed if the datastore used Thin Provisioned Pool LUNs, and the user ran the reclaim command within the datastore where the virtual disks had been vacated.

This functionality is active in VNX OE 31 and later. Note: there may be a performance impact associated with the reclaim processes so use ask your customers to use discretion when running these tasks.”

BTW – the slides he attached showed screenshots of him doing the steps (BTW his example returned 120GB to the pool in ~85s).

… So – those are two interesting threads.   What’s the “meta question”?

Internal to EMC we continue to struggle over the models of how we share – how much is internal, and how much is public.   This is not as easy as it seems.   Consider the following:

• As part of a company, there is risk of leaking EMC confidential information when we are public.   Also, often there is a lot of customer information in there too.   That first example actually contained some of the customer’s internal config info that I cut out.  If I didn’t – it would have been quite bad form at a minimum and have potential consequences that could be far more material.
• As humans, there’s the fear of being OUT THERE.  What if you are wrong?   There’s an ego aspect to it (being wrong and being publicly corrected), but there is also the aspect of "what happens if I say something I’m not SURE about and a customer gets hurt?   That’s why we have processes and structured ways to communicate in support.
• Inevitably, as the leader in the markets we serve, anything we say/do gets picked up by our competition and the media folks – which sometimes spins up out of control.
• There’s a delicate line – sometimes, in disclosing an issue which is low impact and rare, we can inadvertently cause partners/customers to over rotate and in doing so, cause themselves more problems than the “issue” ever caused – increasing harm rather than mitigating it.

There have been a few times where I’ve been bit by all three of these – personally.

Within the walls of EMC, we use a lot of email DLs, and to a growing (but still lesser) degree – our Jive-based internal social media community called ONE EMC, and we are in baby steps of use of Salesforce Chatter.

My personal view is that the epic #FAIL of these mechanisms is their “internal-ness”.   As our business becomes ever-more partner-centric we MUST share outside the walls of the company like crazy.   Further – our customers want and need this information, and frankly often add as much to the dialog as we do.

That’s why I keep pushing for the “if I can’t get to it via the obvious Google/Bing/whatever query on the interweb, it DOESN’T EXIST as far as I’m concerned” model inside EMC for this type of information (obviously the line is drawn at EMC confidential information).    This is a little bit of me “overmessaging”… if you want to move a huge ship by 5 degrees, you need to turn the wheel sharply 90 degrees and wait a while – but I also believe it.

I push like mad for the “Everything ____ at EMC” model (try it on Google, and put in “Oracle” or “SAP” or “Microsoft” or “VMware” – and see what you find).   I push like mad to make our software downloadable on public places like EMC.com rather than internal models.   I push like mad to make our virtual appliances available broadly.

I’m always interested in how we, as humans, mentally calculate and balance risk.  I think we tend to over-weight potential negative consequences of action, and tend to under-weight the opportunity cost and risks in non-action.  I also think that so long as we are guided by good ethics and morals, we will tend to do the right thing.   I also believe that failure and “being wrong” is not as bad as people often think it might be (I actually call this out in the EMC Presales Manifesto – check it out here – “we always do our best to provide the right answers and solutions.   When we are inevitably wrong, we admit it quickly, openly, and learn from mistakes we and others make.”) – and that errors when acknowledged, internalized, and adapted to are incredibly powerful in their own positive right.

But then again, my wife calls me an “idealist” (I prefer “optimist” ;-)

The debate rages (and to be clear, I’m not saying I’m definitively right here – I can see the other side too).    Frankly, dear readers – I would LOVE your input.   Whether you’re an EMCer, an EMC Partner, and EMC Customer, a non-EMC customer, a competitors, in the media – whatever – I would like your view.  

please comment!

Comments

[Habibalby]

Excellent post, I want the right answer too for configuring MPIO on EMC AX4-5i with vSphere or normal VM with iSCSI.

[Brandon Riley]

>>"our customers want and need this information, and frankly often add as much to the dialog as we do."

I cannot emphasize enough how right you are on this one. As a customer, I am so frustrated by the incoherent strategy when it comes to information or recommendations. The documentation is plentiful, but often quite disorganized. As an example, I had to look at 5 different documents just yesterday to try and piece together how the SRA utilities for VSI works, and I STILL never found anything that said this HAS to be installed directly on the SRM server to work.

Often times, I'm tweeting in desperation, and the links sent in response are inaccessible to me on Powerlink. This really makes life as a customer difficult.

One last thing. My EMC sales experience, as I detailed on my blog, was second to none. Not one other vendor even came within striking distance in that regard. The post-purchase experience has been nightmarish. A huge reason for my trauma has been the disheveled, inconsistent recommendations coming from implementation engineers, EMC curriculum, and even support personnel. I won't go into detail here, but I will say that if it weren't for my awesome salesperson being willing to put my questions out to your internal social network, my frustration level would be 5x higher.

So yes. Customers want this information. We want consistent, credible information from people who know what they're talking about. Thanks for the post. I've been needing to vent this to someone who will listen for months.

[Jim Millard]

My approach to these sort of internal conversations is to get the customer problem solved, then blog about the situation in the generic/abstract voice. In a team effort, this requires someone to take the initiative to get it done for each circumstance. I'm sure that Chad has no shortage of folks who can "tidy up" an email thread into an informative post that ultimately gets scraped--and is therefore searchable--by Google and the other search platforms. The hard part is making sure it gets done in a timely fashion: the sooner it surfaces, the sooner it can be searchable.

[Terry Jones]

Chad - Thought provoking as ever. Do you ever ask easy questions?

In my mind this is a case of 'be careful what you wish for'

Organisations I have worked for have struggled with this since the advent of the social web. As it is so easy to publish and share information the expectation of organisations such as EMC and VMware is that we should do so without limitations. The simple thing would seemingly be to do exactly that..

However,

My experience of working on the inside is that the pool of knowledge is huge and for any given topic or question you can always find a number of divergent opinions/answers, the trick is then to know how to assess the provenance of the information i.e If its a question on SRM then Chad's answer is the one I would go with :-) . With open access to data of this type the external party has to make the call….

Of course the danger lies not with the individual who is working diligently and in a vast majority of cases will take the information in the spirit in which it was intended, but with their employer who suffers an outage as a result of a well-intentioned mis-interpretation and starts looking for someone to blame.

At a past employer we gave some key distributor personnel access to our internal KB as this was seen as a way of making them more self sufficient and responsive. They realised that by becoming the arbiter of what was correct they were exposing themselves to risk and there were few complaints when the experiment ended.

As you infer, the decision the supplier has to take is if the goodwill (and $$$) such openness drives is worth the pain it can bring along with it. Unfortunately I suspect that question that will keep the lawyers and accountants busy for a good while!

(VMware employee, but the above is a general point….)

[Michael Ryom]

Put it all you there(as long as it makes sense and legal and all that)! Was looking for an EMC storage appliance the other day, know you made one, but powerlink and google search didn't do it for me... So when with 'free nas'... A small example of how hard it can be and how easy it is to find some one else who will provide it.

For EMC customers a good start would be to clean up powerlink. Haven't really used 'everything xxx @emc' but its a good start and a very good way of getting the knowhow out there.

Videos and guides are an easy way of getting knowledge(Eric Sloof really is a big consumer of this approach) our there and an easy way to consume it. though it might to time consuming.

[robojl]

Chad,
As a former customer and now an EMC'er I couldn't agree more - transparency is important. The only caveat is that different views of the "answer" are needed for different customer audiences. Giving an engineering response to a C Level individual isn't appropriate and will probably cause more confusion, by the same token giving a management response to an engineer won't answer the question. I have seen us shoot ourselves in the foot several times since I came on board a year ago by not ensuring communication is a) Clear b) Concise c) Consistent and d) in Context. A plethora of Powerpoint slides is usually not the answer the customer is looking for ;-)

[Keith Paul]

Awesome, Chad. I agree with you in theory about the limits with EMC ONE. You hint at the "Everywhere" communities on the EMC Community Network... We recently made all of the ECN indexed by Google, including support forums, but excluding sensitive IP or customer info. I bet there's a middle ground there, where we can totally have deep conversations around EMC tech, which over time becomes a type of KB for folks who haven't asked those questions yet.

Jim, your blog approach is spot-on too. In that model you're generating a personal brand of being helpful. We're lucky to have many folks doing this. But by supporting the ECN as a go-to source for this type of info will help transform EMC into a truly human and social brand.

We are indeed turning a ship, and I see it every day!

[Chuck Hollis]

Good discussion on variable risk when communicating technical information.

As a transmitter of information, I think the right mindset is "here is what we believe to be true, here is why we believe it to be true, if you need to use this information to make an important decision, please engage with us directly so we can provide more context".

But there is also a responsibility for the consumers of information as well.

Anyone consuming and using the information needs to understand that this was the best information available at a given time, things may have changed, your situation may be different, etc. -- and if an important decision is being made, make the effort to contact a subject matter expert for the lastest greatest.

When it comes to bad information events, there are two parties involved: the provider *and* the consumer. And, as long as both parties recognize their responsibilities, things are good.

At one time, we all did blogging disclosure statements. Maybe we need "technical information disclosure" statements?

-- Chuck

-- Chuck

[Brennels]

Great dialogue, Another resource that might be beneficial where many of these types of conversations occur are on the EMC Community Network https://community.emc.com/community/connect/everything_vmware

I know there are other knowledge experts, vSpecialists and peers that regularly engage in similar topics...

Hope this helps

[Travis F]

Honesty and transparency are great policies unless you have something to hide.

It's important to make the data and dialogue public unless it reveals trade secrets. I recognize that you'll have negative experiences with public disclosure, but that means you need to do it well and develop appropriate guidelines and formats as you learn from experience. It doesn't mean it can't or shouldn't be done. In each of the examples you provided, it seems like things could done to avoid problems in the vast majority of circumstances. The pro's of the majority overshadow the occasional problem.

Be the leader, take a bold step, show us what it looks like to be open by default.

Customer considering EMC

[JayQ]

We need to share information. there is no question about that. One critical issue to me is not whether to share or not. It is the many different places we put this information and where to look for it consistently. Blogs, Community Boards, Internal websites, twitter, Chatter, etc etc.
Just navigating through can be a frustrating experience in itself. This is why our partners, customers and employees require the Google model you mention above.

In addition, I personally see no delineation from a partner friendly post vs. an internal post. We are either on the same team or not. In my view, Our Partners have the right to educate our customers with the same information I have and not something less.

I also think a big part of the answer revolves around a culture change. We all need to take a different view than we may be used to, we should act as gatekeepers for disseminating information and be pro active about it. This includes not only the information itself but also who the SME might be. The SME should also hold almost nothing close and constantly be a broadcaster of information.

[Craig McQuilkin]

Chadster, old nickname from an old friend. You have been pushing the limits of openness since you walked through the doors into Commercial so many years back. It was a breath of fresh air then, and continues to be one. It's what people need and want. Keep on pushing.

[iwan 'e1' Rahabok]

Hi Chad, always enjoy the contrarian view, as I believe different isn't always better, but better is always different. I'm bookmarking this post and will share.
Sun went through a lot of thought to open source Solaris a few years back. Lots of fear, lots of unknown. Years after it's done, the good is far greater than the bad.