Accessing the Instrumentation

Here are some rather bad photos of my 1972 Holden HQ Kingswood Premier, one of my first ever cars (and one that I sadly no longer own):

IMG_0001 IMG_0009

This was the V8 Four Litre model (actually 253 cubic inches, often jokingly described as having all the power of a 6 cylinder with the fuel economy of an 8).   The engine bay was so huge and empty I could open the bonnet and sit on the side of the car with my feet comfortably inside the bay while I changed spark plugs or cleaned the points.

The Kingswood was not what you would call an instrumented vehicle.   The dashboard had a speedo, a fuel gauge and three lights:   Temperature, Oil and Charging.   I dubbed these three lights the idiot lights: as once they came on,  you were the idiot.  (sorry, no picture; this was the 1980s).

Modern storage infrastructure by comparison is slightly more instrumented.   A vast array of metrics are tracked and these can be used to perform all sorts of analysis.   Analysis like:

  • Are my hosts getting good response times?
  • Are specific disks or arrays being over worked?
  • Are my fibre ports being used in a balanced fashion?

So can you do this with the Storwize products?    Of course!   I documented the built-in tools here (where I talked about the Performance GUI):

https://aussiestorageblog.wordpress.com/2011/12/06/svc-and-storwize-v7000-release-6-3-performance-monitor-panel/

And here (where I talked about the performance CLI):

https://aussiestorageblog.wordpress.com/2012/06/22/storwize-v7000-and-svc-performance-monitoring/

But these tools have only limited usefulness.   They are not granular, in that you cannot look at specific hosts or specific arrays or specific FC ports (meaning the three analysis ideas I suggested above are not even possible).   So how can we do this analysis?

The good news is that Storwize products do track all the metrics needed to do very granular analysis and these are freely accessible.   These files are documented by IBM, here is a fairly old page that documents some of them:

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1003597

But how to turn these into something useful?   There used to be a tool called svcmon but this tool appears to have been killed as per this rather sad blog post:

https://www.ibm.com/developerworks/community/blogs/svcmon?lang=en

There is another IBM Community developed tool called qperf which you can access using the link below:

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105947

With a graphing tool here:

http://www-01.ibm.com/support/docview.wss?uid=tss1td106040

And another tool here:  http://www.stor2rrd.com/

And yet another one here!   https://code.google.com/p/svc-perf/

The challenge for many of these tools is that they require manual setup, usually have a limited database engine and analysis is not always easy or simple.

You can of course use IBM’s TPC:

http://www-03.ibm.com/software/products/en/tivostorprodcent

You could also consider Intellimagic.   Although I have not looked too deeply at this one, these guys wrote IBM’s Disk Magic tool, so they certainly understand storage performance

https://www.intellimagic.com/solutions/san-storage-and-fabric/intellimagic-vision

The challenge for all Storage Admins is that they are not always experts at diagnosing performance issues.    Getting some genuine examples of the thinking process and the flow of getting from problem to solution, is vital.   This makes  BVQ  another good choice.

To see an example of how instrumented data presented in a graphical format can be used to generate a useful problem analysis, check out this blog post here:

https://bvqwiki.sva.de/pages/viewpage.action?pageId=28180509

and another one here:

https://bvqwiki.sva.de/display/BVQ/Performance+bottleneck+analysis+on+IBM+SVC+and+IBM+Storwize+V7000

I really like these posts for two reasons:

  1. They clearly shows just how instrumented the product is
  2. They clearly show how using this data in a graphical format can lead to good and quick root cause analysis.

Also have a look at some of these videos:

https://bvqwiki.sva.de/pages/viewpage.action?pageId=3834329

So how are you instrumenting your Storwize?
What do you find the easiest tool to use?

 

 

 

Advertisements

About Anthony Vandewerdt

I am an IT Professional who lives and works in Melbourne Australia. This blog is totally my own work. It does not represent the views of any corporation. Constructive and useful comments are very very welcome.
This entry was posted in Uncategorized. Bookmark the permalink.

5 Responses to Accessing the Instrumentation

  1. Lee says:

    Spectrum Control Storage Insights is also well worth a look – especially as there is a 1 month free of charge, no obligation, trial ( http://bit.ly/1RRAeqB ).

    I’ve had very positive feedback from the partners that have looked at that – it’s not just doing performance, it’s also doing capacity and space saving recommendations.

  2. James says:

    Galileo is a pretty complete offering for SVC/Storwize.

    http://galileosuite.com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s