We know the preferred hangout of the traditional pirate is a deserted island (preferably one with lots of coconuts and a tame monkey or two). But in Storage IT, the preferred Pirates hangout is definitely the SPC results page.
There is a certain irony in this: the SPC results themselves are produced using strict scientific rules (which I am absolutely certain all the submitting vendors properly adhere to). But what their Marketing Departments do with the results, that’s far less scientific. In fact an eye-patch and cutlass are mandatory costumes….
A case in point: Oracle’s recent release of SPC2 benchmark results for their ZFS Storage 7420 appliance. The results are remarkable, no doubt about it. The competitive comparisons on the Oracle website even more so, comparisons based on two fundamentals: their product has higher throughput and their product has better price performance.
But did they forget to mention anything?
My first concern with SPC2 results is that the prices used are in a word: useless. They start with vendor list prices, which are in most cases astronomical, and then apply discounts, which in my experience do not reflect street prices. We then get to make comparisons, based on these non-real world prices and come up with real world conclusions.
It doesn’t work.
You could of course say that if any vendor feels their SPC prices don’t show them in a suitable light, they are free to lower them, IBM included. But the trick here is to publish late… all you have to do is set your discount rate low enough to leave your competition looking expensive. Or am I being cynical?
The other trick is that SPC prices do NOT include any of the add-on features that are often a key part of the reason you would buy the product. Mirroring, thin provisioning, snapshots, storage virtualization, compression, encryption… the list goes on. The street price of many products goes up significantly when you add in these extra costs. IBM’s XIV is one of the few products which does not have any hidden bonus costs (although I admit I am unclear what extra costs the Oracle solution might generate).
Another omission is the cost of maintenance. Most major sales I have been involved in have included maintenance for 5 years. If we are going to make a genuine TCO calculation, tho cost of those extra years needs to be included.
Then we get to my favourite part of the show: The Unused Storage Ratio. In other words, if you bought the configuration as tested and wanted to replicate the results as tested, how much empty space would you need to leave so that you can short-stroke your disks? The unused storage ratio for the Oracle result is 36.31% versus 2.75% for the XIV results. For an SPC2 result, the Oracle number is quite high.
Finally we have the floor space question and more importantly: the green question. The SPC2 results don’t list power consumption (in kilowatts) or heat output (in BTU). In fact sadly despite the SPC having an SPC2e category to compare energy consumption, only IBM’s XIV has a posted result. What is the hold up? Looking at the Oracle results I can see a hint. They used 384 disks to supply 32 TB of useable capacity. My take on Oracle’s power calculator that I found here suggests their solution would produce 40 K BTU versus 26 K BTU on an XIV Gen3 (that provides 161 TB of useable capacity). Power consumption shows a similar disparity. Unless I am missing something, that’s around 50% more heat for 80% less useable capacity.