Thin Provisioning Buyers Guide

Storage space consumption is always a major bone of contention in all data centers.    It seems 100 TB of new storage can fill up in a blink of an eye and then you have to buy some more.    But what to do?   Lets get below the covers to see what is happening.

When data is written to a volume (I am tempted to say disk, but since most disks are really virtual volumes, that may not actually write to a spinning disk, I will stick with volume) it is written by a file system (or disk space manager of some kind like ASM), to logical block addresses or LBAs (that are 512 bytes in size).    Space in a volume is addressed in LBAs starting at zero and going to the highest address that the disk size allows (so clearly a 5 TB volume has way more LBAs than a 5 GB volume).

From the host servers perspective, if a volume claims it has 5 TB of space available, then the server believes it has the right to write 5 TB.    It is quite common for storage controllers to allow storage administrators to over-allocate space.   Meaning that for a fixed quantity of physical capacity (say 75 TB) you could allocate 150 TB of volumes.    This is over allocation and is only made possible by thin provisioning sometimes combined with other space-saving methods (like compression and deduplication).  Normally over-allocation occurs by creating over-allocated storage pools.

An over-allocated storage pool means the administrator can create virtual volumes whose total volume size (when summed together), exceeds the available storage capacity of that pool.  In other words we can advertise more space than we actually have.   This means the volumes in the pool had better be space efficient in nature.

Now genuine space efficient volume design should follow five principles:

  1. When data gets written to the volume, allocate as little space from the pool as possible to hold that data.  In other words if I write 100KB to a volume, don’t allocate 100GB from the pool to that volume to hold that data.
  2. When zeros get written to the volume, allocate no space from the pool and preferably release the space occupied by those LBAs back to the pool.  In other words, if I write 1 MB of zeros, don’t allocate 1 MB of pool space to hold those zeros.  In fact, have a look at the LBAs I am writing to and if they include address ranges already allocated to the volume from the pool, see if we could de-allocate them from the volume and return that space to the pool.
  3. When allocated space is no longer needed, offer some way to release that space back to the pool (sounds like # 2 but is actually different).  In other words, if I delete a 1 GB file, then that’s really 1 GB of volume space I don’t need anymore.   The file system knows this, but does the underlying disk controller?
  4. If space is running short in the pool, give me plenty of warning so I can do something about it before everything goes wrong.
  5. If data is now being written in a thin fashion, then it is likely the data is not being written sequentially.  When combined with other space-saving technologies this should ideally not create performance issues.

So how well does your storage system do in this regard?    Over the next few posts I will explore these categories in greater depth.   If you have any other characteristics I have missed, happy to add them.

Posted in advice, Uncategorized | Tagged , , , | 1 Comment

Don’t look back in anger

It seems fairly obvious that as you get older you have more and more memories to look back on.  Some of these memories are happy….   some less so.    But seen through the golden haze of nostalgia many things that happened in the past start to become far more glorious than they really were.

I was born and grew up in Perth, so my childhood memories are all from that city.   Recently I found a Facebook page called Lost Perth, clearly run by someone who is close to my age, as the photos being posted really appeal to my sense of nostalgia.   Recently they posted a photo of Perth International Airport as it was back in the 1960s.    I can remember being in this very hall and it was a place of wonders.   When people came from far far away.   It seemed so amazing to my childs mind.

Old Perth Airport

Someone then immediately posted another photo of the same place.   Can you spot the problem?

Packed Airport

Why didn’t my childhood memories contain images of the arrivals hall as an arrivals hell? Maybe I didn’t want to remember it that way?

It’s a bit like your memories of life at former employers.   You can leave a company in anger, blaming terrible management or misguided market dominance plans or crazed short-term thinking…. but it won’t do you any good.   Choose instead to remember the golden years…  and don’t look back in anger.

Posted in Uncategorized | 2 Comments

Your DS3500 needs new firmware to support T10-PI

For those of you who use the IBM DS3500 (a midrange storage controller), you should ensure all your machines are on firmware release or higher since this adds support for T10-PI.   This is because new additional or replacement disk drives may require that support.   Inserting high-level drives into down-level machines can result in a failed drive replacement or unexpected errors.   Ideally you should not be upgrading your machines while there is a failed component, so I recommend you pro-actively upgrade your DS3500s, particularly if you are ordering new drives or additional enclosures.

Note that IBM recommend on this page.  New firmware can be downloaded from here.

If you wondering what on earth T10-PI is, check out this blog here.    If you use AIX there is also a short write up here.  It does not mention DS3500, but I think this is due to the age of the post.

You can tell that T10-PI support is enabled for an array very easily in the upgraded GUI.

T10-PI Screen Cap

Posted in Uncategorized | 1 Comment

Innovate, emulate or evaporate.


The IT Industry is changing rapidly.   New disruptive technologies are changing the whole playing field and vendors who just talk about backup are going the way of the dinosaur.   Actifio saw this more than four years ago and began a new era of Copy Data Management. Finally the other guys are starting to realize the ground has shifted below their feet and have begun talking about doing exactly the same thing (without actually changing anything that they currently do).

But don’t just listen to me, have a read of Chris Mellors analysis and watch the EMC video.   Then talk to Actifio and get today what EMC cannot deliver tomorrow.

Posted in Actifio | Tagged , , , , | Leave a comment

Backblaze Blog » How long do disk drives last?

This is fascinating stuff that pretty well exactly matches my experience with almost any IT product.  Congrats to Backblaze for collecting and sharing this information.

Backblaze Blog » How long do disk drives last?.

Posted in advice | Leave a comment

Your SanDisk USB stick is no longer removable – and it’s Microsofts fault!

I came across something curious last week that failed the question:
Can I find a quick answer for this in Google?

I had purchased a new SanDisk USB stick to boot Linux.   The script I was using to configure it had a check to make sure the disk target (e.g./dev/sdb)  I was installing Linux onto was removable media.   The script kept complaining that my new USB stick was not! I found an older SanDisk USB stick and with that one, my script ran without error.

Had I bought a bad USB stick?   It formatted without issue and I could write to it and read from it without any errors.

I then compared the dmesg output for the two sticks:

Newer USB stick:

[833126.592645] scsi 10:0:0:0: Direct-Access SanDisk Cruzer Switch 1.26 PQ: 0 ANSI: 6
[833126.592794] sd 10:0:0:0: Attached scsi generic sg2 type 0
[833126.593499] usb-storage: device scan complete
[833126.596965] sd 10:0:0:0: [sdb] 15633408 512-byte logical blocks: (8.00 GB/7.45 GiB)
[833126.601274] sd 10:0:0:0: [sdb] Write Protect is off
[833126.601276] sd 10:0:0:0: [sdb] Mode Sense: 43 00 00 00
[833126.601278] sd 10:0:0:0: [sdb] Assuming drive cache: write through
[833126.609031] sd 10:0:0:0: [sdb] Assuming drive cache: write through
[833126.609042] sdb: sdb1
[833126.634951] sd 10:0:0:0: [sdb] Assuming drive cache: write through
[833126.634954] sd 10:0:0:0: [sdb] Attached SCSI disk

Older USB stick:

[151567.256075] scsi 2:0:0:0: Direct-Access SanDisk Cruzer Switch 1.20 PQ: 0 ANSI: 5
[151567.256497] sd 2:0:0:0: Attached scsi generic sg2 type 0
[151567.257279] usb-storage: device scan complete
[151567.261777] sd 2:0:0:0: [sdb] 7821312 512-byte logical blocks: (4.00 GB/3.72 GiB)
[151567.265883] sd 2:0:0:0: [sdb] Write Protect is off
[151567.265888] sd 2:0:0:0: [sdb] Mode Sense: 43 00 00 00
[151567.265897] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[151567.278033] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[151567.278041] sdb: sdb1
[151567.287932] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[151567.287935] sd 2:0:0:0: [sdb] Attached SCSI removable disk

I noticed the difference between the newer SanDisk USB stick and the older USB stick was the killer line at the end.  The old stick was identified as a ‘removable disk’ while the new stick was identified as a ‘disk’.

The other difference was simpler to spot.  The ‘removable’ variable was ’1′ on the old stick and ’0′ on the new stick.

 # cat /sys/block/sdb/removable

I then found this on the SanDisk web site which explained what I was seeing and confirmed that there was nothing wrong with this new USB stick:

Flash Drive shows as Hard Disk Drive (Fixed Disk) in Windows Explorer

Where it told me:

NOTE: SanDisk is beginning production of flash drives configured as fixed disk in 2012 to meet new requirements for Windows 8 Certification.

Historically, flash drives have been configured as removable disks and Windows Explorer displays them as ‘Removable Media’. Windows 8 Certification requires flash drive manufacturers to configure flash drives as fixed disks. Flash drives configured as fixed disk will show up in Windows Explorer as ‘Hard Disk Drives’. Flash drives configured as fixed disks still function the same as those configured as removable disks.

NOTE: SanDisk does NOT support configuring flash drives as bootable device for running an OS.

This change (regardless of what SanDisk says) does not stop you using these new sticks as USB boot media.  But it does mean my Linux install script can no longer programmatically identify SanDisk USB sticks as removable media… and its all because of Microsoft.

I can only imagine Steve Ballmers reaction on learning this:

Posted in advice | Tagged , , , , , , | 15 Comments

Different story but the plot remains the same

I just finished the latest Jack Reacher novel, “Never Go Back”, by Lee Child, and it was everything I expected.   If you have ever read any of the Jack Reacher novels you will know what I mean.   Every Jack Reacher novel contains pretty well the same elements: there are baddies (normally corrupt politicians, businessmen or military officers); their henchmen will foolishly tangle with Jack; some of these bad guys will end up with broken limbs, some will end up dead (some by Jack’s hand);  Jack will show his physical and military prowess; Jack will normally sleep with someone perfect…   and..  well…  it’s perfect.   But at the end, despite the romance, once everything is wrapped up nicely,  Jack disappears into the sunset, on to another town, another corrupt baddie, another perfect love interest.

Its a great formula and it makes for great reading.   Despite knowing that each novel will read much like the previous one, I keep reading them.

EMC announcements are a bit like that, there are goodies and baddies, but always the same story with no surprises.   The story always ends the same way.

Take their recent Backup is Broken pitch.   It’s worth taking the time to watch it (it is four and half minutes).      It sounds great on the surface, but you don’t have to dig too far to start seeing that nothing has changed.   Its funny to watch a pitch that starts off blaming the backup team and their current generation of tools, kindly supplied by EMC, for a whole raft of business shortcomings.   The solution is to then buy warmed over versions of the same products.   Sadly what doesn’t change is that EMC cannot fix the issue that the market has too many point solutions (many of them EMC products); that their backup products do nothing to help slow the growth in test and dev environments and that their product line is all designed from the perspective of EMC and the companies they have acquired, not from the perspective of the client.   It really highlights the innovators dilemma, how can EMC innovate when they also have to protect the revenue base from their existing platforms?

If you are still keen to watch videos, have a listen to what this client has to say about Actifio and how it compares with EMC for backup and recovery.     It makes for an interesting comparison, a story with a much better ending.

Posted in Actifio | Tagged , | Leave a comment