Sure Tape ain’t dead, but what about that microfilm?

The late 1950s and 1960s are a fascinating time in IBMs history. In those early days, innovations in business were achieved with physical machines… yes… actual weird and unique machines that did things mechanically. Huge gains were achieved with both mechanical innovations and just as importantly, through materials engineering, (which was hugely important) and new materials were constantly being considered.

The thing that really stands out looking back is just how bat shit crazy some of these machines and materials were. It is quite clear that regardless of how conservative you think IBM is or was, no idea was off the table and some of those ideas literally paid for the tables.

Now while its well understood that magnetic tape meant the end of punch cards, while the first magnetic tape drives shipped by IBM came out in 1952, by April 1955, IBM was making 72.5 million punched cards per day and in 1965, the number of cards being produced by IBM was still in the tens of billions a year.

Clearly the data explosion of our age was happening then, just at a different scale and onto different media. There was a constant challenge – how to store all of this punched card data?

One totally off the wall idea was to turn punched cards into 35mm microfilm, and so the IBM Vestal lab in New York State developed a set of machines to do exactly that. These were manufactured by IBM in Dayton New Jersey and announced on December 4, 1964.

This was a Micro-Processing System that used microfilm as a space saving method of storing documents that allowed you to still process and retrieve that data on punched cards. It consisted of several different machines:

• At the front centre (to the right of the standing lady) is the Micro-Copier/Reproducer that copied punched cards to microfilm.
• To the left of the standing woman (on the small desk) is the copier that turned microfilm into punched cards.
• At extreme left is a punch card interpreter, that could read punched cards and print information on them.
• At centre rear is a punched card sorter.
• Next to the sorter (on the small desk) is IBM’s copier for thermal film, which was developed by heat and light rather than chemicals.
• The key punch operator at extreme right scans the Micro-Viewer with roll film attachment to obtain descriptive data to be punched onto a master aperture card.

There was some immediate challenges. The product used ammonia to develop the microfilm. This ammonia had to be refilled by the client into a bottle that looked like this:

If they over-pressurised the gas storage tank, it could explode and at at least one client it literally blew the doors off the unit. Michael Caine would have been very impressed, but I am unsure what the client thought. Queue the classic scene from the Italian job:

Ammonia leak issues were common and IBM engineers used sulphur sticks (or their noses) to try and find them. This sounds impressive till you realise that the way a sulphur stick works is you light the stick and if it starts to produce a white smoke, you know there is ammonia nearby. I love the witch doctor image of an IBM Customer Engineer waving a smouldering stick around in the hope of producing white smoke. Again… Lord knows what the clients thought.

Sulphur sticks!

The bigger issue was simple – this was a way of putting punched cards onto a different medium. It was solving the wrong problem. Smarter solutions were needed.

Just before Labour day in 1969 IBM informed their customers they were getting out of microfilm business, although in classic IBM Style, some of these machines remained in service well into the 1970s.

Amusingly those ammonia bottles are also part of a fine tradition of IBM CE recycling. The one pictured was used to chill beer kegs and is now used as part of a nail gun. So not everything was lost.

Now if you are thinking, this was a little crazy, well just you wait! You aint seen nothing yet! In my next post we will go way down the rabbit hole with an IBM Product that makes exploding microfilm copiers look tame!

Sure Mainframe ain’t dead, but what about that toilet water?

That old ‘Mainframe is dead’ chestnut has really rolled around the field for a while, but despite the best hyperbole of many IT pundits, IBM Mainframes continue to thrive and survive, even if they are no longer the biggest source of IT spending in a typical enterprise. When I started at IBM the water cooled mainframes were a huge revenue source and the people who supported them were sort of living gods, especially the more senior ones.

I myself was more of a cable monkey at that time spending more time under the floor than above it, in those glory days of the late 80s and very early 90s, before HDS really started eating IBMs Mainframe Lunch and EMC started to eat their DASD (Disk) Desert.

I did however become a ‘real’ mainframe guy when the air cooled 9672s and 2064s blew HDS out of the compatible market and brought IBM marching back into quite a few of our old accounts with far smaller machines that one or two people could install quite comfortably.

Yet despite the (possibly unexpected) survival of the mainframe, what has truly gone is the almost completely bizarre world of mainframe I/O (Input/Output) devices and the vast number of service representatives who were needed to support them. So in the great tradition of IT War Stories, here are five things you will never get to do in a data centre:

You will never blow mainframe water into a toilet

Yes you read this right… not only will you never get to blow mainframe water into a toilet, I may never get to write that heading again.

To explain: A water cooled mainframe was not shipped with water in it. Apart from the obvious fact that shipping something with water in it is a challenge, these things weighed enough already without 400 litres of water sloshing around inside. So one of the many tasks done as part of installing such a beast was to “just add water”. This involved emptying 100s of litres of distilled water bottles, usually way bigger than this one:

This water would be pumped to chillers (normally on the roof) to take away the heat from the many many heat producing TCMs (Thermal Conductor Modules) that made the magic happen. Here is a terrible screen capture showing a TCM and some of the vast amount of plumbing hoses. What makes this photo ‘real’ is the fact that the gent is wearing an ESD (electrostatic discharge) wrist band.

Now when a water cooled mainframe was uninstalled, it could not be shipped with the water in it, which meant blowing the water out of the many pipes and hoses using a big old bottle of nitrogen, usually way bigger than this one:

But what to blow the water into?

Why the toilets of course!

Normally these were the grimy under-cleaned toilets just outside the computer hall.

We would hook up the hoses end to end and then with the hose aimed firmly into the bowl, until you eventually got down to just gas with spits of moisture. The disappointing thing is that while there are many marketing photos showing impossible scenes like the one below, none show anyone blowing mainframe water into a toilet. And no one had mobile phone cameras in those days (mainly because your current mobile phone has more of everything that is shown this photo, except the printers):

Now before you proudly inform me that the current IBM z15 has a water cooled option, just like in the ‘good old days’, the amount of water being used is comparatively tiny and the water is simply drained into a jug using a special tool like this. And seriously, this ‘tool’ looks like my Uncle Ted designed it.

You will never drag 100s and 100s of feet of dirty bus and tag cables around and under the floor.

A good computer room floor would give you two or three feet of clearance, but it was not unusual to get very shallow floors. The parallel bus and tag cables used from the 1960s to 1990s were seriously heavy and accumulated in great numbers under the floor. Sometimes when cables needed to be re-routed it was easier to drop the old cables into the floor (perhaps after beheading them) and just run new ones. Such sins could accumulate until the under-floor section became literally clogged with these cables, like weird veins of some great dead beast. In addition these cables seemed to accumulate filth. After twelve hours of dragging them through the floor you often found your hands and clothes filthy and blackened.

After removing around 2000kgs of cables from one data centre, we learned from our recycling agent that the wires themselves were aluminium and hard to recycle, but the connector pins (in those big fat connectors that just loved to snag on pretty well everything), were gold plated and if you got enough of them, you could actually make some real money.

I stole this image from Wikipedia, and the cable is clearly slightly grimy, an aspect I find oddly pleasing:

A dirty bus/tag cable

You will never get to ask for a 36 hour outage to install a new system

The thing is that mainframes were huge and hard to install and computer rooms were not always designed to let you install a new one alongside the old one. Even if you could, this would require a mountain of cabling to make it work. So for many customers, the installation schedule would look like this.

1. Sometime on Saturday morning, once the Friday night batch and backups were finished, team one would begin. They would power off and de-install the old mainframe and remove it from the floor. This could take many hours and many people.
2. Once this was done, team 2 would come in and re-cable the under floor region, especially if the layout had changed. This could also take many hours.
3. Once this was done, team 3 would assemble the new mainframe and commission it. This could (you guessed it), also take 12-15 hours.
4. Things would normally close out some times as late as Sunday evening with the debug team (team 4) fixing any issues so the client could be up and running by 8am on Monday morning.

This extended outage was quite normal, even for banking systems.

Tell that to dev-ops kids now-a-days and they won’t believe you.

You will never get to roll enormously heavy motor generators across the computer room floor

An IBM 3090 ran on 400 hertz power, meaning it took your 50/60 hertz power and then converted it to 400 hertz using a motor generator called an IBM 3089, which weighed 1075 Kg. Meanwhile the IBM 3097 that distributed power and cooling could weigh up to 1309 kg. The I/O frames meanwhile were a touch top heavy, so they had wings you would pivot out when rolling them across the floor to remove the risk of them toppling over.

You may never install a computer that comes with a desk.

Thats right, every IBM 3090 came with its own desk, officially known as the console table. It’s not unusual to visit a computer room that has not seen a water cooled mainframe for over 20 years, and yet those humble desks are still doing service, possibly because they weigh nearly 90 kilos and no one is strong enough to remove them (although the green screens that once sat on them are hopefully long gone).

A 3090 console table

Anyone still got an IBM Mainframe desk or been dragging some bus and tag cables? Let me know. Maybe take a photo with your smart phone.

Why the IBM SSIC is the best place to get your supported hardware information

Its been a while since I last blogged, but here is something that might interest you.  To spice things up I even recorded a quick video blog of the same information you will find written here.  So skip the video and keep reading, or watch the video and skip the reading, the choice is yours!

For many years I have used the ‘Supported Hardware List’ websites from IBM to qualify SVC support.    If you want to know if an Infinidat is supported behind IBM SVC, which version and what code level,  it’s all there.

So traditionally I would go to here.  From there you get a great list of code levels, choose your code level and then look up your product:

However I have always had some tiny misgivings about these sites.  After all, they obey no law of sorting I have ever seen.   Alphabetical order any one?   It’s like the Web Admin is a worshiper of Cthulhu and has managed to translate non-euclidian geometry into a list of vendor names.   Or maybe the site is just TOO HARD TO MAINTAIN.

Take a look and suggest a logic to this list of vendors (please I beg you):

But there is a bigger problem:  Theses sites are just slightly out of date.

Lets use that example I first raised,  if look here I find Infinidat version 2.0 is supported with SVC 7.8:

But if I then go to the SSIC here.   I get told version 3.0.x is also supported:

This was not a one-off, I found multiple products where the SSIC seemed to reflect newer information than the Supported  Hardware websites.

Moral of the story?   Always use the SSIC to confirm support, not the Supported Hardware pages.

Actifio at Tech Field Day 11

In this blog post I want to inform you about Actifio’s presentation to Tech Field Day 11.

Tech Field Days are one of my favourite technical information sources.    They involve a group of prominent bloggers and industry personalities being given a briefing and demonstration (normally somewhere between 2-4 hours) by an IT company about their products and viewpoint.   It is a chance for both IT entrants and established IT Companies to tell their story, explain the why, have their ideas challenged by some smart people and get some relatively free publicity, while the bloggers get the chance to gather material to write blogs, learn about our rapidly challenging world and sometimes show how clever and insightful they are at the same time.   It is a genuine win-win for everybody.

Actifio were last at Tech Field Day 4 in 2010, which explains why competitive information about Actifio is often so laughably wrong.  I think other Vendors watch these (in IT terms), ancient videos and presume nothing has changed since!    The good news for Actifios competitors and prospective and existing customers is they can now update their knowledge of Actifio by watching Actifio present at Tech Field 11 in 2016

The really nice thing is that the presentations have been split into five easily consumed videos, each about 20 minutes long. So please drop by the Tech Field Day page, take a look at the presented subjects and learn about Copy Data Management and how Actifio’s products bring a new and unique way for our customers to move to the hybrid cloud, dramatically improve their agility and modernise their business resiliency.

To make it easy, I have reposted all the Actifio video links below, but you can also get to them from here where you can also check out the other vendors who presented.

Actifio Welcome with Ash Ashutosh

Watch on Vimeo

Actifio CEO and Founder, Ash Ashutosh, introduces the company and its technology to the Field Day delegates.

Personnel: Ash Ashutosh

Actifio Architecture Overview

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, introduces the company’s product architecture.

Personnel: Brylan Achilles, Chandra Reddy

Actifio Resiliency Director Overview and Demo

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, introduces Actifio Resiliency Director.

Personnel: Brylan Achilles, Chandra Reddy

Actifio Global Manager Overview and SQL Server Demo

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, introduce Actifio Global Manager and demonstrate its use with Microsoft SQL Server.

Personnel: Brylan Achilles, Chandra Reddy

Actifio Global Manager Oracle and Ansible Demo

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, demonstrate Actifio Global Manager with Oracle and orchestration with Ansible.

Personnel: Brylan Achilles, Chandra Reddy

Actifio ReadyVault and Object Storage

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, introduce Actifio ReadyVault and show how the product can work with object storage. Ash Ashutosh, CEO and Founder, then returns to answer questions.

 

Bits are cheap! Don’t sell yourself short on key length

Using SSH keys to perform password-free login is quite common in Unix hosts and in  Appliances that have embedded Unix (like Storwize products).

You effectively have a public key which is shared  and a private key (usually with a PPK extension) that is not shared.  Think of the public key like the lock in your front door, that  everyone can see.   Think of the private key like the door key in your pocket or hand-bag.   If you keep your private key secure, your door is relatively secure.  If you lose your keys, your door is most likely no longer secure (unless they are down the back of the couch).

Sticking with the door analogy, the risk with a door lock is that someone could still just try to kick your door in (brute force attack) or pick your lock.   The bit length of the key can make this harder to achieve: the longer the bit length the harder it is to crack.

It is not unusual to see instructions that suggest you use a command like this to generate keys, where a bit length of 1024 is specified with an RSA key:

ssh-keygen –b 1024 -t rsa -f ~/.ssh/id_rsa

Of if using PuTTYgen to create the keys, to see instructions like this:

1. Start PuTTYgen by clicking Start > Programs > PuTTY > PuTTYgen. The PuTTY Key Generator panel is displayed.
2. Click SSH-2 RSA as the type of key to generate.
   Note: Leave the number of bits in a generated key value at 1024.

The problem is that these instructions are all old.  In fact using the ssk-keygen command syntax example shown above would represent a down-grade in what is now the default setting.   The wiki and man pages for ssh-keygen both confirm that for RSA, the default length is now 2048 bits (not 1024 bits).

To confirm what key length you get by default, simply make a test key and then read it back.  In this example I create a new public/private key pair called testkey  without specifying a bit length (there is no -b 1024):

anthony$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/anthony/.ssh/id_rsa): testkey
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in testkey.
Your public key has been saved in testkey.pub.
The key fingerprint is:
SHA256:or3Yhykd0W569QcHtGk4ZMSdQDYlaM9ko+TiWQZ7pp4 anthony@Anthonys-Actifio-MacBook-Pro.local
The key's randomart image is:
+---[RSA 2048]----+
| =B+.. |
| . +.Bo+ |
| B O + o |
| + O = = |
| ..@S o . |
| o=.o . . . |
| .o.B . . o |
| .oE.o . . |
| ..oo . |
+----[SHA256]-----+

I then read the file back using the -l and -f params (specifying the name of the file) and confirm the bit length, which in this case is 2048 bits as highlighted by the red text:

anthony$ ssh-keygen -l -f testkey
2048 SHA256:or3Yhykd0W569QcHtGk4ZMSdQDYlaM9ko+TiWQZ7pp4 anthony@Anthonys-Actifio-MacBook-Pro.local (RSA)

When using PuTTYGen, if you use a recent version you will note that the default bit length is now 2048 (as indicated by the red circle).   If you load a key you should see the bit length of the loaded key as indicated by the orange circle.

So if you see instructions specifying the creation of a 1024 bit key, I suggest you ignore them and use 2048 bits or at the least question this with your vendor.   Equally if you are using older keys, it is well worth checking their bit length and generating new keys, since this will give you the now default bit length of 2048, but also renew them, reducing the risk of someone using an older (and potentially leaked) key inappropriately.

 

 

 

Triple 000 app – a recommended app for all your smart devices

I want to draw my Australian friends to an app called “Emergency +”, available for your smart device (Apple, Android and Windows).

The scenario is simple.  You see something terrible:  a fire; a car crash; a natural disaster.  The standard response is simple:   You should dial 000 (the 911 equivalent in Australia).

One of the first things you are asked is usually:

“What is your location?  What are you?”

Now that’s easy if you are at home….  but what if you are on the road, or at a store,  or walking the dog?

The idea is to eliminate confusion over your location.

First you open the App and see this:

Select the map and determine your location:

Then dial 000 using the App (you will get a pop-up like this):

 

It will start a phone call, at which point you should switch to  speaker mode (hands free) and jump back to the app.  You now have your address and your exact location (to a number of meters) for you to share with the responder on the phone.

Details of the app are here:
http://www.triplezero.gov.au/Pages/EmergencySmartphoneApp.aspx

Look in your smart device app store for an app with this icon:

I urge you to install this app and also encourage your friends to do so too.
Sit down with your family and install it on everyone’s phone.  Do it tonight.

It might save someones life.

Exact MSP Space Accounting on a Storwize Pool

I have blogged in the past about the classic IT Story, The Cuckoo’s Egg by Clifford Stoll.   A true story that details how Clifford discovered a hacker while trying to account for 9 seconds of mainframe processing time.

I was reminded of this recently while doing an MSP Space Accounting project.  MSPs (Managed Service Providers) are understandably cost focused as they try to compete with low-cost IAAS (Infrastructure As A Service) providers like Amazon.   To control costs, shared resources are normally employed as well as thin-provisioning and its cousin over-provisioning (don’t confuse them,  thin-provisioning just means using only the exact resources needed for an objective, where over-provisioning means promising or committing to more resources than you actually have, in the hope that no one calls your bluff.   You can always use thin-provisioning without using over-provisioning).

A Storwize pool can use both thin and over-provisioning.   As an MSP if you are looking at pool usage you may want to be clear exactly how much space each client in the shared pool is using.   Now I don’t want to burn time explaining the exact workings of thin provisioning (something that Andrew Martin explains very well here), but I wanted to point out a quirk that may confuse you while trying to do space accounting.

In this example I have a Storwize pool that is 32.55 TiB in size and is showing 22.93 TiB Used.  You can clearly see we have over-allocated the 32.55 TiB of disk space by having created 75.50 TiB of virtual volumes!

Now this is significant because if I wanted to do space accounting I would expect the Used capacity of all volumes in the pool to sum up  22.93 TiB of space.  In other words if five end clients are sharing this space and I know which volumes relate to which client, I would expect the sum total of all volumes used by all clients to equal 22.93 TiB.

If I bring up the properties panel for the pool I can clearly see metrics for the pool including the extent size (in this example 2.00 GiB, remember that, it is significant later).

Now for each thin provisioned volume I get three size properties:

Used: 768.00 KiB   

Real: 1.02 GiB   

Total: 100.00 GiB  

To explain what these are:

• Used capacity is effectively how much data has been written to the volume (which includes the B-Tree to track thin space allocation).
• Real capacity is how much space in grains has been pre-allocated to the volume from extents allocated from the pool.
• Total capacity is the size advertised to the hosts that can access this volume.

This means I could sum either Used capacity or Real capacity.   Since Real capacity is always larger than Used capacity, it makes more sense to sum this.  Especially if this is the number I am using to determine usage by clients inside a shared pool.

To get the used space size of all volumes we need to differentiate between fully provisioned (Generic) volumes and Thin-Provisioned volumes.

This command will grab all the Generic volumes in a specific pool (in this example called InternalPool1):

lsvdisk -bytes -delim ,  -filtervalue se_copy_count=0:mdisk_grp_name=InternalPool1

This command will grab all the thin volumes in a specific pool (in this example called InternalPool1):

lssevdiskcopy -bytes -delim , -nohdr -filtervalue mdisk_grp_name=InternalPool1

Add the -nohdr option if you wish to use these in a script.

So for the generic volumes we can sum the capacity field.   In this example pool, I used a spreadsheet and found it sums to 19,404,662,243,328 byes

So for the thin volumes we can sum the real capacity field.   In this example pool,  I used a spreadsheet and found it sums to 5,260,831,053,824 bytes.

This brings us to a combined total of 24,665,493,297,152 bytes which is 22.43 TiB.

The problem here is obvious.   I expected to account for 22.93 TiB of space, but summing the combined total of actual capacity for full-fat volumes and real-capacity for thin volumes doesn’t add up to what I expect.  In fact in this example I am short by around 0.5 TiB of used capacity.  How do I allocate this space to a specific client if no volume owns up to using it?

I can actually spot this in the CLI as well using just the lsmdiskgrp command.  If I subtract real capacity 24,665,493,297,152 from total capacity 35,787,814,993,920 I get 11,122,321,696,768 bytes, which is nowhere near reported free capacity of  10,578,504,450,048 bytes.  This again reveals 543,817,246,720 bytes (0.494 TiB) of allocated space that is not showing against volumes.

IBM_Storwize:Actifio1:anthonyv>lsmdiskgrp -bytes 0
 id 0
 name InternalPool1
 status online
 mdisk_count 1
 vdisk_count 525
 capacity 35787814993920
 extent_size 2048
 free_capacity 10578504450048
 virtual_capacity 83010980413440
 used_capacity 23916077907968
 real_capacity 24665493297152

The answer is that the space is actually allocated to volumes, but is not being accounted for at a volume level.   If you scroll up to the second screen shot showing the Pool overview you can see the Extent Size is 2 GiB.   That means the minimum amount of space that gets  allocated to a volume is actually 2 GiB.  But if we look at the volume properties of a single volume, there is no indication that this volume is actually holding down 2 GiB of pool space.     In this example I can see only 1.02 GiB of space being claimed.  So for this example volume there is actually 0.98 GiB of space allocated to the volume which is not actually being acknowledged as being dedicated to that volume.

So how do I cleanly allocate this 0.5 TiB?

I see two choices.   The first is to simply determine the shortfall, divide it by the number of thin allocated volumes and then add that usage to each thin volume.     In this example I have 519 thin volumes, so if I divide  543,817,246,720 by 519 thats pretty well 1 GiB per volume I could simply add to that volume’s space allocation.

The second is to accept it as a space tax and simply plan for it.   The issue is far less pronounced if the volume quantity is small and the volume size is large.  The issue is also far less pronounced with smaller extent sizes.   At very small extent sizes it in fact will most likely not occur at all or be truly trivial in size (like Clifford’s 9 seconds). In this example simply using 1 GiB extents would have pretty well masked the issue.    But remember that the smaller your extent size, the smaller your maximum cluster size can be.  A 2 GiB extent size means the maximum cluster size is 8 PiB.

 

 

Mapping Linux RDMs to Storwize Volumes

As a follow-up to my previous post about MPIO software and RDMs, I suggested SDDDSM could help you map Windows volumes to Storwize volumes.    This led to the obvious question:   What about Linux VMs?

In a distant time there was a version of IBM SDD for Linux (in fact you can still download it).  But because it was closed source and used compiled binaries, it meant that users could only use specific Linux distributions/Kernel versions.    This was rather painful (especially if you upgraded your Linux version due to some other bug and then found SDD no longer worked).    Fortunately native Multipathing for Linux rapidly matured and offered a simple and native option that is definitely the way to go (and please don’t listen to the vendors pushing proprietary MPIO software, integration native to the Operating System using vendor plug-ins is in my opinion  the only acceptable MPIO solution).

Either way, it turns out you don’t even need multi path software to map a Storwize Volume to an Operating System device.

In this example I have created a volume on a Storwize V3700 with a UID then ends in 0043.

It is mapped as a pRDM to a VM, I can see the same UID under the Manage Paths window.  You can see the same UID at the top of the window (ending with 0043).

On the Linux VM that is using this VM, I want to confirm if the device /dev/sdb matches the pRDM.   In this example we use the smartctl command.   We can clearly see the matching Logical Unit ID  (ending in 0043), so we know that /dev/sdb is indeed our pRDM.

[root@demo-oracle-4 ~]# smartctl -a /dev/sdb
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-573.3.1.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor: IBM 
Product: 2145 
Revision: 0000
User Capacity: 5,368,709,120 bytes [5.36 GB]
Logical block size: 512 bytes
Logical Unit id: 0x60050763008083020000000000000043
Serial number: 00c02020c080XX00
Device type: disk
Transport protocol: Fibre channel (FCP-2)
Local Time is: Sat Apr 16 23:16:09 2016 EDT
Device does not support SMART

Error Counter logging not supported
Device does not support Self Test logging
[root@demo-oracle-4 ~]#

If you find smartctl is not installed, then install the smartmontools package:

yum install smartmontools

If we have Linux multipath configured, we can also use the multi path -l (or -ll) command to find the UID and determine which Storwize Volume is which Linux device.  Again I can easily spot that mpathb (sdb) is my Storwize volume with the UID ending in 0043.

[root@centos65 ~]# multipath -ll
mpathb(360050763008083020000000000000043) dm-6 IBM,2145
size=5G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
 `- 5:0:1:0 sdb 8:96 active ready running

So Linux users will actually find it quite easy to map OS disks back to the Storwize volume.

 

Do RDMs need MPIO?

I got a great question the other day regarding VMware Raw Device Mappings:

If an RDM is a direct pass though of a volume from Storage Device to VM, does the VM need MPIO software like a physical machine does?

The short answer is NO,  it doesn’t.  But I thought I would show why this is so, and in fact why adding MPIO software may help.

First up, to test this, I created two volumes on my Storwize V3700.

I mapped them to an ESXi server as LUN ID 2 and LUN ID 3.  Note the serials of the volumes end in 0040 and 0041:

On ESX I did a Rescan All and discovered two new volumes, which we know match the two I just made on my V3700, as the serial numbers end in 40 and 41 and the LUN IDs are 2 and 3:

I confirmed that the new devices had multiple paths, in this example only two (one to each Node Cannister in the Storwize V3700):

I then mapped them to a VM as RDMs, the first one as a Virtual RDM (vRDM), the second as a Physical (pRDM):

Finally on the Windows VM I Scanned for New Devices and brought up  the properties of the two new disks.   Firstly you note that the first disk (Disk 1) is a VMware Virtual disk while the second disk (Disk 2) is an IBM 2145 Multi-Path disk.   This is because the first one was mapped as a vRDM, while the second was mapped as a pRDM.

So here is the question, if the Physical RDM is a multi-path device, does it have one path or many?      The first hint is that we only got one disk for each RDM.  But what do I see if I actually install MPIO software?    So I installed SDDDSM and displayed path status using the datapath query device command

C:\Program Files\IBM\SDDDSM>datapath query device

Total Devices : 1

DEV#: 0 DEVICE NAME: Disk2 Part0 TYPE: 2145 POLICY: OPTIMIZED
SERIAL: 60050763008083020000000000000040
============================================================================
Path#    Adapter/Hard Disk          State  Mode    Select Errors
    0  Scsi Port2 Bus0/Disk2 Part0  OPEN   NORMAL      86      0

C:\Program Files\IBM\SDDDSM>

What the output above shows is that there is only one path being presented to the VM, even though we know the ESXi HyperVisor can see two paths.

So this proves we didn’t actually need to install SDDDSM to manage pathing, as there is only one path being presented to the disk (the HyperVisor is handling the multiple paths using its own MPIO capability VMW-SATP-ALUA, which we can see in the ESXi pathing screen capture further up above.

Having said all that, there is one advantage from the Windows VM perspective to have SDDDSM installed, which is that I can see that Disk2 maps to the V3700 volume with a serial that ends in 40 (rather than 41).   So If I wanted to remove the vRDM volume (Disk 1) I know with safety that the volume ending in ’41’ is the correct one to target.

 

Evergreen Storage? Can it actually work?

Pure Storage is one of several hot flash vendors in the market right now.   Despite some negativity about their recent IPO, it actually shows that the market thinks they have got their product and execution right.

One challenge for every Flash vendor out there (and there are quite a few) is to be able to explain the why.   Why my product and not another vendors?

One thing Pure Storage promote as a strong ‘why us‘  is their concept of Evergreen Storage, described here:

http://www.purestorage.com/content/dam/purestorage/pdf/datasheets/Pure_Storage_Datasheet_Evergreen.pdf

Fundamentally they are saying that as technology evolves, their modular physical design and stateless software design will allow you to upgrade components without having to move data or do any of these forklift upgrades.  Here is an image from their brochure:

Even with Storage vMotion, the need to move data between storage arrays remains a major additional cost of replacing or upgrading storage hardware, and the ability to minimise or eliminate this work is definitely a huge plus.

But can they actually do it?  Do we have working examples of other vendors achieving this?

There is actually a good working model of a product that has done exactly this since 2003: The IBM SAN Volume Controller.     When IBM released the SVC in 2003, the first model (the 4F2), had only 4 GB of RAM per node with 2 Gbps FC adapters.   Since then, IBM have released a succession of new models as Intel hardware has evolved, with the current nodes having at least 32 GB of RAM, dramatically more cores, and optional 16 Gbps FC adapters!

The neat thing is that clients who invested in licensing in 2003, have been able to upgrade their nodes, with data in place, over successive years.   The cost of new nodes has been relatively low compared to the performance and functional benefits that each release has provided.   So I know for a fact that this idea of an Evergreen storage product is not only possible, but positively demonstrated by IBM.

The challenge for any vendor trying to do this is three fold:

1. The technology really has to support seamless upgrades.   While the IBM SVC certainly did and does, there were some minor hiccups along the way.   One example was that first model, the 4F2, could not support the later 64 bit firmware releases, which meant that if you held off upgrading for too long, upgrading to new hardware needed some special help or a double hop to get the upgrade going.    Another example is bad racking:   Racked and stacked badly, pulling one node out could result in a partner node being disturbed (something I sadly have seen).
2. The vendor needs to remain committed to the product.   While I laud IBM’s success with the SVC (now going even stronger with its Storwize brothers),  a sister product released at the same time, the Storage File System (sometimes called Storage Tank), did not get market traction and did not progress very far before being replaced by GPFS (which was not exactly a one for one replacement).  And while the DS8000 continues going strong (long after Chuck Hollis, in a classic piece of EMC FUD,  declared it dead),  its little sister, the DS6800, truly was dead within months of being released.   Its early months were so drama laden (sometimes sadly referred to as a crit-sit in a box) that new models were never released, which was equally sad, as once the code stabilised it became a great product.
3. The vendor needs to hang around.   This one seems fairly obvious.   Clearly if someone were to buy Pure Storage (if the structure of the company allowed someone to do this), they also need to support this strategy.

So can Pure Storage do it?   Only time will tell, but they have made a great start and the industry has shown the concept is possible.   I will watch their progress with great interest!