It’s Halloween, so let’s visit a graveyard…. an IBM Graveyard… filled with typewriters

Hey kids, did you to go trick or treating this Halloween?

Yes? No?

Ok… how about we visit a spooky graveyard scattered with the bones of products long since dead?

Thats right kids… its time to visit an IBM Graveyard.

In this series of articles I am going to take you to some of the more interesting graveyards and recount tales of great success and of miserable failure from my beloved IBM. Oh yes.. let’s go to one of IBMs graveyards!

But where to start?? What’s that sound? Is it the whiring of an IBM Selectric Golfball?
Why yes… its 1973 and the IBM Typewriter business is king! No not just king but Emperor! By some accounts IBM had over 75% of the electric business typewriter market at that time. In 1973 you did NOT get fired for buying IBM Typewriters…

But actually our story starts forty years earlier in 1933 when IBM bought a struggling typewriter company called Electromatic Typewriters Inc. It invested heavily in its new typewriter division and in 1935 brought out the first IBM Typewriter: The Model 01, which was an immediate sales success.

• Quiz question – did IBM bring out the first commercially successful typewriter?
• No! That was Remington – the same Remington who made guns!

However a true first was the IBM Executive Typewriter, announced in 1941 and first shipped in 1944. It literally changed the world of typing, since it introduced proportional spacing, just like a book printer could do. In this example of typed output you can see non-proportional font, every letter is the same width:

However the new IBM Typewriter used a proportional font meaning narrow letters were narrow and wide letters were wide. Suddenly typed letters looked like they had been sent to the printers.

• Quiz question – did IBM bring out the first commercially successful electric typewriter?
• No! That was the company they bought, Electromatic, although many companies made electric typewriters before that.

However while IBM continued to innovate bringing out more reliable and quieter models, the big change happened in 1961 with the introduction of the Selectric golfball typewriter. The ideas that IBM explored to create the Selectric were not new. The difference was that IBM was the first company to create a commercially and mechanically successful product using the concepts they contained. Ironically they also kept designing and manufacturing typebar style typewriters (with letters on individual arms), principally because a true speed demon typist actually could type faster on a typebar than a golfball. Still, the removable golfball meant one typewriter could produce over 40 different fonts (you would need 40 typebar typewriters to do that!).

• Quiz question – Did IBM introduce the first typewriter with a replaceable font?
• NO! That was Blickensderfer in the 1880s (yes you read that right, nearly 70 years earlier)

However it was the introduction of the Correcting Selectric II in 1973, one that could literally remove typing errors straight off the page, that literally blew away the remaining competition. At one time the delivery time on new orders went from 4-6 weeks to 6-9 months. In Lexington Kentucky, there were 5 manufacturing lines building them, each working 24 by 7. One of the five was a ‘clean’ line designed for tours. The other four dedicated to just cranking them out. In 1974 the Lexington plant made over 1 million correcting Selectrics. This feels like a peak time for American manufacturing. And there were also IBM typewriter factories all over the world (including Wangaratta in Australia!)

I personally feel the Correcting Selectric is something like IBM’s iPhone. Now if you think that’s a stretch, consider my reasons:

• It was truly pervasive, probably the most visible product IBM have ever released. It was also sold into the most clients, since it worked in every industry vertical and any business size as well as for personal use. More than any other product they made, IBM and their partners could sell into nearly any type or size of business.
• The Correcting Selectric generated a mania that reminds me of iPhone fever. It was full on. Once a secretary in the office got one, the clamouring from all the other ones began…
• There was a real focus on design, a focus that I have never seen on any other IBM product. They truly wanted it to look good. Compare the Selectric to this later 1967 Model D Executive (which is a TypeBar, not a Golfball). The difference is chalk and cheese.

The Selectric become a real symbol of a 1960s office. So much so that the TV show Mad Men chose to use the Selectric II in many episodes, even though the episodes set in the 1960s was showing a model introduced in 1971.

The colour choices were also outstanding, including, but not limited to:

• Raven Black
• Garnet Rose
• Emerald Green
• Pearl White
• Topaz Bronze
• Classic Blue
• Sandstone Beige

And it was common for clients to request specific colours, like University of Texas orange or Kansas State University purple. The sales reps for Law firms would send fabric samples from their desk chairs to ensure their typewriters blended in (unlike this cat).

Garnet Rose

Emerald Green

Classic Blue (http://ibmselectric.blogspot.com/)

Raven Black. (http://ibmselectric.blogspot.com/)

Topaz Bronze http://ibmselectric.blogspot.com/2016/03/topaz-bronze-selectric-ii-with.html

Marlin Blue Selectric III (http://ibmselectric.blogspot.com)

Pearl White Selectric III

Now checkout this video. It is short and perfectly explores how the GolfBall worked. What strikes me in this video is just how good the print quality looks even today.

Then watch this video and marvel at how it worked at all:

And shipping millions of typewriters meant 1,000s of service engineers. A CE (Customer Engineer) as they were called would routinely get a patch of 1000-1500 machines, often doing 10-20 service calls a day while trying to fit in regular Preventative Maintenance (PM) on each machine in their patch. Sometimes a client literally had over 1,000 typewriters in a single campus or building, so the CE never left the site. The HQ for JC Penney on 6th Avenue in New York City had 1100 typewriters – all in one building. The IBM CE became a familiar face, a part of the family (sometimes literally meeting their future partners while taking service calls), walking into schools and office buildings across the country.

These were the IBM Territories for the City of Melbourne in 1975. Territory 7 looked like way more walking then territory 10!

The Selectric also appeared in other products, like the IBM 7030 Stretch, which led to some amusing requests from the Large Systems engineers for Office Products Engineers to come and help them fix their mainframe typewriter.

Are they waiting for the typewriter repairer?

Of course they could also have asked their local KGB spy for help, since the KGB turned out to be expert at bugging US Consulate Selectrics. Below are Soviet era bugs planted in US Embassy typewriters in the USSR. Literally one of the first ever keystroke loggers:

Bugged Selectric Power Switches (found by the NSA)

Apart from being an easy device to spy on, it was also an easy device to lend out. These photos were shared to me by Henry Mydlarz, an IBMer who volunteered at the Melbourne Royal Children’s Hospital Good Friday Appeal in 1975. IBM lent the television station several dozen Golfball typewriters to be used in the donations room by the station staff. IBM also provided volunteer typewriter technicians to stand by round the clock and attend to any problems working in shifts. Note the colour of that gorgeous Raven Black unit that the child is playing with.

Under lights and camera… the operators are typing up those donations!

The fascinating thing about the Selectric is how slow the release program looks and yet how enormously successful it was. The Selectric came out in 1961. The Selectric II came out in 1971. The Selectric III did not come out till 1980! By the 1980s the enthusiasm to innovate in typewriters seemed to be gone. IBM instead was focused on things like producing one of the worlds first hybrid photocopier/laser printers (the IBM 6670) and one of the worlds first office inkjet printers (the IBM 6640).

With the release of the IBM PC in 1981, the focus was then truly elsewhere. If the goal in the past was to get a typewriter onto every desk, now it was to get a PC onto every desk alongside that typewriter. That launch hysteria of 1972 was a distant memory and companies like Olivetti and Xerox were competing with better and cheaper offerings. IBM released an electronic version of the Selectric III, internally called the E-tron, a term they had to ban when they realised Étron in French means excrement (as Audi found in 2019.) The nickname was appropriate though, as the machines extensive use of plastic parts and dodgy power supplies resulted in major client dissatisfaction and a much higher call rate than it’s all-mechanical predecessors. All the hard earned lessons about how to make a mechanical machine truly reliable seemed to be forgotten and replaced with a plastic carriage, not built for heavy duty work.

The Wheelwriter follow-on used a daisywheel, which was not an IBM invention and one that IBMs competitors (like Xerox) already had. Wang and other competitors meanwhile brought out high quality desk top word processors and the magic spell was truly broken. Although to be clear, the first company to claim they had developed a Word Processor was IBM themselves with their 1964 release of the Magnetic Tape/Selectric Typewriter or MT/ST, which could record typed material on a tape and allow the operator to ‘play it back’, having the typewriter type the recorded material automatically.

In 1991 the mighty Lexington plant, a site that produced among other things millions of typewriters and the legendary IBM model M keyboard, was sold to what became Lexmark. By 1995 the electric typewriter market was effectively dead, killed by the word processor and the PC.

And the Selectric? It became the favourite of typewriter enthusiasts and collectors and the goto story by IBM marketing. The typewriter, once a core part of “Machine” in International Business “Machines” went to the IBM Graveyard.

Which product should I cover next? Have you got a favourite IBM Graveyard you would like to visit? There are so many to choose from….

Remembering the time that IBM said all tape cartridges could have a good lie down

I recently stumbled across this fascinating study about what happens if an LTO tape drive is mounted vertically.

So rather than mounting it horizontally in a rack like this:

They tried mounting it vertically like this:

Now if they had rung me and asked if this was a good idea, I could have saved them a mountain of time by sharing my mildly qualified opinion:

“Hell no”

But apparently they didn’t have my number.

So instead they ran a whole battery of backup tests and worked this out scientifically. And yet they came to the exact same (but slightly longer) conclusion:

Based on the data from testing the new IBM LTO8 drive, when the drive is mounted in the vertical direction, the drive delivered significantly worse performance than in the horizontal position. In this calibrated test, LTO Tape Driver Doctor, during repeated tests, measured as much as 3X slower performance when the drive was mounted in the vertical direction as compared the horizontal direction. Furthermore, the drive reported an order of magnitude increase in write servo skips in this drive. Large servo skip count indicates the potential for large losses of cartridge capacity that will reduce the total useable capacity of the LTO8 tape drive. Based on this test, we recommend mounting IBM LTO8 drives in the horizontal direction.

Resulting in this brilliant and informative graphic:

The reality is that all cartridge drives starting with IBMs 3480 and marching on from there through the 3490, 3590, 3592 and LTO drives are all mechanically designed for the cartridge to be inserted horizontally. The mechanics of moving tape reliably are hard enough without complicating things by making the orientation of the drive optional.

So it is settled then: tape cartridges are horizontal creatures.

But here is something curious. Look through the square viewing window of this STK ACS:

Or this picture of IBM’s classic 3495:

What do you notice about the tapes?

THEY ARE ALL MOUNTED VERTICALLY!

And yet the carts need to be horizontal to go into the drives, making the robots job much more complex than it needs to be.

Compare this to the IBM TS4500 where the carts are all horizontal, making the robots life much easier:

So what gives? Why were early tape libraries designed to make things harder than they needed to be?

The answer lies in these reel to reel tapes (thanks Columbia University for this great wide-angle picture):

For any sort of reel to reel tape (including audio, film as well as data) it is a well established fact that flat storage even during transport could cause wraps of tape to slip and push against the flanges, resulting in edge damage and in some cases erratic tape motion. This requires all reel tapes to be hung vertically so that the weight is supported by the tape hub.

So when IBM brought out the 3480 cartridge, the same logic was applied. There was even an official manual shipped with every single cartridge drive:

 Care and Handling of the IBM Magnetic Tape Cartridge, GA32-0047

This declared you could only lay your cartridges flat temporarily:

Although cartridges are shipped and stored on their sides, you can lay the cartridges flat temporarily while moving them. The bottom of each cartridge has two raised areas that fit into the indented label area on the top of another cartridge. This construction helps prevent the cartridges from sliding while moving them.

Do not stack more than six cartridges.

However when IBM began work to add auto-loaders to their 3480 cart drives, they realised that tapes could end up sitting horizontally in the loaders for long periods of time (while waiting to be loaded or unloaded). So IBM began to study what if any impact was being experienced by this, literally running 1000s of cycle tests along with bench tests and scientific modelling. There was no tape servo positioning data in these early drives, meaning everything had to be written and read back and temporary errors and retries carefully monitored. It became clear that the cartridge leader block (which earlier reel to reel tapes did not have) acted like an anchor to the outside of the wound tape. So when properly rewound with the leader block properly in place, the cart could be safely stored horizontally.

So the requirement to store cartridges on their side was removed and because this happened while they were developing the 3584 (aka the TS4500) it allowed a much simpler robot to be designed. It also meant the above statement literally changed to just this:

Do not stack more than six cartridges.

IBM Distinguished Engineer Ric Bradshaw tells me that each time IBM has released new cartridge types (such as 3490 to 3590 to 3592 and LTO) plus extended length and density carts, similar testing and modelling continues to be done, with increasing scientific accuracy due to improved servo positioning data. This ensure new carts can always be safely stored horizontally.

And so allowing all tape cartridges going forward, to have a good lie down.

Sleep well sweet cartridge…

Remembering that time when IBM brought a beehive into the datacenter

I love the idea of biomimicry: the art of solving engineering problems by observing nature. If you have ever used a piece of velcro, then you have used something invented through biomimicry.

Japanese bullet trains (or Shinkansen) are also famous for using biomimicry, copying features of Owls, Adelie Penguins and KingFishers to dramatically reduce noise and vibration and resulting in their ability to travel at remarkable speeds.

Japanese Bullet Trains

But do you know that IBM also once embraced biomimicry? Thats right…. in 1974 the B in IBM also stood for BEEHIVE!

This all started in the late 1960s when IBM began developing something they had never tried before: a genuine robotic tape library. Developed at IBM’s Lab in Boulder, Colorado, the developers concluded the most efficient shape to store data cartridges was in a honeycomb array. The product they developed was code named Comanche and was announced as the IBM 3850 Mass Storage System (MSS) in October 1974.

It contained a huge number of firsts for IBM. For one thing it used a whole new circular tape cartridge that looked like this:

Each of these bullet shaped cartridges were two inches in diameter and four inches long, holding a spool with 770 inches of tape. The cartridges could hold 50 MB and were read diagonally. They were designed to be robotically fed into a whole new tape drive known as a DRD or “Data Recording Device” – pronounced “DiRD”. These drives were going to be called a “Tape Recording Device”, but that was changed when the developers starting shortening that to “TiRD”.

The carts were stored in a honeycomb library known as an IBM 3851 with two accessors (robots). Given the time period, it is very hard to find decent photos (or videos) but here are a couple of the accessor and the honeycombs:

This product was packed with firsts for IBM:

• First robotic tape library (with not one but two accessors!)
• First circular tape cartridge
• Brand new tape drive design
• Brand new Control unit (the IBM 3830-003)
• Disk virtualisation using automated tape. Thats right, this thing could take two 50 MB tape carts and use them to emulate a 100 MB removable disk pack. This meant the tape library could appear to the mainframe as a mountain of removable disk packs, with up 16 online at any time. This feature was phenomenal and users loved it because it gave them what seemed like unlimited storage for huge batch jobs and testing new apps with copied production data.

And in case you are unimpressed by 100 MB of disk space, compare the relative sizes (for the time), remembering those little carts were just four inches long….

Now at this point you may be thinking, wow this sounds fantastic – how come I have never heard of it?

The problem was that list of ‘firsts’: there was simply too much new technology all in one product. The MSS was a master class in complex and demanding microcode (firmware) and there was way too much dependence on exact mechanical performance. Unless the IBM Engineers who installed and maintained it were truly expert at their job, way too many things could go wrong. The pickers had to be very well aligned to get the cartridges in and out of their honeycomb positions. Different sections of honeycomb could be imperfectly aligned when assembled together, or worse the honeycomb itself could be distorted (or badly manufactured). If a cartridge was forced into an undersized honeycomb position it could cause severe damage to the machine and destroy the data in the cartridge. The accessors were also notorious for occasionally crashing into badly inserted carts or even dropping and losing carts.

War stories about the MSS abound, such as the attempt to detect badly sized cells using a slightly over-sized test cart. A library microcode engineer brought one to a troubled install at the Johnson Space Center in Texas. They began the test routine and within minutes it found a badly shaped cell. The accessor strained and strained to insert the test cart into the bad cell, trying so hard it tripped not only its circuit breaker, but a whole series of upstream circuit breakers, powering off the whole NASA data centre. A classic ‘what have you done’ moment.

Over-sized test cartridge that turned NASAs computer centre off

Another amusing story concerns the photoshoot for when the MSS was featured on the cover of the 1974 IBM Annual Report. The position of the accessors for the cartridges was based on light-based indexing. If anything went wrong they would park themselves in garages at either end of the aisle. The photographer was shooting at an actual customer location (Montgomery Wards Data Center in Lenexa, Kansas) and needed more light to capture the honeycomb, so he instructed the account rep to get inside the machine with an auxiliary flash. As soon as the flash went off, the accessor concluded it was broken and headed for it’s garage, moving at five mile per hour. The salesman’s comment:

“I never realized how fast 5 mph could seem until I saw that thing headed for my belly-button.”

He was not injured, but the unexpected accessor crash literally caused a system outage (oops).

An example of IBMs struggle to master the microcode was that while there were two accessors, only one could be active at a time. IBM could never get the two to work simultaneously, as they would sometimes violently crash into each other. IBM only managed to finally achieve this feat more than 20 years later in a different product, the IBM 3584, with a feature we called dancing robots, or officially “dual active accessors”.

Another problem with the product was IBMs obsession with moving things and people around (IBM at the time truly stood for “I’ve been moved”). Having developed and initially manufactured the MSS in Boulder, in 1976 they moved MSS manufacturing to San Jose, California to make more space in Boulder to manufacture IBMs Copiers. And having moved the MSS to San Jose, in 1979 they then moved it to Tucson Arizona to make more space in San Jose for disk manufacturing. And then having moved it to Tucson, they then decided to cease manufacturing completely at that site and moved MSS manufacturing back to San Jose. Each move lost experienced staff and resulted in an overall loss of “tribal knowledge”.

So things did not go well for the MSS. In fact the big hint that things were not going well was when in 1984, IBM came out with a whole new 200MB cartridge (for the new IBM 3480), that was…. square! It clearly could not be used in the MSS.

At its peak there were only 350 MSS subsystems installed world wide (peaking at 274 in North America). Rapid advances in disk sizes and disinterest inside IBM, all meant the end for the humble honeycomb robot. It was withdrawn in August 1986 (without replacement) and maintenance was stopped in 1991 (which was quite fast for those days).

The MSS is now best remembered by IBM Collectors, who love to show off (and sometimes also sell) their souvenir MSS parts. You can see from left to right a data cartridge that has been opened, a green customer engineer test cartridge and a yellow system engineer test cart, as well as a honeycomb segment.

In terms of classic IBM recycling, the cartridges came in plastic trays, which were perfect to put in your drawers to sort out small bits of stationary. It is quite possible people are still using these, with no clue where they came from or what they were originally for:

Want to read more? Check these out:

https://www.ibm.com/ibm/history/exhibits/storage/storage_3850.html

https://en.wikipedia.org/wiki/IBM_3850

http://www.columbia.edu/cu/computinghistory/mss.html

Remembering the time a giant yellow robot saved IBMs Tape Business.

In 1987 IBM’s Executives in Armonk hated Robots.

THEY JUST HATED THEM.

Now to be clear I am not talking about this kind of robot:

Lost in Space Robot and Dr Smith in 1965

Or even this kind of robot:

Arnold as a robot in 1984

No… the kind that IBM hated was this kind… the kind that moved tapes for a living:

Photo from 1992, ok I cheated a little with the timing

The problem was that while IBM led the world in developing automated and virtual tape libraries, none of these products had ever been a commercial success.

Here is a short list of just some of IBMs robot products:

• In 1961 IBM developed the Tractor tape library for the NSA, which turned out to be a bit of a one off.
• In the early 60s IBM released the slightly insane Walnut photostorage library developed for the CIA. Total sales: 7
• In 1974 IBM released the Mass Storage System (the MSS, known as the 3850). Which was in hindsight an overly ambitious product, years and years ahead of its time. It was one of the first IBM products to be totally dependent on microcode, had a disk virtualisation layer (tape being used to virtualise disk in 1974!) and was inspired by a bee-hive! It was also by all accounts unreliable and not economical, which led IBM to finally withdraw it in 1986.
• In 1982 IBM announced the RS 1 robotic Manufacturing System followed by the RS 2. However despite several updates and releases, by 1986 it was also clearly not a commercial success. Pictured is the IBM 7565 Manufacturing System, introduced in 1983 (I think the robot should have made the operator a taller chair).

The IBM 7565 Manufacturing System controlled by an IBM Series/1

So by the late 1980s the last thing any IBM executive wanted to hear was a proposal for another robot. It may surprise you to also hear that there were IBM executives at the time who believed tape was dead and further investment in tape was wasted money.

This is what one IBM Vice President publicly said around this time:

The market for a robotic tape library is negligible, since 90 percent of the 300 million computer tapes in the world do not need to be accessed at the speed of a robot.

Now for those of you who have read The art of possibility, guess what management at StorageTek heard him say?

IBM just said there are thirty million tapes that need robotic access! HELL YES, SHOW ME THE MONEY!

Now contrary to what some people think, StorageTek (STK) was not founded to make tape libraries. It was started way earlier (in 1969) by four ex-IBMers who planned to make IBM compatible I/O (like tape drives and printers). They actually then went on to drive STK to bankruptcy trying to build an IBM compatible Mainframe. Tape Libraries started as a skunk works project (probably by ex-MSS IBMers) and were not in the companies plans until well into the 1980s, just as they were coming out of bankruptcy. They released their first Tape Library, the StorageTek Automated Cartridge System (ACS) 4400 tape library in 1987 and its followup, the PowderHorn, in 1992.

In short STK rapidly found they were on a winner with a product that customers loved and by 1993 they had sold over 5000 ACSs and were selling into pretty well every IBM Mainframe shop in the world. Later their libraries (bizarrely and impossibly modified to make the robot more visible) even starred in movies like Clear and Present Danger and Eraser (staring Arnold, not playing a robot):

Scene from Eraser with Vanessa Williams in a chair

IBM was now in serious trouble in the tape business. Selling IBM tape drives into STK silos was VERY hard work… it was like a walled garden (literally). The company that basically created the magnetic media market was being driven from it. Something had to be done.

IBMs only automation solution was the addition of auto-loaders to their tape drives. Taking this (a string of IBM 3480 Tape Cartridge Drives):

And turning them into this by adding auto-loaders. The benefit was that you could pre-stack tapes in the loaders at the start of batch and empty them at the end. So an IBM Marketing Rep could argue: “Look, you can load up to 40 tapes at once!”

This attempt at automation was so meagre that Thomas Henkel of the Boston based Yankee Group thought it was part of some secret plan. He was quoted in Computerworld magazine in June 9, 1986 as saying: “I think IBM has some tape library on the way”.

He could not have been more wrong.

The follow up product, the IBM 3490, announced in 1991, offered really cool display panels, but only slightly larger autoloaders.

Meanwhile the STK reps counterargument to these auto-loaders was, “Hold my beer and keep your 40 cartridges, because my ACS can store up to 6000 tapes!”.

Which meant STK was selling 1000s of these:

So the real solution was obvious: IBM also needed a robot! But clearly to outcompete STK they needed a big robot… no… bigger… NO.. I SAID BIGGER…. AND BRIGHT FREAKING YELLOW!!!!!

And thus in 1993 the world got the IBM 3495.

The 3495 remains IBM’s longest ever product. A model L50 (the largest model) was 28 meters long (that’s 92 feet)! That’s crazy huge for a datacenter. The internal IBM Code name was Caballero, but in the field it was sometimes referred to as Conan (as in Conan the Librarian). The robot itself was a repurposed General Motors Fanuc automotive industry robot. And it was yellow… really really yellow.

The thing I love about this product is that the robot was designed to make cars in a production line. The robot was bolted to the floor and the cars came to the robot. But now the robot needed to come to the tapes and so IBM made it a buggy to ride in. Yes that’s right, IBM made a car, for a robot that makes cars. And not just any car, but a car that could move a 408 Kg (900 pound) robot at 9 kmh.

Now the 3495 was not a huge seller, I believe total sales were not much over 100. In my home country of Australia, IBM sold precisely 8 of them. One issue was finding the level floor space to install one. But it freaked STK management out (they would threaten to kick you out of the STK Users Group if you bought one), it generated huge buzz and held the line while a far more sensible product was rapidly being finished (as well as Virtual Tape Library technology).

There are two videos on YouTube that are worth watching. The first is a slightly amazing marketing video made in 1992 that I was given when I started blogging at IBM. Given IBM had no official portal for videos, I posted this on my personal YouTube channel, where it has amassed over 40,000 views, seriously out competing one of my most popular personal videos: my Uncle starting a Lanz Bulldog with a blow torch.

This second video is equally amusing, shot I believe at a Kraft datacenter, it also has some footage of other old kit at the end. But there is a scene that shows the robot in its little car! It’s about 58 seconds into the video, don’t miss it!

The 3495 was withdrawn in 1998, but by then IBM had the very successful 3494 which IBM are still shipping (with a great many improvements) as the TS4500. The IBM TS7700 Virtual Tape Solution (VTS) that attached to it was also hugely successful. In Melbourne we used the VTS to totally evict STK from several Mainframe clients, tearing out dozens and dozens of STK silos.

STK meanwhile was bought by Sun in 2005, who were then bought by Oracle in 2010, who still ship tape products and libraries under the StorageTek brand.

And where did the big yellow robots go? Well they did not go onto to become our new robot overlords in their IBM designed cars. However I know at least one of them is still doing loyal service… guarding a letter box in Canberra. Possibly the finest recycling effort of an IBM product that I have ever seen.

Stand strong big yellow robot, you helped keep IBM in the tape business.

In 1957 the CIA asked IBM to invent Google Images, and IBM said yes.

This story starts with a scene straight out of a spy movie and I swear nearly every word is true.

In the mid 1950s, a CIA spook comes to IBM with a request: “Hey Blue suited dudes, make us a machine that can store millions of images that are searchable by keywords. In other words a data lake of images – just like Google Images, except Google is still 40 years in the future.”

The IBMers didn’t even blink (or ask what a Google was). They replied, Hell yeah!” (or something similar), and the resulting product (code named Walnut) was delivered to the CIA in 1961.  

It kinda worked like this:

• The Walnut has 100 large cylindrical carousels called document stores.
• Each document store held it’s data in in 200 small boxes that IBM referred to as cells.
• Each cell contained 50 strips of photographic film
• Each strip of film contained 99 photographs arranged in a 3 by 33 grid.

In total, each document store contained images of 990,000 documents, and up to 100 document stores could be used in a single Walnut system, for a total storage of 99,000,000 pages

Users would look up keywords stored on a separate system that used an IBM 1405, identifying individual documents to be retrieved. The index produced punched cards that were inserted into the Walnut. The Walnut system then retrieved the documents, copied them onto a blank film strip and developed them. The output could be viewed using a photo-negative viewer, or developed for full-sized printouts.

So to get this right, our CIA Operative searches for photos of villains using keywords and gets a list of images on punched cards. They then feed them into the Walnut, which then spits out a negative they can use to develop photos.

Imagine the James Bond scene: “Walnut, show me all images of Blofeld!” (followed by a 10 minute wait involving punched cards and photo development).

This is just like Siri, but with punched cards! (now hang on, which one is the real Blofeld?):

IBM tried to sell a commercialised version of this but got no takers. It seemed no one wanted to buy a product called Walnut (although it was now known as the IBM 1350).

However IBM realised that if they stored actual data on those negatives, rather than images, they might have a truly giant online read-only file system.

So they then productised this idea in response to demand from the Atomic Energy Commission to store 1 terabit of information to support super computer simulations. Yes you read that right – 1 whole terabit… that’s 128 GB! At that time no single system could address that much data.

The resulting product however was mildly insane and bizarrely huge and was an incredible mix of pneumatics and robotics combined with an electron gun and automatic photographic development. The product looked nothing like this:

It actually looked like this (a little bit larger, but how could you not love those blue covers?):

The Lawrence Livermore National Laboratory also loved it, using it for over 10 years as a (for that time) enormous online archive of read-only data (albeit with a response time in minutes). But despite the enormous amount of data you could potentially store in such a beast, it proved as equally impossible to sell commercially as it’s photo storing cousin.

The Computer History Museum in Mountain View has the only remaining example of an IBM 1360 (from Lawrence Livermore National Lab) shown in the following video. Note the tour guide claims it was retired because all the Service Reps who knew how to fix it had retired, but the truth is IBM no longer wanted to make spare parts for it and the client could not find any one to take over.

You can learn more here:

https://www.computer-history.info/Page4.dir/pages/Photostore.dir/index.html

and here: https://en.wikipedia.org/wiki/IBM_1360

And if you want to read about another story about IBM and the NSA, check out this one:

https://en.wikipedia.org/wiki/IBM_7950_Harvest

and here: https://www.ibm.com/ibm/history/exhibits/specialprod2/specialprod2_2.html

Sure Tape ain’t dead, but what about that microfilm?

The late 1950s and 1960s are a fascinating time in IBMs history. In those early days, innovations in business were achieved with physical machines… yes… actual weird and unique machines that did things mechanically. Huge gains were achieved with both mechanical innovations and just as importantly, through materials engineering, (which was hugely important) and new materials were constantly being considered.

The thing that really stands out looking back is just how bat shit crazy some of these machines and materials were. It is quite clear that regardless of how conservative you think IBM is or was, no idea was off the table and some of those ideas literally paid for the tables.

Now while its well understood that magnetic tape meant the end of punch cards, while the first magnetic tape drives shipped by IBM came out in 1952, by April 1955, IBM was making 72.5 million punched cards per day and in 1965, the number of cards being produced by IBM was still in the tens of billions a year.

Clearly the data explosion of our age was happening then, just at a different scale and onto different media. There was a constant challenge – how to store all of this punched card data?

One totally off the wall idea was to turn punched cards into 35mm microfilm, and so the IBM Vestal lab in New York State developed a set of machines to do exactly that. These were manufactured by IBM in Dayton New Jersey and announced on December 4, 1964.

This was a Micro-Processing System that used microfilm as a space saving method of storing documents that allowed you to still process and retrieve that data on punched cards. It consisted of several different machines:

• At the front centre (to the right of the standing lady) is the Micro-Copier/Reproducer that copied punched cards to microfilm.
• To the left of the standing woman (on the small desk) is the copier that turned microfilm into punched cards.
• At extreme left is a punch card interpreter, that could read punched cards and print information on them.
• At centre rear is a punched card sorter.
• Next to the sorter (on the small desk) is IBM’s copier for thermal film, which was developed by heat and light rather than chemicals.
• The key punch operator at extreme right scans the Micro-Viewer with roll film attachment to obtain descriptive data to be punched onto a master aperture card.

There was some immediate challenges. The product used ammonia to develop the microfilm. This ammonia had to be refilled by the client into a bottle that looked like this:

If they over-pressurised the gas storage tank, it could explode and at at least one client it literally blew the doors off the unit. Michael Caine would have been very impressed, but I am unsure what the client thought. Queue the classic scene from the Italian job:

Ammonia leak issues were common and IBM engineers used sulphur sticks (or their noses) to try and find them. This sounds impressive till you realise that the way a sulphur stick works is you light the stick and if it starts to produce a white smoke, you know there is ammonia nearby. I love the witch doctor image of an IBM Customer Engineer waving a smouldering stick around in the hope of producing white smoke. Again… Lord knows what the clients thought.

Sulphur sticks!

The bigger issue was simple – this was a way of putting punched cards onto a different medium. It was solving the wrong problem. Smarter solutions were needed.

Just before Labour day in 1969 IBM informed their customers they were getting out of microfilm business, although in classic IBM Style, some of these machines remained in service well into the 1970s.

Amusingly those ammonia bottles are also part of a fine tradition of IBM CE recycling. The one pictured was used to chill beer kegs and is now used as part of a nail gun. So not everything was lost.

Now if you are thinking, this was a little crazy, well just you wait! You aint seen nothing yet! In my next post we will go way down the rabbit hole with an IBM Product that makes exploding microfilm copiers look tame!

Sure Mainframe ain’t dead, but what about that toilet water?

That old ‘Mainframe is dead’ chestnut has really rolled around the field for a while, but despite the best hyperbole of many IT pundits, IBM Mainframes continue to thrive and survive, even if they are no longer the biggest source of IT spending in a typical enterprise. When I started at IBM the water cooled mainframes were a huge revenue source and the people who supported them were sort of living gods, especially the more senior ones.

I myself was more of a cable monkey at that time spending more time under the floor than above it, in those glory days of the late 80s and very early 90s, before HDS really started eating IBMs Mainframe Lunch and EMC started to eat their DASD (Disk) Desert.

I did however become a ‘real’ mainframe guy when the air cooled 9672s and 2064s blew HDS out of the compatible market and brought IBM marching back into quite a few of our old accounts with far smaller machines that one or two people could install quite comfortably.

Yet despite the (possibly unexpected) survival of the mainframe, what has truly gone is the almost completely bizarre world of mainframe I/O (Input/Output) devices and the vast number of service representatives who were needed to support them. So in the great tradition of IT War Stories, here are five things you will never get to do in a data centre:

You will never blow mainframe water into a toilet

Yes you read this right… not only will you never get to blow mainframe water into a toilet, I may never get to write that heading again.

To explain: A water cooled mainframe was not shipped with water in it. Apart from the obvious fact that shipping something with water in it is a challenge, these things weighed enough already without 400 litres of water sloshing around inside. So one of the many tasks done as part of installing such a beast was to “just add water”. This involved emptying 100s of litres of distilled water bottles, usually way bigger than this one:

This water would be pumped to chillers (normally on the roof) to take away the heat from the many many heat producing TCMs (Thermal Conductor Modules) that made the magic happen. Here is a terrible screen capture showing a TCM and some of the vast amount of plumbing hoses. What makes this photo ‘real’ is the fact that the gent is wearing an ESD (electrostatic discharge) wrist band.

Now when a water cooled mainframe was uninstalled, it could not be shipped with the water in it, which meant blowing the water out of the many pipes and hoses using a big old bottle of nitrogen, usually way bigger than this one:

But what to blow the water into?

Why the toilets of course!

Normally these were the grimy under-cleaned toilets just outside the computer hall.

We would hook up the hoses end to end and then with the hose aimed firmly into the bowl, until you eventually got down to just gas with spits of moisture. The disappointing thing is that while there are many marketing photos showing impossible scenes like the one below, none show anyone blowing mainframe water into a toilet. And no one had mobile phone cameras in those days (mainly because your current mobile phone has more of everything that is shown this photo, except the printers):

Now before you proudly inform me that the current IBM z15 has a water cooled option, just like in the ‘good old days’, the amount of water being used is comparatively tiny and the water is simply drained into a jug using a special tool like this. And seriously, this ‘tool’ looks like my Uncle Ted designed it.

You will never drag 100s and 100s of feet of dirty bus and tag cables around and under the floor.

A good computer room floor would give you two or three feet of clearance, but it was not unusual to get very shallow floors. The parallel bus and tag cables used from the 1960s to 1990s were seriously heavy and accumulated in great numbers under the floor. Sometimes when cables needed to be re-routed it was easier to drop the old cables into the floor (perhaps after beheading them) and just run new ones. Such sins could accumulate until the under-floor section became literally clogged with these cables, like weird veins of some great dead beast. In addition these cables seemed to accumulate filth. After twelve hours of dragging them through the floor you often found your hands and clothes filthy and blackened.

After removing around 2000kgs of cables from one data centre, we learned from our recycling agent that the wires themselves were aluminium and hard to recycle, but the connector pins (in those big fat connectors that just loved to snag on pretty well everything), were gold plated and if you got enough of them, you could actually make some real money.

I stole this image from Wikipedia, and the cable is clearly slightly grimy, an aspect I find oddly pleasing:

A dirty bus/tag cable

You will never get to ask for a 36 hour outage to install a new system

The thing is that mainframes were huge and hard to install and computer rooms were not always designed to let you install a new one alongside the old one. Even if you could, this would require a mountain of cabling to make it work. So for many customers, the installation schedule would look like this.

1. Sometime on Saturday morning, once the Friday night batch and backups were finished, team one would begin. They would power off and de-install the old mainframe and remove it from the floor. This could take many hours and many people.
2. Once this was done, team 2 would come in and re-cable the under floor region, especially if the layout had changed. This could also take many hours.
3. Once this was done, team 3 would assemble the new mainframe and commission it. This could (you guessed it), also take 12-15 hours.
4. Things would normally close out some times as late as Sunday evening with the debug team (team 4) fixing any issues so the client could be up and running by 8am on Monday morning.

This extended outage was quite normal, even for banking systems.

Tell that to dev-ops kids now-a-days and they won’t believe you.

You will never get to roll enormously heavy motor generators across the computer room floor

An IBM 3090 ran on 400 hertz power, meaning it took your 50/60 hertz power and then converted it to 400 hertz using a motor generator called an IBM 3089, which weighed 1075 Kg. Meanwhile the IBM 3097 that distributed power and cooling could weigh up to 1309 kg. The I/O frames meanwhile were a touch top heavy, so they had wings you would pivot out when rolling them across the floor to remove the risk of them toppling over.

You may never install a computer that comes with a desk.

Thats right, every IBM 3090 came with its own desk, officially known as the console table. It’s not unusual to visit a computer room that has not seen a water cooled mainframe for over 20 years, and yet those humble desks are still doing service, possibly because they weigh nearly 90 kilos and no one is strong enough to remove them (although the green screens that once sat on them are hopefully long gone).

A 3090 console table

Anyone still got an IBM Mainframe desk or been dragging some bus and tag cables? Let me know. Maybe take a photo with your smart phone.

Why the IBM SSIC is the best place to get your supported hardware information

Its been a while since I last blogged, but here is something that might interest you.  To spice things up I even recorded a quick video blog of the same information you will find written here.  So skip the video and keep reading, or watch the video and skip the reading, the choice is yours!

For many years I have used the ‘Supported Hardware List’ websites from IBM to qualify SVC support.    If you want to know if an Infinidat is supported behind IBM SVC, which version and what code level,  it’s all there.

So traditionally I would go to here.  From there you get a great list of code levels, choose your code level and then look up your product:

However I have always had some tiny misgivings about these sites.  After all, they obey no law of sorting I have ever seen.   Alphabetical order any one?   It’s like the Web Admin is a worshiper of Cthulhu and has managed to translate non-euclidian geometry into a list of vendor names.   Or maybe the site is just TOO HARD TO MAINTAIN.

Take a look and suggest a logic to this list of vendors (please I beg you):

But there is a bigger problem:  Theses sites are just slightly out of date.

Lets use that example I first raised,  if look here I find Infinidat version 2.0 is supported with SVC 7.8:

But if I then go to the SSIC here.   I get told version 3.0.x is also supported:

This was not a one-off, I found multiple products where the SSIC seemed to reflect newer information than the Supported  Hardware websites.

Moral of the story?   Always use the SSIC to confirm support, not the Supported Hardware pages.

Actifio at Tech Field Day 11

In this blog post I want to inform you about Actifio’s presentation to Tech Field Day 11.

Tech Field Days are one of my favourite technical information sources.    They involve a group of prominent bloggers and industry personalities being given a briefing and demonstration (normally somewhere between 2-4 hours) by an IT company about their products and viewpoint.   It is a chance for both IT entrants and established IT Companies to tell their story, explain the why, have their ideas challenged by some smart people and get some relatively free publicity, while the bloggers get the chance to gather material to write blogs, learn about our rapidly challenging world and sometimes show how clever and insightful they are at the same time.   It is a genuine win-win for everybody.

Actifio were last at Tech Field Day 4 in 2010, which explains why competitive information about Actifio is often so laughably wrong.  I think other Vendors watch these (in IT terms), ancient videos and presume nothing has changed since!    The good news for Actifios competitors and prospective and existing customers is they can now update their knowledge of Actifio by watching Actifio present at Tech Field 11 in 2016

The really nice thing is that the presentations have been split into five easily consumed videos, each about 20 minutes long. So please drop by the Tech Field Day page, take a look at the presented subjects and learn about Copy Data Management and how Actifio’s products bring a new and unique way for our customers to move to the hybrid cloud, dramatically improve their agility and modernise their business resiliency.

To make it easy, I have reposted all the Actifio video links below, but you can also get to them from here where you can also check out the other vendors who presented.

Actifio Welcome with Ash Ashutosh

Watch on Vimeo

Actifio CEO and Founder, Ash Ashutosh, introduces the company and its technology to the Field Day delegates.

Personnel: Ash Ashutosh

Actifio Architecture Overview

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, introduces the company’s product architecture.

Personnel: Brylan Achilles, Chandra Reddy

Actifio Resiliency Director Overview and Demo

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, introduces Actifio Resiliency Director.

Personnel: Brylan Achilles, Chandra Reddy

Actifio Global Manager Overview and SQL Server Demo

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, introduce Actifio Global Manager and demonstrate its use with Microsoft SQL Server.

Personnel: Brylan Achilles, Chandra Reddy

Actifio Global Manager Oracle and Ansible Demo

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, demonstrate Actifio Global Manager with Oracle and orchestration with Ansible.

Personnel: Brylan Achilles, Chandra Reddy

Actifio ReadyVault and Object Storage

Watch on Vimeo

Brylan Achilles and Chandra Reddy of Actifio, introduce Actifio ReadyVault and show how the product can work with object storage. Ash Ashutosh, CEO and Founder, then returns to answer questions.

 

Bits are cheap! Don’t sell yourself short on key length

Using SSH keys to perform password-free login is quite common in Unix hosts and in  Appliances that have embedded Unix (like Storwize products).

You effectively have a public key which is shared  and a private key (usually with a PPK extension) that is not shared.  Think of the public key like the lock in your front door, that  everyone can see.   Think of the private key like the door key in your pocket or hand-bag.   If you keep your private key secure, your door is relatively secure.  If you lose your keys, your door is most likely no longer secure (unless they are down the back of the couch).

Sticking with the door analogy, the risk with a door lock is that someone could still just try to kick your door in (brute force attack) or pick your lock.   The bit length of the key can make this harder to achieve: the longer the bit length the harder it is to crack.

It is not unusual to see instructions that suggest you use a command like this to generate keys, where a bit length of 1024 is specified with an RSA key:

ssh-keygen –b 1024 -t rsa -f ~/.ssh/id_rsa

Of if using PuTTYgen to create the keys, to see instructions like this:

1. Start PuTTYgen by clicking Start > Programs > PuTTY > PuTTYgen. The PuTTY Key Generator panel is displayed.
2. Click SSH-2 RSA as the type of key to generate.
   Note: Leave the number of bits in a generated key value at 1024.

The problem is that these instructions are all old.  In fact using the ssk-keygen command syntax example shown above would represent a down-grade in what is now the default setting.   The wiki and man pages for ssh-keygen both confirm that for RSA, the default length is now 2048 bits (not 1024 bits).

To confirm what key length you get by default, simply make a test key and then read it back.  In this example I create a new public/private key pair called testkey  without specifying a bit length (there is no -b 1024):

anthony$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/anthony/.ssh/id_rsa): testkey
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in testkey.
Your public key has been saved in testkey.pub.
The key fingerprint is:
SHA256:or3Yhykd0W569QcHtGk4ZMSdQDYlaM9ko+TiWQZ7pp4 anthony@Anthonys-Actifio-MacBook-Pro.local
The key's randomart image is:
+---[RSA 2048]----+
| =B+.. |
| . +.Bo+ |
| B O + o |
| + O = = |
| ..@S o . |
| o=.o . . . |
| .o.B . . o |
| .oE.o . . |
| ..oo . |
+----[SHA256]-----+

I then read the file back using the -l and -f params (specifying the name of the file) and confirm the bit length, which in this case is 2048 bits as highlighted by the red text:

anthony$ ssh-keygen -l -f testkey
2048 SHA256:or3Yhykd0W569QcHtGk4ZMSdQDYlaM9ko+TiWQZ7pp4 anthony@Anthonys-Actifio-MacBook-Pro.local (RSA)

When using PuTTYGen, if you use a recent version you will note that the default bit length is now 2048 (as indicated by the red circle).   If you load a key you should see the bit length of the loaded key as indicated by the orange circle.

So if you see instructions specifying the creation of a 1024 bit key, I suggest you ignore them and use 2048 bits or at the least question this with your vendor.   Equally if you are using older keys, it is well worth checking their bit length and generating new keys, since this will give you the now default bit length of 2048, but also renew them, reducing the risk of someone using an older (and potentially leaked) key inappropriately.