UPDATE 14 August 2015
When I initially posted this blog, there was a major error in my base spreadsheet, that made the time periods shorter than they actually were (because it was only using weekdays, 5 days per week, not 7 days per week, which was my mistake).
I withdrew the post and corrected it and this is an edited update. The conclusions remain pretty well the same, but the time periods are larger than initially stated.
A common question I get is this:
A new version of code has just been released for my SVC or Storwize product, should I upgrade or should I wait?
The challenge for many customers is that these upgrades:
- Need change windows
- Cannot be backed out
- Rely on redundancy to avoid downtime
- Take over an hour to complete
So when is the right point to get the most reliability and access to new features and hardware support, but with the least number of change windows? It occurred to me only recently that rather than rely on a bunch of potentially subjective or gut feel decision points, could I just use maths here? Go all Freakonomics on this subject.
Now it turns out IBM actually made this fairly easy as they publish the build dates for the entire release history right here:
They are usually in a format like 115.51.1507081154000 where the 1507081154000 can be read as 11:54 on the 8th of July 2015. Now I don’t work for IBM engineering, but my take is that if they publish a build date then I am fairly confident that this will be when the code was built from source. Normally it is then fed to QA and if it passes their sometimes real world tests (I am only being slightly sarcastic), it hits the field release process. So the build date is not when it was released, but when it was built.
So I took all of these dates and put them into a spreadsheet and then calculated time periods between builds (which I will call release dates, knowing they were NOT the actual date of release).
I considered three periods:
- The time period between major releases (i.e. days between between 7.3 and 7.4). I made an executive decision to treat releases 4.1.0 and 4.1.1 as major even though they are probably not. You will see why as we go through. This metric shows how often major releases come out.
- Days between updates within a release (for instance how many days between 220.127.116.11 and 18.104.22.168 compared to how many days between 22.214.171.124 and 126.96.36.199). This metric shows how often patches come out.
- Days between the build date of each update and the build date of its major release. For instance, days between 188.8.131.52 and 184.108.40.206 versus days between 220.127.116.11 and 18.104.22.168. This metric shows the patch release lifecycle of each release.
I then graphed each metric to get a visual impression of these. Lets check them out…
Time period between major releases
This one is interesting as the trend is clear, a new major release is coming out roughly every 180 days. This shows that a sixth monthly release cycle is definitely being worked to.
However there is a large glitch in the center, which makes you wonder whether there were some delays in certain releases. We know that Release 5.0 had 64 bit changes and Release 6.0 brought the Storwize platform into play, so that helps explain that spike.
Note that 22.214.171.124 doesn’t appear as there was no release before that!
Days between updates within a release
This one is quite interesting. If you go to a release, how many days will pass before another build hits the field? In other words, I just upgraded… how many days will pass before I may potentially have to upgrade again (presuming I am determined to always run the latest release).
The short answer is that the interval based on the trend line Excel added, shows it is becoming shorter as time goes by. It started at 70 days and is now closer to 45 days. However that’s using a linear trendline, a logarithmic trendline is much flatter and stays almost constantly on 50 days.
Days between updates from the build of that major release
So this is the most interesting graph of all. Once a major release comes out, a series of patch releases come out for that minor release. How quickly do they get released? If the first five updates come out in the first 60 days and then there is a 30 day gap, does that mean I should wait 90 days?
This endless series of hills shows the way release histories run. At the start of each cycle there are lots of releases, each one coming out on a slightly longer period than the previous one. Usually there is always a very late update, probably a roll-up for the slow upgraders (thus the sudden spike to the top of each hill). It also shows the effective lifespan of a code release is around 500-550 days as new updates simply don’t come out after a certain point. The more recent code levels taper off on those peaks, as they are not that old yet.
This leaves the one killer question… how long should I wait? I looked at just release 7.1 to 7.5 to see if I could work out just from the release cycle, when to jump. The red line shows the days between each release while the blue line shows the cumulative days within the release. From a user perspective, the higher the red line gets the better, as this means less code churn. So I look for the first major red peak and then find the matching point on the blue graph to see when that occurred.
I was looking for when the red line gets over 60 days and stays there, which typically occurs between 120 and 250 days after a release. I think waiting for 5 months is a very long interval, it really depends on how conservative you want to be. I do note that release 7.4 has the best correlation between the red and blue graphs, which is a very good sign.
So what can we learn from all of this?
I certainly learnt several things:
- Major release cycles are six monthly
- Patch updates taper off as the release ages and eventually stop after around 18 months
- Patch updates are coming out on average every 50 days
- It takes between 150 and 250 days before new patch release intervals start to slow down within each build
What I need to do next is look at the incidence of hyper releases (ones that have severe impact) to see if we can add any extra metrics to this study.
For reference, attached it the spreadsheet I built with the IBM release dates.