Your vendor hired some really smart people – but are you cutting them off?

At the recent Melbourne VMUG I ran into a fellow blogger who works for HDS.   Now I have known this person for some years and interact with him regularly, I have a lot of respect for him and for the work he does (after meeting him I have even more).   But here is the thing:   Up until that day, we had never met in person.

The reality is that in this day and age, actually meeting someone face to face is not needed for you to have a professional courteous and beneficial working relationship; in fact as the world becomes more digital and businesses become more global, this paradigm of “I know this person really well, trust and respect them”, can happily include  ”but I have never met this person face to face”.

The reason I bring this up is the never-ending struggle I have to get remote access enabled for the products that I support and sell.   When I worked at IBM this was always a challenge.   IBM is a huge company with enormous resources, but without remote access, the only way to leverage those resources when resolving an issue, was by offloading logs and waiting for them to be analysed.

The simple truth is that without either access to the device in question, or access to the correct logs from the device in question, whatever problem you are having is going to take much longer to fix.   And while that remote support person is waiting for your logs, they can work on someone else’s problem:  not yours.

Ouch.

I like to call this delay time and frankly when you add up all the hours it took to resolve an issue, delay time will be a major component.

So how to kill delay time?   The simple answer is remove the root cause:   Lack of remote access.   If you enable the remote support teams to actually access the source of your pain directly, you will inevitably get that issue resolved faster.   By how to do this?  Most vendors offer a remote access method, that inevitably forms one of four techniques:

  • Modems – yes those old faithful, slow and cumbersome squealers are still out there, but their days are numbered.  Modems are slow and the interface does not lend itself to anything too smart.   In the past, security concerns about modems related to fears about war-dialling hackers, but I am not certain most script kiddies even have modems anymore?   Of course having mentioned war-dialling, you cannot go past the classic film Wargames (apologies for the advertisement you may be forced to watch 5 seconds of).

  • Shared desktop – Things like GotoMeeting and Cisco Webex are examples of this.  This is my least favourite method as it usually demands the use of someones desktop/laptop in the client data center.
  • VPN in – This is a process where the vendor logs into your network using a tool like Citrix or Endpoint.     This is quite common (I use Citrix Access Gateway in particular on a regular basis), but often needs devices like key fobs or is tied to unique signature files (which can make it hard to share this access with team members).   For a vendor this is usually the easiest way to get remote access, where it is already a corporate approved method.
  • VPN out – this is a process where your product builds a tunnel outbound to a server at the vendors support HQ.   This form of private tunnel is controlled from the client side, but once built, allows the vendor to access the device from their remote location.  The client can then collapse the tunnel when the problem is resolved.   This is my favoured process as the process is always outbound and is normally easily controlled by the client, however equally it is often the hardest to get approval for.  This is quite simply because it may not match what is currently a corporate approved method, plus it may require firewall changes, which always raises eyebrows.

My main beef is that many organisations do not have any set strategy.   Ideally they don’t want to have any issues, which would allow them to avoid having to even think about this.   But in reality when bad things happen, you want the smartest most capable staff at your vendor plugged into and resolving your issue in real-time, not delay time.   Those people will almost certainly be in a different city or country.   You want their help, you need their help and you DON’T want them working on someone else’s problems while waiting for log files from yours.

So please… think about a remote support strategy and bring those smart people into your circle of trust.  You wont regret it.

 

Posted in advice | Tagged , , , , , | 4 Comments

Storwize V3700 – First impressions

The IBM Storwize V7000 has a new stable mate:  IBM’s hot new seller, the Storwize V3700.   I recently got a chance to try one out and I liked what I saw.  I have always tried to share useful information on this blog, so here are four things you may find  useful about IBM’s new little midrange storage offering:

Node Canisters

The Node Canisters (Controllers) are side by side and both right way up.  I really like this change.  Hopefully all future models will follow this pattern and avoid upside down components.  One thing you will spot from the picture is that the Fibre Cards are optional.  What you might think are Fibre Ports in this picture are actually SAS ports.  The fibre card goes where that large black square is on the right hand side of each canister.

rear v3700

Environmentals

The Storwize V3700 can report power consumption and operating temperature via both the GUI and CLI.   This is a great extra piece of information.

V3700 Environmentals

Being able to get this information via CLI is also critical as it allows you to script it for those shops where rack power consumption is constrained so check out the lsenclosurestats command.

IBM_2072:Cluster:anthonyv>lsenclosurestats 
enclosure_id stat_name stat_current stat_peak stat_peak_time 
1            power_w   124          125       130128230402 
1            temp_c    19           19        130128230707 
1            temp_f    66           66        130128230707

License Tab

I looked for the license tab…. but there isn’t one!  This is because Flashcopy is included, external virtualization ( as a migration tool) is included and remote copy is not possible.  This makes for very simple purchasing; all you need to do is decide what disks, RAM and adapters you want.  Nice!

I did find one (tiny) bug that is easily corrected, but is stealing 40 MB of your cache!   If you display the bitmap memory, you may find 20MB dedicated to remote copy, despite the fact that you cannot create remote copies.

IBM_2072:Cluster:anthonyv>lsiogrp 0
id 0
name io_grp0
node_count 2
vdisk_count 3
host_count 1
flash_copy_total_memory 20.0MB
flash_copy_free_memory 20.0MB
remote_copy_total_memory 20.0MB
remote_copy_free_memory 20.0MB
mirroring_total_memory 20.0MB
mirroring_free_memory 20.0MB
raid_total_memory 40.0MB
raid_free_memory 39.3MB
maintenance no
compression_active no
accessible_vdisk_count 3
compression_supported no

You can easily correct this by running the following command that drops that bitmap to zero.  You can run this command at any time, there is no risk in doing so.  You will get 40MB of cache back (20MB per node canister).

chiogrp -feature remote -size 0 io_grp0

WWPN Determination

I spotted two interesting things about the WWPNs for the Storwize V3700 ports.   Firstly IBM has broken with the 1,2,3,4 pattern we found with Storwize V7000 and gone to 04,08,0C,10.  Frankly this is not a big deal and given the Node Canisters are side by side, it is just a case of knowing the pattern.  The WWPN is based on: 50:05:07:68:03:YY:xx:xx  where xx:xx is unique for each node canister and the YY value is taken from the port position as per the image below.  I suspect these values may go up to 05, 09,0D,11 over time as they exhaust the serial number range possibilities of 00:00 to FF:FF

V3700 FC Ports

I did spot what I think is a great new command in V6.4.1 that also lets you display the WWPNs.  It is lsportfcid.  Try it out on your machine.

IBM_2072:Cluster_1:anthonyv>lsportfcid 
fc_io_port_id port_id type port_speed node_id node_name WWPN             nportid status                attachment
0  1             1       fc   8Gb        1       node1     5005076803040046 010500  active                switch    
1  2             2       fc   8Gb        1       node1     5005076803080046 010000  active                switch    
2  3             3       fc   N/A        1       node1     50050768030C0046 000000  inactive_unconfigured none      
3  4             4       fc   N/A        1       node1     5005076803100046 000000  inactive_unconfigured none      
6  1             1       fc   8Gb        2       node2     5005076803040047 010400  active                switch    
7  2             2       fc   8Gb        2       node2     5005076803080047 010100  active                switch    
8  3             3       fc   N/A        2       node2     50050768030C0047 000000  inactive_unconfigured none      
9  4             4       fc   N/A        2       node2     5005076803100047 000000  inactive_unconfigured none

I did spot one thing when displaying the FC ports in the GUI.  They are currently listed back to front,  just something to be aware of:

V3700 Ports

So are you running a V3700?  How is it working out for you?

Posted in IBM, IBM Storage, Storwize V3700, Storwize V7000, Uncategorized | Tagged , , , , | 20 Comments

If you dedup empty space, is it unbelievable?

My colleague Micah Waldman,  recently pointed out this blog post from Veeam: How to Get Unbelievable Deduplication Results with Windows Server 2012 and Veeam Backup & Replication! | Veeam Software …

This post leads you to a YouTube video that you can find here:

In this post, Veeam appear to get around 33 to 1 data reduction when doing backups, by reducing a backup of 240 GB of VMs down to 7.2 GB using a combination of Veeam and Windows dedup.   I was initially impressed with the result, but then noticed some curious things.

Firstly they started with 4 VMs, each with a 60GB volume (so a total of 240GB, something that gets mentioned several times).  These are the  provisioned sizes but since they used thin provisioning, the actual used space is a total of 50.3GB.  Here’s a screenshot of these numbers (the VMs are ausrv06 to 09).  I apologize for the truly awful quality – I genuinely tried to get a better image.   So the curious thing here is that rather than refer to the actual ‘thin’ data size, the video keeps referring to the ‘full fat’ size.

VM Size

They then protected these 4 VMs using Veeam. Veeam has volume/job level deduplication, which reduced the total storage for these VMs from 50.3 GB (not 240 GB since these are thin provisioned VMs) to 34.1 GB (15.1 GB plus 17 GB). Here are screenshots of the backup job results (there are two jobs each with 2 VMs):

BackupJob2 Job1

So Microsoft have come out with a new deduplication feature in Windows Server 2012 which in the example shown, Veeam are using as a post backup reduction tool.  They define a 100GB Windows Server 2012 volume and enable deduplication on it. This is deduplication of files on the volume that is done as post-processing for files that are older than X days. For applications it looks like a normal file system but underneath Windows Server stores files more efficiently, thereby allowing you to store more capacity than the physical volume size.

Using this new deduplicated volume as storage for a Veeam backup repository, Windows Server 2012 was able to store the 34.1GB of backup data in 7.2GB of disk space, a 5:1 reduction.

So what did we have here?

  • 240 GB of allocated VM storage.  This in reality is actually:
  • 50.3 GB of used VM data by VMware thin provisioning. This gets reduced to:
  • 34 GB of backup data by Veeam volume-based dedup. This gets reduced to:
  • 7 GB of storage space by Windows server 2012 dedup.

Overall, a 7:1 real dedup (from 50GB to 7GB), which is not too bad. But most of it came from Windows Server 2012.   In reality out of the 233GB saved (starting from 240GB down to 7GB), only less than 7% came from Veeam (16GB) while 81% came from VMware thin provisioning and close to 12% came from Microsoft!    The fact that Windows dedup was so effective even after Veeam’s dedup just goes to show how ineffective job-based 256KB block dedup really is (though I understand you can tune this by selecting the ‘right’ VMs to group together – which sounds like admin overhead).

One other technical side note: Veeam had to disable their compression and only use their dedup so as not to nullify the effect of Windows Server dedup, meaning they lose one space-saving technology to take advantage of another one.  I believe there is some tuning that can be done there too.

Posted in Uncategorized | Tagged , , , , , | 12 Comments

IBM Storage WWPN Determination Guide v6.6

For those who used my IBM Storage WWPN Determination Guide, there is finally an updated version (v6.6).    You can find it right here:

IBM Storage Systems WWPN Determination Version 6.6

New additions are the Storwize V3700 and the DS8870.

Posted in DS8800, IBM, IBM Storage, Storwize V7000 | 1 Comment

Three smart letters: CBT

When it comes to protecting your VMs (and their data), there are many methods available:

  • Installing backup agents
  • Copying files
  • Copying/cloning the entire VMDK
  • Snapshotting the entire datastore

At Actifio we have found the smartest way to protect a VM is to use VMwares own APIs to create an image of each VMDK, which allows for instant mount and therefore instant access to either the data within the VM, or to spin up an entirely new VM.   Even smarter, we keep that image of the VMDK updated using changed block tracking, meaning the updates can be captured very quickly with very little data movement (while using our own snapshot and dedup technology to give you SLA managed rollback points).

When Actifio starts protecting a VM, the first thing it does is turn on Changed Block Tracking (or CBT).  You can see this in the vCenter event log where the message ‘Reconfiguring Virtual Machine’ appears.

However this only works if your VM is using Hardware Version 7 or above.   It is not uncommon to find VMs still running Hardware Version 4.   It is a bit like having a  laptop/PC/Server with an older motherboard, but in this case a virtual motherboard.   We need that higher hardware version because without it, every time we protect the VMDK, we will need to move all the blocks that make up the VMDK, not just the blocks that changed since the last time we created a snapshot.

My colleague Jeff O’Connor wrote up some great information on Hardware Version upgrades which I thought I would share with you:

Virtual hardware can be thought of as a virtual motherboard, with certain capabilities and restrictions. Newer motherboards are capable of using things like newer and more CPUs or memory.

Hardware version 4 was available from ESX 3.x and above (prior to vSphere branding). With vSphere 4 hardware version 7  was released (going from hardware version 4 to 7 in one release!).  However VMs running hardware version 4 are also backwards compatible with vSphere 4.  To update a VM to hardware version 7, the customer must be running vSphere 4.x or above.

The conversion is fairly easy but there is about a very small chance the VM might not boot after the upgrade (usually if the VM came from a P2V conversion with SCSI drivers), which means you should always perform a VM snapshot prior to the upgrade (so you can recover it).   Your VM may also lose its IP addresses, as the virtual ethernet card may appear to change after the upgrade.

A good practice is to do the following in this order:

  1. Take a note of your VMs IP addresses and WINs setup (if  WINS is in use).
  2. Upgrade VMTools to the latest version on the VM.  This will normally require a reboot.
  3. Shutdown the VM (the VM has to be shutdown to upgrade the virtual hardware version — again think of it like a motherboard replacement).
  4. Create a VMware snapshot from vCenter (for purposes of rollback).
  5. Convert the VM to HW v7 (or higher) by right selecting the VM and selecting upgrade hardware version.  Confirm you want to proceed.
  6. Power on the VM, login and check the IP address settings are fine. For Windows machines the WINS addresses are generally dropped if the customer is using static WINS Server entries, so re-enter these if they are required.
  7. Restart the VM (this step is important as the hardware upgrade is the same process as a motherboard upgrade) and upon first boot new hardware is often found and plug and play kicks in.
  8. Once the VM is restarted, login again, check all services are started upon boot and IP addressing is still in place.
  9. If everything looks OK, you can then delete the VM snapshot from vCenter, which can be done online without requiring the VM to be shutdown.

You’re VM is then running hardware version 7/8/or 9 or whatever is the highest version available. There are other methods to do this, and some people skip the step of taking the snapshot, but the above method offers rollback. Even if you are on vSphere 4.x and have not upgraded the virtual hardware version, there are performance benefits for doing this on top of changed block tracking benefits.

Once you are comfortable, you could script the whole process with PowerCLI, especially if you have 100s or 1000s of VMs, snapshotting either all your VMs, or just a limited subset (the really important ones).

Below are the hardware versions which came out with the release mentioned.

HW v4 = ESX/ESXi 3.0 and above

HW v7 = ESX/ESXi 4.0 and above

HW v8 = ESXi 5.0 and above

HW v9 = ESXi 5.1 and above.

Don’t think that HW v4 means vSphere 4, this is not the case.

Posted in Uncategorized | Leave a comment

Cat videos: Being inspired by the kitten

January 2013 brought us the amusing story of a man in California who outsourced his job to a Chinese company, allowing him to slack off  and spend time watching cat videos.   I personally haven’t spent a lot of time on cat videos, but I have discovering the joy of making them since we got a new kitten.  Here she is two days after she came home:

What struck me about our new kitten is that it has one huge advantage over me:   She was  born with a self-training program.   What looks to us like youthful hijinks is actually the cat teaching itself to hunt and kill small animals.   It is learning all about how to use its body, how to judge things like distance and speed, all the time following a workout program that would make any personal trainer very proud.

So what can we learn from the kitten?   Well while some of us sit and watch cat videos, that cat is turning itself into a fit and agile hunter, ready to succeed in the world.   It is born with this program already set in its head, while we need to motivate ourselves to do similar things:  get ourselves fit and improve our brains.

So if you’re looking for a great kick-start to 2013, how about starting with a VMware User Group User Conference?   Check out the Melbourne and Sydney agendas, there is a truly great lineup including Stephen Foskett!

Oh and welcome to 2013,  hopefully it will be a great year for all of us.

Posted in advice | Tagged , , | 2 Comments

“SORRY, OUR SYSTEMS ARE DOWN…” GRRR! AS A CUSTOMER DO YOU REALLY CARE?

Reblogged from Advanced Information Technology for Productivity:

Click to visit the original post
  • Click to visit the original post
  • Click to visit the original post
  • Click to visit the original post
  • Click to visit the original post
  • Click to visit the original post

All of us at one point or another have experienced the terrible ramifications and loss of productivity when a business critical system fails. It might be that we’re at the airport trying to check-in for a flight that’s taking us to a critically important meeting only to be confronted by a terminal with thousands of people lined up while airline staff struggle to handle hundreds of flights via a manual process (nightmare stuff!) Or perhaps we've gone to get money from our bank only to find one after another ATM closed and the Internet banking site down with the media reporting our bank is having an IT system failure and urgently working on restoring the facilities.

Read more… 1,623 more words

Great new blog post by Philip Dartnell from Linktechsolutions, one of Actifio's Business Partners here in Australia.
Posted in Uncategorized | 2 Comments