Checking for “Dead” paths on HBAs with PowerCLI

Posted: 27/08/2011 in PowerShell, Virtualisation

This week I have been tasked with assisting a storage engineer while he replaces a couple of Fibre Channel switches. My task was simple, check the HBAs for any “Dead” paths (See below screen shot for an example) across all the ESX hosts every hour for a weekend. We have a fair few VMHosts so this gave me the opportunity to use PowerCLI (Think PowerShell with a VMware snapin).

It didn’t take long at all to knock up a little script that can report on path states, this is what I came up with:

$VMHosts = Get-VMHost  | ? { $_.ConnectionState -eq "Connected" } | Sort-Object -Property Name
$results= @()

foreach ($VMHost in $VMHosts) {
Get-VMHostStorage -RescanAllHba -VMHost $VMHost | Out-Null
[ARRAY]$HBAs = $VMHost | Get-VMHostHba -Type "FibreChannel"

	foreach ($HBA in $HBAs) {
    $pathState = $HBA | Get-ScsiLun | Get-ScsiLunPath | Group-Object -Property state
    $pathStateActive = $pathState | ? { $_.Name -eq "Active"}
    $pathStateDead = $pathState | ? { $_.Name -eq "Dead"}
    $pathStateStandby = $pathState | ? { $_.Name -eq "Standby"}
    $results += "{0},{1},{2},{3},{4},{5}" -f $VMHost.Name, $HBA.Device, $VMHost.Parent, [INT]$pathStateActive.Count, [INT]$pathStateDead.Count, [INT]$pathStateStandby.Count
    }

}
ConvertFrom-Csv -Header "VMHost","HBA","Cluster","Active","Dead","Standby" -InputObject $results | Ft -AutoSize

The output is pretty simple but does the job:

VMHost                            HBA    Cluster        Active Dead Standby
------                            ---    -------        ------ ---- -------
d1-vmesx-001.*************.uk    vmhba2 Cluster05      40     0    24
d1-vmesx-001.*************.uk    vmhba3 Cluster05      40     0    24
d2-vmesx-001.****************.uk vmhba2 Cluster02      98     0    80
d2-vmesx-001.****************.uk vmhba3 Cluster02      96     0    80
d2-vmesx-011.****************.uk vmhba2 Cluster03      76     0    88
d2-vmesx-011.****************.uk vmhba3 Cluster03      75     0    85
d2-vmesx-012.****************.uk vmhba2 Cluster03      76     0    88
d2-vmesx-012.****************.uk vmhba3 Cluster03      75     0    85
d2-vmesx-013.****************.uk vmhba1 Cluster03      83     0    85
d2-vmesx-013.****************.uk vmhba2 Cluster03      83     0    85
d2-vmesx-014.****************.uk vmhba1 Cluster03      83     0    85
d2-vmesx-014.****************.uk vmhba2 Cluster03      83     0    85
d2-vmesx-015.****************.uk vmhba1 Cluster03      83     0    85
d2-vmesx-015.****************.uk vmhba2 Cluster03      83     0    85
d2-vmesx-016.****************.uk vmhba1 Cluster03      83     0    85
d2-vmesx-016.****************.uk vmhba2 Cluster03      83     0    85
d2-vmesx-017.****************.uk vmhba1 MSCS Cluster01 44     0    308
d2-vmesx-017.****************.uk vmhba2 MSCS Cluster01 44     0    308
d2-vmesx-018.****************.uk vmhba1 MSCS Cluster01 44     0    308
d2-vmesx-018.****************.uk vmhba2 MSCS Cluster01 44     0    308
d2-vmesx-019.****************.uk vmhba1 Cluster04      68     0    52
d2-vmesx-019.****************.uk vmhba2 Cluster04      68     0    52
d2-vmesx-021.****************.uk vmhba1 Cluster04      68     0    52
d2-vmesx-021.****************.uk vmhba2 Cluster04      68     0    52

If you wanted to work with this information further then I would suggest creating a custom object and populating the properties with Add-Member instead of doing the format string method above, I just did that as it’s quick. Well this is my first PowerCLI script, I hope to share some more.

Bye for now.

jfrmilner

Comments
  1. […] me a simple report of the status of each of the LUN paths to each of the HBAs on my hosts. I found John Milner’s post to be very helpful, and it gave me exactly the results that I wanted. However, it took forever to […]

  2. sam says:

    Hi, any chance i could reset / rescan-all that ‘dead’ link on a FibreChannel connection? on our dev-box here, when i tried the Get-VMhost $ServerVM | Get-VMHostStorage -RescanAllHba, i keep getting SoftwareIScsiEnabled = False

    • jfrmilner says:

      Sam,

      The SoftwareIScsiEnabled message is informational and you should see in the Windows GUI the “Rescan all HBAs” with a Completed message in resent tasks, you could also try Right-Clicking on one of your VMHosts and selecting “Rescan for Datastores”.

      Hope this helps.

      Jfrmilner.

  3. Matt K says:

    Hello,
    How can I run this script against a single ESX host? Say I have this:

    connect-viserver -Server “vcdev.ndcvc.com” -protocol https

    $VMHostname = Read-Host “Please enter a hostname”

    Can the script be run against JUST $VMHostname?

    Thanks!

    • Matt K says:

      One other thing, when I run the script I get this error:

      out-lineoutput : The object of type “Microsoft.PowerShell.Commands.Internal.For
      mat.FormatStartData” is not valid or not in the correct sequence. This is likel
      y caused by a user-specified “format-table” command which is conflicting with t
      he default formatting.
      + CategoryInfo : InvalidData: (:) [out-lineoutput], InvalidOperat
      ionException
      + FullyQualifiedErrorId : ConsoleLineOutputOutOfSequencePacket,Microsoft.P
      owerShell.Commands.OutLineOutputCommand
      PowerCLI C:\scripts>

      Any insight into that error? Is it possible to dump the output to a .htm file?

      Thanks!

      • jfrmilner says:

        I’m unable to replicate the error your getting but I suspect that in your effort to copy and paste the code into your script file you may have picked up some extra html formatting, try using the “copy to clipboard” or “view source” feature on the code block and try again.

        Dumping to html is possible with the ConvertTo-Html cmdlet, please check the PowerShell help documentation for further details.

        Regards,

        jfrmilner

    • jfrmilner says:

      Matt,

      Yes, the first line collects the server(s) in the $VMHosts variable and this can be a single host or a collection.

      Regards,

      jfrmilner

  4. […] me a simple report of the status of each of the LUN paths to each of the HBAs on my hosts. I found John Milner’s post to be very helpful, and it gave me exactly the results that I wanted. However, it took forever to […]

  5. wojcieh says:

    Really useful script. Thanks!

  6. StuartCz says:

    To run it against one host just add a hostname after the get-vmhost. The below format checks for a specific vendor, runs a bit faster (no rescans), and exports it to a gridview. It also should run cleanly as a script

    connect-viserver **myvcenter**
    $VMHosts = get-cluster **mycluster** | Get-VMHost | ? { $_.ConnectionState -eq “Connected” } | Sort-Object -Property Name
    $allresults= @()

    foreach ($VMHost in $VMHosts) {
    write-host “Working on host $VMHost”
    [ARRAY]$HBAs = $VMHost | Get-VMHostHba -Type “FibreChannel”

    foreach ($HBA in $HBAs) {
    $results = “” | select Name, Device, Cluster, ActiveP, DeadP, StandbyP
    $pathState = $HBA | Get-ScsiLun | ? { $_.Vendor -match “EMC”} | Get-ScsiLunPath | Group-Object -Property state
    $results.Name = $VMHost.Name
    $results.Device = $HBA.Device
    $results.Cluster = $VMHost.Parent
    $results.ActiveP = [INT]($pathState | ? { $_.Name -eq “Active”}).Count
    $results.DeadP = [INT]($pathState | ? { $_.Name -eq “Dead”}).Count
    $results.StandbyP = [INT]($pathState | ? { $_.Name -eq “Standby”}).Count
    $allresults += $results
    }

    }

    # Display the results in Gridview
    $allresults | out-gridview
    # Or export to a CSV
    # $allresults | export-csv “myfile”

    disconnect-viserver **myvcenter** -confirm:$false

  7. […] jfrmilner: Check for “Dead” paths on HBAs with PowerCLI […]

  8. kylecis says:

    This is really cool. Now if I can add in sending an email if there is a dead path, then run it on a schedule I will be set.

  9. aniket says:

    Hi Sam,

    thanks for script , but i had few queries as when we run the script and it scan all hba of host,After first scan it paths,target and devices , so all dead luns information get refreshed and gets active state so we are not able to get exact dead path information. can we remove parameter and execute
    Get-VMHostStorage -RescanAllHba -VMHost $VMHost | Out-Null

    with regards
    Aniket

    • jfrmilner says:

      I’m not sure who Sam is? Still, you can remove the line you mentioned and your results will come from the vCenter cache.

  10. Paul says:

    Hallo,

    First of all, thanks for the script! But when I use the script all my results are 0, Active, Dead and Standby. Any thoughts on that. Help is being appreciated!

    With regards,

    Paul

    • you may get a blank output if you have non FibreChannel hbas like isci etc .

      you may need to specify -Type “” as per your environment

      line 6 : [ARRAY]$HBAs = $VMHost | Get-VMHostHba -Type “FibreChannel”

Leave a comment