Orphaned Snapshots in VMWare

Considering that the snapshot are not listed in VMWare snapshot manager and presuming that already we’ve tried to create a new snapshot ant then delete all and nothing is happening, in order to remove the orphaned snapshots we need:

  • First step is to find all snapshots and delta disks on the datastores: from SSH on the host
    find /vmfs/volumes/ -name *-delta*;find /vmfs/volumes/ -name *-0000*

    (if you know the VM this is not required)

  • This will list the vmdk’s which  are mounted by VMX file for the affected VM
     cat /vmfs/volumes/[id]/[VM]/VM.vmx | grep vmdk

    (this should be checked also because we don’t know if all disks which resides in the VM folder are used)

  • If you use cat on this vmdk files you should see what underlying file on the datastore they reference under  Extent Description and  Change tracking file
    cat VM_X.vmdk

    Then you should check if the vmkd file are locked by any hosts –delta , -ctk and –flat

    vmkfstools -D VM-000002-delta.vmdk

    (If it was locked by an ESXi host the MAC of the host would be shown in the owner)

  • Then make sure they are not locked by a process, touch them and see that the timestamp updates:
    touch /vmfs/volumes/[id]/[VM]/*-00000* | ls -ltr | grep vmdk

    Being able to touch the file, run vmkfstools -D finding no locks, find no references in vmdk descriptor files generally means it isn’t in active use and is safe to move/remove. (Please be aware this command should be used from other host, not from the one where VM resides )

  • Create a new directory and move the suspect files to it and check for problems with the VM
mkdir toberemoved 
mv *-00000* toberemoved/.
  • Check the VM state and if it is operating normally delete the directory
rm –r toberemoved/

 

Netstat and Powershell

 

The correct name should be: how to trace connections and monitor them using PowerShell. The wide used tool to see open connection in and out from a computer is through netstat command from cmd promt. Of course we can use Wireshark to do deep troubleshooting.

How about capturing the output and store them in an array so we can automate the process and do periodic checks to see if we have any errors. Usually the output will display listening and established states but something due to network bottlenecks and/or applications hanging we may get other states.

The script below is capturing the netstat -ano output and split it so we can store it in an array and use it after. This function has few parameters:

.PARAMETER ProcessName
	Gets connections by the name of the process. The default value is '*'.
	
.PARAMETER Port
	The port number of the local computer or remote computer. The default value is '*'.

.PARAMETER Address
	Gets connections by the IP address of the connection, local or remote. Wildcard is supported. The default value is '*'.

.PARAMETER Protocol
	The name of the protocol (TCP or UDP). The default value is '*' (all)
	
.PARAMETER State
	Indicates the state of a TCP connection. The possible states are as follows:
		
	Closed	 	- The TCP connection is closed. 
	CloseWait 	- The local endpoint of the TCP connection is waiting for a connection termination request from the local user. 
	Closing 	- The local endpoint of the TCP connection is waiting for an acknowledgement of the connection termination request sent previously. 
	DeleteTcb 	- The transmission control buffer (TCB) for the TCP connection is being deleted. 
	Established - The TCP handshake is complete. The connection has been established and data can be sent. 
	FinWait1 	- The local endpoint of the TCP connection is waiting for a connection termination request from the remote endpoint or for an acknowledgement of the connection termination request sent previously. 
	FinWait2 	- The local endpoint of the TCP connection is waiting for a connection termination request from the remote endpoint. 
	LastAck 	- The local endpoint of the TCP connection is waiting for the final acknowledgement of the connection termination request sent previously. 
	Listen	 	- The local endpoint of the TCP connection is listening for a connection request from any remote endpoint. 
	SynReceived - The local endpoint of the TCP connection has sent and received a connection request and is waiting for an acknowledgment. 
	SynSent 	- The local endpoint of the TCP connection has sent the remote endpoint a segment header with the synchronize (SYN) control bit set and is waiting for a matching connection request. 
	TimeWait	- The local endpoint of the TCP connection is waiting for enough time to pass to ensure that the remote endpoint received the acknowledgement of its connection termination request. 
	Unknown		- The TCP connection state is unknown.
	
	Values are based on the TcpState Enumeration:
	http://msdn.microsoft.com/en-us/library/system.net.networkinformation.tcpstate%28VS.85%29.aspx

All the above parameters are not mandatory so we can execute this function with or without them.

function Get-NetworkStatistics
{
	[OutputType('System.Management.Automation.PSObject')]
	[CmdletBinding(DefaultParameterSetName='name')]
	
	param(
		[Parameter(Position=0,ValueFromPipeline=$true,ParameterSetName='port')]
		[System.Int32]$Port,
		
		[Parameter(Position=0,ValueFromPipeline=$true,ParameterSetName='name')]
		[System.String]$ProcessName='*',
		
		[Parameter(Position=0,ValueFromPipeline=$true,ParameterSetName='address')]
		[System.String]$Address='*',		
		
		[Parameter()]
		[ValidateSet('*','tcp','udp')]
		[System.String]$Protocol='*',

		[Parameter()]
		[ValidateSet('*','Closed','CloseWait','Closing','DeleteTcb','Established','FinWait1','FinWait2','LastAck','Listen','SynReceived','SynSent','TimeWait','Unknown')]
		[System.String]$State='*'
		
	)
    
	begin
	{
		$properties = 'Protocol','LocalAddress','LocalPort'
    		$properties += 'RemoteAddress','RemotePort','State','ProcessName','PID'
	}
	
	process
	{
	    netstat -ano | Select-String -Pattern '\s+(TCP|UDP)' | ForEach-Object {

	        $item = $_.line.split(' ',[System.StringSplitOptions]::RemoveEmptyEntries)

	        if($item[1] -notmatch '^\[::')
	        {           
	            if (($la = $item[1] -as [ipaddress]).AddressFamily -eq 'InterNetworkV6')
	            {
	               $localAddress = $la.IPAddressToString
	               $localPort = $item[1].split('\]:')[-1]
	            }
	            else
	            {
	                $localAddress = $item[1].split(':')[0]
	                $localPort = $item[1].split(':')[-1]
	            } 

	            if (($ra = $item[2] -as [ipaddress]).AddressFamily -eq 'InterNetworkV6')
	            {
	               $remoteAddress = $ra.IPAddressToString
	               $remotePort = $item[2].split('\]:')[-1]
	            }
	            else
	            {
	               $remoteAddress = $item[2].split(':')[0]
	               $remotePort = $item[2].split(':')[-1]
	            } 
				
				$procId = $item[-1]
				$procName = (Get-Process -Id $item[-1] -ErrorAction SilentlyContinue).Name
				$proto = $item[0]
				$status = if($item[0] -eq 'tcp') {$item[3]} else {$null}				
				
				
				$pso = New-Object -TypeName PSObject -Property @{
					PID = $procId
					ProcessName = $procName
					Protocol = $proto
					LocalAddress = $localAddress
					LocalPort = $localPort
					RemoteAddress =$remoteAddress
					RemotePort = $remotePort
					State = $status
				} | Select-Object -Property $properties								


				if($PSCmdlet.ParameterSetName -eq 'port')
				{
					if($pso.RemotePort -like $Port -or $pso.LocalPort -like $Port)
					{
					    if($pso.Protocol -like $Protocol -and $pso.State -like $State)
						{
							$pso
						}
					}
				}

				if($PSCmdlet.ParameterSetName -eq 'address')
				{
					if($pso.RemoteAddress -like $Address -or $pso.LocalAddress -like $Address)
					{
					    if($pso.Protocol -like $Protocol -and $pso.State -like $State)
						{
							$pso
						}
					}
				}
				
				if($PSCmdlet.ParameterSetName -eq 'name')
				{		
					if($pso.ProcessName -like $ProcessName)
					{
						if($pso.Protocol -like $Protocol -and $pso.State -like $State)
						{
					   		$pso
						}
					}
				}
	        }
	    }
	}
<#

.SYNOPSIS
	Displays the current TCP/IP connections.

.DESCRIPTION
	Displays active TCP connections and includes the process ID (PID) and Name for each connection.
	If the port is not yet established, the port number is shown as an asterisk (*).	
	
.PARAMETER ProcessName
	Gets connections by the name of the process. The default value is '*'.
	
.PARAMETER Port
	The port number of the local computer or remote computer. The default value is '*'.

.PARAMETER Address
	Gets connections by the IP address of the connection, local or remote. Wildcard is supported. The default value is '*'.

.PARAMETER Protocol
	The name of the protocol (TCP or UDP). The default value is '*' (all)
	
.PARAMETER State
	Indicates the state of a TCP connection. The possible states are as follows:
		
	Closed	 	- The TCP connection is closed. 
	CloseWait 	- The local endpoint of the TCP connection is waiting for a connection termination request from the local user. 
	Closing 	- The local endpoint of the TCP connection is waiting for an acknowledgement of the connection termination request sent previously. 
	DeleteTcb 	- The transmission control buffer (TCB) for the TCP connection is being deleted. 
	Established - The TCP handshake is complete. The connection has been established and data can be sent. 
	FinWait1 	- The local endpoint of the TCP connection is waiting for a connection termination request from the remote endpoint or for an acknowledgement of the connection termination request sent previously. 
	FinWait2 	- The local endpoint of the TCP connection is waiting for a connection termination request from the remote endpoint. 
	LastAck 	- The local endpoint of the TCP connection is waiting for the final acknowledgement of the connection termination request sent previously. 
	Listen	 	- The local endpoint of the TCP connection is listening for a connection request from any remote endpoint. 
	SynReceived - The local endpoint of the TCP connection has sent and received a connection request and is waiting for an acknowledgment. 
	SynSent 	- The local endpoint of the TCP connection has sent the remote endpoint a segment header with the synchronize (SYN) control bit set and is waiting for a matching connection request. 
	TimeWait	- The local endpoint of the TCP connection is waiting for enough time to pass to ensure that the remote endpoint received the acknowledgement of its connection termination request. 
	Unknown		- The TCP connection state is unknown.
	
	Values are based on the TcpState Enumeration:
	http://msdn.microsoft.com/en-us/library/system.net.networkinformation.tcpstate%28VS.85%29.aspx

.EXAMPLE
	Get-NetworkStatistics

.EXAMPLE
	Get-NetworkStatistics iexplore

.EXAMPLE
	Get-NetworkStatistics -ProcessName md* -Protocol tcp 

.EXAMPLE
	Get-NetworkStatistics -Address 192* -State LISTENING 

.EXAMPLE
	Get-NetworkStatistics -State LISTENING -Protocol tcp

.OUTPUTS
	System.Management.Automation.PSObject
#>	
}

The output will look like:

PS C:\Users\Administrator> Get-NetworkStatistics | ft *

Protocol        LocalAddress    LocalPort       RemoteAddress  RemotePort     State          ProcessName    PID           
--------        ------------    ---------       -------------  ----------     -----          -----------    ---           
TCP             0.0.0.0         135             0.0.0.0        0              LISTENING      svchost        656           
TCP             0.0.0.0         445             0.0.0.0        0              LISTENING      System         4             
TCP             0.0.0.0         1025            0.0.0.0        0              LISTENING      wininit        476           
TCP             0.0.0.0         1026            0.0.0.0        0              LISTENING      lsass          544           
TCP             0.0.0.0         1027            0.0.0.0        0              LISTENING      svchost        784           
TCP             0.0.0.0         1028            0.0.0.0        0              LISTENING      svchost        820           
TCP             0.0.0.0         1029            0.0.0.0        0              LISTENING      spoolsv        940           
TCP             0.0.0.0         1030            0.0.0.0        0              LISTENING      services       536           
TCP             0.0.0.0         3389            0.0.0.0        0              LISTENING      svchost        1576          
TCP             0.0.0.0         5985            0.0.0.0        0              LISTENING      System         4             
TCP             0.0.0.0         47001           0.0.0.0        0              LISTENING      System         4             
TCP             10.0.0.14       139             0.0.0.0        0              LISTENING      System         4             
TCP             192.168.1.159   139             0.0.0.0        0              LISTENING      System         4             
TCP             192.168.1.159   1389            189.99.255.140 443            ESTABLISHED    MySecureAPP    2724          
TCP             192.168.1.159   3389            10.121.8.79    54785          ESTABLISHED    svchost        1576          
UDP             0.0.0.0         3389            *              *                             svchost        1576          
UDP             0.0.0.0         5355            *              *                             svchost        952           
UDP             10.0.0.14       137             *              *                             System         4             
UDP             10.0.0.14       138             *              *                             System         4             
UDP             192.168.1.159   137             *              *                             System         4             
UDP             192.168.1.159   138             *              *                             System         4             

We can also use parameters:

PS C:\Users\Administrator> Get-NetworkStatistics  -ProcessName svchost| ft *

Protocol        LocalAddress    LocalPort       RemoteAddress  RemotePort     State          ProcessName    PID           
--------        ------------    ---------       -------------  ----------     -----          -----------    ---           
TCP             0.0.0.0         135             0.0.0.0        0              LISTENING      svchost        656           
TCP             0.0.0.0         1027            0.0.0.0        0              LISTENING      svchost        784           
TCP             0.0.0.0         1028            0.0.0.0        0              LISTENING      svchost        820           
TCP             0.0.0.0         3389            0.0.0.0        0              LISTENING      svchost        1576          
TCP             192.168.1.159   3389            10.255.2.79    54785          ESTABLISHED    svchost        1576          
UDP             0.0.0.0         3389            *              *                             svchost        1576          
UDP             0.0.0.0         5355            *              *                             svchost        952           
UDP             fe80::2066:4... 546             *              *                             svchost        784

The above function will be useful in case we want to trace broken connections or we want to see when our computer is generating request on web or trace specific protocols. I’m usual using this to monitor open ports and listen for incoming connection when debugging some applications.

Active Directory health check – DCDIAG friendly view

We can monitor all Domain Controllers events using different monitoring tools like System Center Operation Manager or any other tools from any other 3rd parties. But how about automating a health check of a domain controller and instead of having a text document with all the information inside where we need to look after  errors and warning we can create a user friendly table easy to read and interpret. We just need somehow to run a DCDIAG and then parse the output using Regex and display it.

DCDIAG is as an end-user reporting program, a command-line tool that encapsulates detailed knowledge of how to identify abnormal behavior in the system. Dcdiag displays command output at the command prompt.

What we need now is to create a Powershell function that takes the output from dcdiag, analyze the output and then present it in a easy readable format.

The function will be:

Function Test-ADInfrastructure {
    
	Param (
		[Parameter(Mandatory=$true)]
        $domain,                                       # Domain

        [Parameter(Mandatory=$false)]
        [string]$namingconvention = "*",                # Server naming convention

        [Parameter(Mandatory=$false)]
        $tmpFile = [System.Io.Path]::GetTempFileName(),  # temp file
        
        [String]$OutputFile
	)
	
    # run dcdiag and get the output

    dcdiag.exe /e /n:$domain | Out-File $tmpFile

    $dcDiagOutput = Get-Content $tmpFile
    Copy-Item $tmpFile $outputFile
    Remove-Item $tmpFile


    $regex = "\.{25}\s(.*?)\s(passed|failed)\stest\s(.*?)$"

    # Obtain am array with elements
    # of type @{Server, Test, Passed}
    $testResults = @()
    $dcDiagOutput | Foreach-Object {

    	if ( $_ -match $regex) {
    		#$matches
    		
    		$testResults += New-Object PSObject -Property @{
    			'Server' = $matches[1] -as [String];
    			'Test' = $matches[3] -as [String];
    			'Passed' = ($matches[2] -eq 'passed')
    		} | Select Server,Test,Passed
    	}
    }
    # group elements by Server
    $dcDiagTmp = $testResults | Where { $_.Server -like $namingconvention } | Group-Object -Property Server -AsHashTable


    # return elements of type
    # @{Server, TestName1, TestName2, FrsEvent, ... , VerifyReferences}
    #
    # e.g.
    #
    #   Server             : SRV-DC-01
    #   Connectivity       : True
    #   Advertising        : True
    #   FrsEvent           : True
    #   DFSREvent          : True
    #   SysVolCheck        : True
    #   KccEvent           : True
    #   KnowsOfRoleHolders : True
    #   MachineAccount     : True
    #   NCSecDesc          : True
    #   NetLogons          : True
    #   ObjectsReplicated  : True
    #   Replications       : True
    #   RidManager         : True
    #   Services           : False
    #   SystemLog          : True
    #   VerifyReferences   : True
    #
    foreach ($server in $dcDiagTmp.Keys) {
    	
    	$tmp = New-Object PSObject -Property @{'Server' = $server} 

    	$dcDiagTmp.$($server) | Foreach-Object {
    		$tmp | Add-Member NoteProperty -Name $($_.Test) -Value $($_.Passed)
    	}
    	
    	Write-Output $tmp
    }
}

There is just one mandatory parameter which is -domain. Also this function has other non-mandatory parameters like: -namingconvention and -tmpfile. Domain was set as mandatory parameter in order to be able to run this function on multiple domains in one go. We can set it as Mandatory=$false and replace the null value with $env:userdomain. This variable will pick up the domain from where you run the command.

In order to have a easy readable output I’ve used a regex syntax, Regex stands for regular expression.

$regex = "\.{25}\s(.*?)\s(passed|failed)\stest\s(.*?)$"

All about Regular expresion can be found here:https://en.wikipedia.org/wiki/Regular_expression

For all the servers we will get an output like:

# return elements of type
    # @{Server, TestName1, TestName2, FrsEvent, ... , VerifyReferences}
    #
    # e.g.
    #
    #   Server             : SRV-DC-01
    #   Connectivity       : True
    #   Advertising        : True
    #   FrsEvent           : True
    #   DFSREvent          : True
    #   SysVolCheck        : True
    #   KccEvent           : True
    #   KnowsOfRoleHolders : True
    #   MachineAccount     : True
    #   NCSecDesc          : True
    #   NetLogons          : True
    #   ObjectsReplicated  : True
    #   Replications       : True
    #   RidManager         : True
    #   Services           : False
    #   SystemLog          : True
    #   VerifyReferences   : True
    #

Based on the above output we can create a table and read it easily. Also we can colour the output so in case some test failed we can spot it quickly.

dcdiag2

Powershell & C# Notifyicon

Some time ago I’ve managed to create a tool to show me in real time when a new high priority incident is being logged and when one of the priority incidents was closed. In the background the tool was querying Service Now and extracting some XML file which then were stored in memory and displayed in a grid. To make some room on my desktop, the tool was configured to run in the background and display small notification balloons in the corned with what’s changed in the database.

Based on the above, I’ve tried replicating the same in Powershell. This may help when running time consuming scripts and  when we want periodic notification regarding the status of the script.

In C# the code is pretty simple:

notifyIcon1 = new NotifyIcon()


if (notifyIcon1.Visible != true)
{
   notifyIcon1.Visible = true;
}                                                                 // balloon made visibil
notifyIcon1.BalloonTipText = StringNotification;                  // set text
notifyIcon1.BalloonTipTitle = Convert.ToString("NEW: " + newIncidentList.Count + " || CLOSED: " + oldIncidentList.Count);
notifyIcon1.ShowBalloonTip(60000);                                 // How long

The result:

CsharpCapturePNG

 

In PowerShell in order to use a Notifyicon we will need to create an object type System.Windows.Forms.NotifyIcon. After that, the code is almost as the C# one.

The full function will be:

function Show-BalloonTip  
{
 
  [CmdletBinding(SupportsShouldProcess = $true)]
  param
  (
    [Parameter(Mandatory=$true)]
    $Text,
   
    [Parameter(Mandatory=$true)]
    $Title,
   
    [ValidateSet('None', 'Info', 'Warning', 'Error')]
    $Icon = 'Info',
    $Timeout = 10000
  )
 
  Add-Type -AssemblyName System.Windows.Forms

  if ($script:balloon -eq $null)
  {
    $script:balloon = New-Object System.Windows.Forms.NotifyIcon
  }

  $path                    = Get-Process -id $pid | Select-Object -ExpandProperty Path
  $balloon.Icon            = [System.Drawing.Icon]::ExtractAssociatedIcon($path)
  $balloon.BalloonTipIcon  = $Icon
  $balloon.BalloonTipText  = $Text
  $balloon.BalloonTipTitle = $Title
  $balloon.Visible         = $true

  $balloon.ShowBalloonTip($Timeout)
}

We can then use this function in a time consuming script and when we need something to be displayed we will append this function. The mandatory parameters are -text and -title

Show-BalloonTip -text "Some text here" -title "Powershell..... 64.... char"

The output:

notifyIcon

Events – Parsing the Event Viewer

The plan for today was to speak about Regular expression and how we can use regex in scripts, but I’ve been diverted to look at some logins and how can I track who’s connecting to some servers and from where.

The output:

07:02:06
=================================
VERBOSE: Connecting to computer: tst-server1
VERBOSE: ..WMI status: OK
VERBOSE: ...Found 168 events.
VERBOSE: ....Logon:         86
VERBOSE: ....Logoff:        82
07:02:24
=========================================
SELECT * FROM Win32_NTLogEvent WHERE (LogFile = 'Security') AND (EventIdentifier = 4634 OR EventIdentifier = 4624) 
AND (Message LIKE '%super-admin%')

ComputerName EventType LogonID    Timestamp             Account     SourceIP      
------------ --------- -------    ---------             -------     --------      
tst-server1  Logon     0xc4542909 4/22/2016 6:31:33 AM  super-admin 192.168.177.9 
tst-server1  Logon     0xc453df73 4/22/2016 6:31:31 AM  super-admin -             
tst-server1  Logon     0xc453a794 4/22/2016 6:31:29 AM  super-admin -             
tst-server1  Logon     0xa6785b38 4/21/2016 1:11:05 PM  super-admin 172.16.101.16 
tst-server1  Logon     0xa66d99e4 4/21/2016 1:09:24 PM  super-admin 172.16.101.16 
tst-server1  Logon     0xa2e04ec2 4/21/2016 11:16:05 AM super-admin 172.16.101.16 
tst-server1  Logon     0x9ee9524c 4/21/2016 9:27:47 AM  super-admin 192.168.177.9 
tst-server1  Logon     0x9ee8e0f3 4/21/2016 9:27:44 AM  super-admin -             
tst-server1  Logon     0x9ee86f5d 4/21/2016 9:27:41 AM  super-admin -             
tst-server1  Logon     0x784c9158 4/20/2016 11:16:46 AM super-admin 172.16.110.154
tst-server1  Logon     0x753a3945 4/20/2016 9:06:45 AM  super-admin 192.168.177.9

The script that was created to get the above data is pretty simple.  The parameters set for this are: computername which is not mandatory as we can run it locally on a server and the username we are after as a mandatory parameter.

Function Get-LogOnOffEvents
{

    [CmdletBinding()]
    
    Param (
        [Parameter(Mandatory=$false)]
        [String]$ComputerName = $Env:ComputerName,

        [Parameter(Mandatory=$true)]
        [String]$userAccount = $Env:username
    )

… then we get the data by using a select in the NTLogEvent table through a WQL query. Instead of parsing the Event viewer using gwmi -class Win32_NTlogEvent we are searching directly in the namespace table as the query is around 8 times faster that regular query using Get-WMIObject -class.. 

$query =    "SELECT * FROM Win32_NTLogEvent "
$query +=   "WHERE (LogFile = 'Security') "
$query +=   "AND (EventIdentifier = 4634 OR EventIdentifier = 4624) "
$query +=   "AND (Message LIKE '%$userAccount%')"

We can query any type of log files and get any type of information from the log file. We just need to know what we are after.

Depending if there is a firewall between the computer where we run the query and the remote one, the script can run successful or fail. This may be due to a WMI job that is hanged or not feeding back any information. As all knows, WMI query doesn’t have a time out in place so we need to define one.

The way I’ve done it here is by using the query as a job and setting a time windows until the job should return some output. If the Job state will be timeout or failed then the next query will not be initiated.

The full function will be:

Function Get-LogOnOffEvents
{

    [CmdletBinding()]
    
    Param (
        [Parameter(Mandatory=$false)]
        [String]$ComputerName = $Env:ComputerName,

        [Parameter(Mandatory=$true)]
        [String]$userAccount = $Env:username
    )
    $results = @()


    $query =    "SELECT * FROM Win32_NTLogEvent "
    $query +=   "WHERE (LogFile = 'Security') "
    $query +=   "AND (EventIdentifier = 4634 OR EventIdentifier = 4624) "
    $query +=   "AND (Message LIKE '%$userAccount%')"

    $query
    Write-Host '================================='

    Write-Verbose ("Connecting to computer: {0}" -f $ComputerName)
    
    # WMI related SYSTEM variables
    $wmiStatus = $null
    $wmiJob = $null
    $wmiResult = $null
    
    # Init the WMI query AS A JOB
    $wmiJob = Get-WmiObject -Query $query -ComputerName $ComputerName -AsJob
    
    $wmiJob | Wait-Job -Timeout 600 | Out-Null
    
    if ($wmiJob.JobStateInfo.State -eq 'Running')
    {
        $wmiStatus = 'Timeout'
        Stop-Job -Job $wmiJob
    }
    elseif ($wmiJob.JobStateInfo.State -eq 'Failed')
    {
        $wmiStatus = 'Failed'
    }
    else
    {
        $wmiStatus = 'OK'
        $wmiResult = Receive-Job $wmiJob
    }
    Remove-Job -Job $wmiJob
    
    Write-Verbose ("..WMI status: {0}" -f $wmiStatus)
    
    
    if ($wmiStatus -eq 'OK')
    {
        Write-Verbose ("...Found {0} events." -f $wmiResult.Count)

        $logonEvents = $wmiResult | ? { $_.CategoryString -eq 'Logon' }
        $logoffEvents = $wmiResult | ? { $_.CategoryString -eq 'Logoff' }
        
        Write-Verbose ("....Logon: {0, 10}" -f $logonEvents.Count)
        Write-Verbose ("....Logoff: {0, 9}" -f $logoffEvents.Count)

        foreach ($event in $logonEvents)
        {
            $results += New-Object PSObject -Property @{
                'ComputerName' = $ComputerName;
                'EventType' = 'Logon';
                'LogonID' = $event.InsertionStrings[7];
                'Timestamp' = $event.ConvertToDatetime($event.TimeGenerated);
                'Account' = $event.InsertionStrings[5];
                'SourceIP' = $event.InsertionStrings[18];
            } | Select ComputerName,EventType,LogonID,Timestamp,Account,SourceIP
        }

        #$logoffEventsResults = @()
        foreach($event in $logoffEvents)
        {
            $results += New-Object PSObject -Property @{
                'ComputerName' = $ComputerName;
                'EventType' = 'Logoff';
                'LogonID' = $event.InsertionStrings[3];
                'Timestamp' = $event.ConvertToDateTime($event.TimeGenerated);
                'Account' = $event.InsertionStrings[1];
                'SourceIP' = 'N/A'
            } | Select ComputerName,EventType,LogonID,Timestamp,Account,SourceIP
        }
    }
    
    Write-Output $results
}

Now we just need to run it:

Write-Host (Get-Date -Format 'HH:mm:ss')
$tmp = GetLogOnOffEvents -ComputerName 'tst-server1' -userAccount "super-admin" -Verbose
Write-Host (Get-Date -Format 'HH:mm:ss')

Write-Host '========================================='



$tmp | ft * -AutoSize

 

 

Monitoring virtual machines through scripts

How about monitoring VM’s using scripts instead of going to vCenter client? Is this posible?

As an administrator you can select a VM or an ESX and go to Performance tab and see the stats.

vc-perf-001

The stats provided by vCenter will look something similar with the ones below:

vc-perf

But how about creating your own charts based on some stats you are setting manually?

The first step will be to extract all the info you need by querying the virtual machines.

# import the necessary snapins

Add-PSSnapin VMware.VimAutomation.Core
Add-PSSnapin VMware.VimAutomation.Vds

# define the input file
# this will be the list of servers based on which the script will gather data
$inputFile = "C:\somepathhere\stats-vcenter.csv"

# define & create the output location
$start = Get-Date
$outputLocation = "C:\somepathhere\Get-VMPerfRealtime\{0}-{1:D2}-{2:D2} {3:D2}{4:D2}\" -f $start.Year, $start.Month, $start.Day, $start.Hour, $start.Minute
New-Item -ItemType Directory -Path $outputLocation | Out-Null

# connect to vCenter
Connect-VIServer My-vCenter-server -WarningAction SilentlyContinue | Out-Null

# get the VM objects
# get the list of servers either from an imput document - *.txt or *.csv or by using a query directly pointed to vCenter.
$vms = Get-VM -Name (import-csv $inputFile)


$vms | Foreach-Object {

	$serverStats = $_ | Get-Stat -Realtime -Stat (($_ | Get-StatType -Realtime) | Where { ($_ -like "cpu.entitlement*") -or ($_ -like "cpu.demand*") }) | Sort Timestamp,MetricId,Instance | Select Timestamp,MetricId,Value,Unit | Group Timestamp
	
	$results = @()
	
	$serverStats | Foreach-Object {

		$cpuDemand = $_.Group | Where { $_.MetricId -like "cpu.demand*" } | Select -ExpandProperty Value
		$cpuEntitlement = $_.Group | Where { $_.MetricId -like "cpu.entitlement*" } | Select -ExpandProperty Value

		$results += New-Object PSObject -Property @{
			'Timestamp' = ($_.Name -split " ")[1].Substring(0,5);
			'CPU Demand (MHz)' = $cpuDemand;
			'CPU Entitlement (MHz)' = $cpuEntitlement
		}
	}
	
	$filename = "{0}{1}.csv" -f $outputLocation, $_.Name
	
	$results | Export-Csv $filename -NoTypeInformation
}

Disconnect-VIServer My-vCenter-server -Confirm:$false

The above script is getting the CPU demand and CPU Entitlement (in Mhz) stats and store them in a *.csv format somewhere on disk. The *.csv file will contain some stats for each server we’ve queried.

"CPU Demand (MHz)","Timestamp","CPU Entitlement (MHz)"
"7095","8:55:","5845"
"7258","8:55:","5845"
"5289","8:56:","5845"
"4053","8:56:","5845"
"3029","8:56:","5845"
"4210","8:57:","5845"
"5019","8:57:","5845"
"4821","8:57:","5845"
"3599","8:58:","5845"
"2411","8:58:","5845"
"2736","8:58:","5845"
"2331","8:59:","5845"
"2723","8:59:","5845"
"2326","8:59:","5845"
"3221","9:00:","5845"

… then based on the stats we already captured we can create our own charts. This is not something very easy to do in Powershell, but as Powershell is very well integrated with all C# classes we can easily use the System.Windows.Forms namespace. The System.Windows.Forms namespace contains classes for creating Windows-based applications that take full advantage of the rich user interface features available in the Microsoft Windows operating system. (https://msdn.microsoft.com/en-us/library/system.windows.forms(v=vs.110).aspx)

We will need to also use “Reflection” to add the necessary assemblies in Powershell. You may refer Scripting Guy! Blog.

# load the appropriate assemblies 
[Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms") | Out-Null
[Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms.DataVisualization") | Out-Null

Now that we added the assemblies we will need to create a window for type chart, declare the chart area, create the chart legend and draw all necessarily lines.

We will end with a script like the below one:

########################################
# Plot the data
########################################



# load the appropriate assemblies 
[Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms") | Out-Null
[Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms.DataVisualization") | Out-Null

foreach ($file in (Get-ChildItem $outputLocation -Filter *.csv)) {

	$chart1 = New-object System.Windows.Forms.DataVisualization.Charting.Chart
	$chart1.Width = 1200
	$chart1.Height = 500
	$chart1.BackColor = [System.Drawing.Color]::White

	# title 
	$chart1.Titles.Add($file.BaseName) | Out-Null
	$chart1.Titles[0].Font = "Arial,13pt"
	$chart1.Titles[0].Alignment = "topLeft"

	# chart area 
	$chartarea = New-Object System.Windows.Forms.DataVisualization.Charting.ChartArea
	$chartarea.Name = "ChartArea1"
	$chartarea.AxisY.Title = "MHz"
	$chartarea.AxisX.Title = "Time"
	$chartarea.AxisY.Interval = 1000
	$chartarea.AxisX.Interval = 10
	$chartarea.AxisY.Maximum = 10000
	$chartarea.AxisY.IsStartedFromZero = $true
	$chartarea.AxisX.MinorGrid.Enabled = $false
	$chartarea.AxisX.MajorGrid.Enabled = $false

	$chart1.ChartAreas.Add($chartarea)

	# legend 
	$legend = New-Object system.Windows.Forms.DataVisualization.Charting.Legend
	$legend.name = "Legend1"
	$chart1.Legends.Add($legend)

	$datasource = Import-Csv $file.FullName

	[void]$chart1.Series.Add("CpuDemand")
	$chart1.Series["CpuDemand"].ChartType = "Line"
	$chart1.Series["CpuDemand"].BorderWidth  = 2
	$chart1.Series["CpuDemand"].IsVisibleInLegend = $true
	$chart1.Series["CpuDemand"].chartarea = "ChartArea1"
	$chart1.Series["CpuDemand"].Legend = "Legend1"
	$chart1.Series["CpuDemand"].color = "#62B5CC"
	$datasource | ForEach-Object {$chart1.Series["CpuDemand"].Points.addxy( $_.Timestamp , $_.'CPU Demand (MHz)') | Out-Null }

	[void]$chart1.Series.Add("CpuEntitlement")
	$chart1.Series["CpuEntitlement"].ChartType = "Line"
	$chart1.Series["CpuEntitlement"].BorderWidth  = 2
	$chart1.Series["CpuEntitlement"].IsVisibleInLegend = $true
	$chart1.Series["CpuEntitlement"].chartarea = "ChartArea1"
	$chart1.Series["CpuEntitlement"].Legend = "Legend1"
	$chart1.Series["CpuEntitlement"].color = "#FF0000"
	$datasource | ForEach-Object { $chart1.Series["CpuEntitlement"].Points.addxy( $_.Timestamp , $_.'CPU Entitlement (MHz)') | Out-Null }

	$chart1.SaveImage("$outputLocation$($file.BaseName).png","png")


	
	
}

The $outputLocation variable  will have the same path declared as in the 1st script where the *.csv’s are stored. Of course the *.csv files can be populated with any type of information. The only thing we need to take in consideration is that for each column in the *.csv  file we need to create a Chart series in the 2nd script like the one below.

[void]$chart1.Series.Add("CpuDemand")
	$chart1.Series["CpuDemand"].ChartType = "Line"
	$chart1.Series["CpuDemand"].BorderWidth  = 2
	$chart1.Series["CpuDemand"].IsVisibleInLegend = $true
	$chart1.Series["CpuDemand"].chartarea = "ChartArea1"
	$chart1.Series["CpuDemand"].Legend = "Legend1"
	$chart1.Series["CpuDemand"].color = "#62B5CC"
	$datasource | ForEach-Object {$chart1.Series["CpuDemand"].Points.addxy( $_.Timestamp , $_.'CPU Demand (MHz)') | Out-Null }

The System.Windows.Forms namespace is just an example of namespaces that can be added using Reflection into an Powershell script. Of course, not all namespaces from C# will work with Powershell, but most of them will.

In the end, the charts will look something like the below ones:

vc-perf-002

 

vc-perf-003