Sort using LINQ to XML

The hobby for the last few days was to “play” with XML and LINQ syntax.

Something that I’ve tried doing was to sort an XML document not using XSL stylesheet but using LINQ. I’ve start by sorting the root elements from the XML documents.

        private static XElement Sort(XElement element)
        {
            return new XElement(element.Name,
                    element.Attributes(),
                    from child in element.Nodes()
                    where child.NodeType != XmlNodeType.Element
                    select child,
                    from child in element.Elements()
                    orderby child.Name.ToString()
                    select Sort(child));
        }

        private static XDocument Sort(XDocument file)
        {
            return new XDocument(
                    file.Declaration,
                    from child in file.Nodes()
                    where child.NodeType != XmlNodeType.Element
                    select child,
                    Sort(file.Root));
        }

Next step was to sort  deep XNodes. In order to this we need to identify the node:

                IEnumerable<XElement> currentNode =
                    (from el in xDocReport.Root.Elements("Incident")
                     where (string)el.Attribute("INumber") == refrence.ToString()
                     select el                  
                     );

… then based on the node we just do the sorting

                var xmlQuery = from item in currentNode.First().Descendants("Update")
                            orderby (Int32)item.Attribute("UNumber") descending
                            select item;

… then just replace the container with the one sorted

currentNode.First().Element("Updates").ReplaceWith(new XElement ("Updates", xmlQuery.Take(5)));

Take.(5) means that we are just taking the lsat 5 elemenst from the query.

All the above will be done in a foreach statement:

// Normal

foreach (var itemTest in xDocReport.Root.Elements("Incident").Attributes("INumber")) { }

// multi threading 

Parallel.ForEach(xDocReport.Root.Elements("Incident").Attributes("INumber"), itemTest => { });

The full schema based on which the XML was created and the LINQ query applied was looking as the one below:

<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="Report">
    <xs:complexType>
      <xs:sequence>
        <xs:element maxOccurs="unbounded" name="Incident">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="Company" type="xs:string" />
              <xs:element name="Description" type="xs:string" />
              <xs:element name="Priority" type="xs:unsignedByte" />
              <xs:element name="RemainingSLA" type="xs:string" />
              <xs:element name="SLA-Stage" type="xs:string" />
              <xs:element name="Updates">
                <xs:complexType>
                  <xs:sequence minOccurs="0">
                    <xs:element maxOccurs="unbounded" name="Update">
                      <xs:complexType>
                        <xs:sequence>
                          <xs:element name="Text" type="xs:string" />
                          <xs:element name="UpdatedBy" type="xs:string" />
                          <xs:element name="UpdateTime" type="xs:string" />
                        </xs:sequence>
                        <xs:attribute name="UNumber" type="xs:unsignedShort" use="required" />
                      </xs:complexType>
                    </xs:element>
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
            <xs:attribute name="INumber" type="xs:string" use="required" />
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

 

 

Orphaned Snapshots in VMWare

Considering that the snapshot are not listed in VMWare snapshot manager and presuming that already we’ve tried to create a new snapshot ant then delete all and nothing is happening, in order to remove the orphaned snapshots we need:

  • First step is to find all snapshots and delta disks on the datastores: from SSH on the host
    find /vmfs/volumes/ -name *-delta*;find /vmfs/volumes/ -name *-0000*

    (if you know the VM this is not required)

  • This will list the vmdk’s which  are mounted by VMX file for the affected VM
     cat /vmfs/volumes/[id]/[VM]/VM.vmx | grep vmdk

    (this should be checked also because we don’t know if all disks which resides in the VM folder are used)

  • If you use cat on this vmdk files you should see what underlying file on the datastore they reference under  Extent Description and  Change tracking file
    cat VM_X.vmdk

    Then you should check if the vmkd file are locked by any hosts –delta , -ctk and –flat

    vmkfstools -D VM-000002-delta.vmdk

    (If it was locked by an ESXi host the MAC of the host would be shown in the owner)

  • Then make sure they are not locked by a process, touch them and see that the timestamp updates:
    touch /vmfs/volumes/[id]/[VM]/*-00000* | ls -ltr | grep vmdk

    Being able to touch the file, run vmkfstools -D finding no locks, find no references in vmdk descriptor files generally means it isn’t in active use and is safe to move/remove. (Please be aware this command should be used from other host, not from the one where VM resides )

  • Create a new directory and move the suspect files to it and check for problems with the VM
mkdir toberemoved 
mv *-00000* toberemoved/.
  • Check the VM state and if it is operating normally delete the directory
rm –r toberemoved/

 

Netstat and Powershell

 

The correct name should be: how to trace connections and monitor them using PowerShell. The wide used tool to see open connection in and out from a computer is through netstat command from cmd promt. Of course we can use Wireshark to do deep troubleshooting.

How about capturing the output and store them in an array so we can automate the process and do periodic checks to see if we have any errors. Usually the output will display listening and established states but something due to network bottlenecks and/or applications hanging we may get other states.

The script below is capturing the netstat -ano output and split it so we can store it in an array and use it after. This function has few parameters:

.PARAMETER ProcessName
	Gets connections by the name of the process. The default value is '*'.
	
.PARAMETER Port
	The port number of the local computer or remote computer. The default value is '*'.

.PARAMETER Address
	Gets connections by the IP address of the connection, local or remote. Wildcard is supported. The default value is '*'.

.PARAMETER Protocol
	The name of the protocol (TCP or UDP). The default value is '*' (all)
	
.PARAMETER State
	Indicates the state of a TCP connection. The possible states are as follows:
		
	Closed	 	- The TCP connection is closed. 
	CloseWait 	- The local endpoint of the TCP connection is waiting for a connection termination request from the local user. 
	Closing 	- The local endpoint of the TCP connection is waiting for an acknowledgement of the connection termination request sent previously. 
	DeleteTcb 	- The transmission control buffer (TCB) for the TCP connection is being deleted. 
	Established - The TCP handshake is complete. The connection has been established and data can be sent. 
	FinWait1 	- The local endpoint of the TCP connection is waiting for a connection termination request from the remote endpoint or for an acknowledgement of the connection termination request sent previously. 
	FinWait2 	- The local endpoint of the TCP connection is waiting for a connection termination request from the remote endpoint. 
	LastAck 	- The local endpoint of the TCP connection is waiting for the final acknowledgement of the connection termination request sent previously. 
	Listen	 	- The local endpoint of the TCP connection is listening for a connection request from any remote endpoint. 
	SynReceived - The local endpoint of the TCP connection has sent and received a connection request and is waiting for an acknowledgment. 
	SynSent 	- The local endpoint of the TCP connection has sent the remote endpoint a segment header with the synchronize (SYN) control bit set and is waiting for a matching connection request. 
	TimeWait	- The local endpoint of the TCP connection is waiting for enough time to pass to ensure that the remote endpoint received the acknowledgement of its connection termination request. 
	Unknown		- The TCP connection state is unknown.
	
	Values are based on the TcpState Enumeration:
	http://msdn.microsoft.com/en-us/library/system.net.networkinformation.tcpstate%28VS.85%29.aspx

All the above parameters are not mandatory so we can execute this function with or without them.

function Get-NetworkStatistics
{
	[OutputType('System.Management.Automation.PSObject')]
	[CmdletBinding(DefaultParameterSetName='name')]
	
	param(
		[Parameter(Position=0,ValueFromPipeline=$true,ParameterSetName='port')]
		[System.Int32]$Port,
		
		[Parameter(Position=0,ValueFromPipeline=$true,ParameterSetName='name')]
		[System.String]$ProcessName='*',
		
		[Parameter(Position=0,ValueFromPipeline=$true,ParameterSetName='address')]
		[System.String]$Address='*',		
		
		[Parameter()]
		[ValidateSet('*','tcp','udp')]
		[System.String]$Protocol='*',

		[Parameter()]
		[ValidateSet('*','Closed','CloseWait','Closing','DeleteTcb','Established','FinWait1','FinWait2','LastAck','Listen','SynReceived','SynSent','TimeWait','Unknown')]
		[System.String]$State='*'
		
	)
    
	begin
	{
		$properties = 'Protocol','LocalAddress','LocalPort'
    		$properties += 'RemoteAddress','RemotePort','State','ProcessName','PID'
	}
	
	process
	{
	    netstat -ano | Select-String -Pattern '\s+(TCP|UDP)' | ForEach-Object {

	        $item = $_.line.split(' ',[System.StringSplitOptions]::RemoveEmptyEntries)

	        if($item[1] -notmatch '^\[::')
	        {           
	            if (($la = $item[1] -as [ipaddress]).AddressFamily -eq 'InterNetworkV6')
	            {
	               $localAddress = $la.IPAddressToString
	               $localPort = $item[1].split('\]:')[-1]
	            }
	            else
	            {
	                $localAddress = $item[1].split(':')[0]
	                $localPort = $item[1].split(':')[-1]
	            } 

	            if (($ra = $item[2] -as [ipaddress]).AddressFamily -eq 'InterNetworkV6')
	            {
	               $remoteAddress = $ra.IPAddressToString
	               $remotePort = $item[2].split('\]:')[-1]
	            }
	            else
	            {
	               $remoteAddress = $item[2].split(':')[0]
	               $remotePort = $item[2].split(':')[-1]
	            } 
				
				$procId = $item[-1]
				$procName = (Get-Process -Id $item[-1] -ErrorAction SilentlyContinue).Name
				$proto = $item[0]
				$status = if($item[0] -eq 'tcp') {$item[3]} else {$null}				
				
				
				$pso = New-Object -TypeName PSObject -Property @{
					PID = $procId
					ProcessName = $procName
					Protocol = $proto
					LocalAddress = $localAddress
					LocalPort = $localPort
					RemoteAddress =$remoteAddress
					RemotePort = $remotePort
					State = $status
				} | Select-Object -Property $properties								


				if($PSCmdlet.ParameterSetName -eq 'port')
				{
					if($pso.RemotePort -like $Port -or $pso.LocalPort -like $Port)
					{
					    if($pso.Protocol -like $Protocol -and $pso.State -like $State)
						{
							$pso
						}
					}
				}

				if($PSCmdlet.ParameterSetName -eq 'address')
				{
					if($pso.RemoteAddress -like $Address -or $pso.LocalAddress -like $Address)
					{
					    if($pso.Protocol -like $Protocol -and $pso.State -like $State)
						{
							$pso
						}
					}
				}
				
				if($PSCmdlet.ParameterSetName -eq 'name')
				{		
					if($pso.ProcessName -like $ProcessName)
					{
						if($pso.Protocol -like $Protocol -and $pso.State -like $State)
						{
					   		$pso
						}
					}
				}
	        }
	    }
	}
<#

.SYNOPSIS
	Displays the current TCP/IP connections.

.DESCRIPTION
	Displays active TCP connections and includes the process ID (PID) and Name for each connection.
	If the port is not yet established, the port number is shown as an asterisk (*).	
	
.PARAMETER ProcessName
	Gets connections by the name of the process. The default value is '*'.
	
.PARAMETER Port
	The port number of the local computer or remote computer. The default value is '*'.

.PARAMETER Address
	Gets connections by the IP address of the connection, local or remote. Wildcard is supported. The default value is '*'.

.PARAMETER Protocol
	The name of the protocol (TCP or UDP). The default value is '*' (all)
	
.PARAMETER State
	Indicates the state of a TCP connection. The possible states are as follows:
		
	Closed	 	- The TCP connection is closed. 
	CloseWait 	- The local endpoint of the TCP connection is waiting for a connection termination request from the local user. 
	Closing 	- The local endpoint of the TCP connection is waiting for an acknowledgement of the connection termination request sent previously. 
	DeleteTcb 	- The transmission control buffer (TCB) for the TCP connection is being deleted. 
	Established - The TCP handshake is complete. The connection has been established and data can be sent. 
	FinWait1 	- The local endpoint of the TCP connection is waiting for a connection termination request from the remote endpoint or for an acknowledgement of the connection termination request sent previously. 
	FinWait2 	- The local endpoint of the TCP connection is waiting for a connection termination request from the remote endpoint. 
	LastAck 	- The local endpoint of the TCP connection is waiting for the final acknowledgement of the connection termination request sent previously. 
	Listen	 	- The local endpoint of the TCP connection is listening for a connection request from any remote endpoint. 
	SynReceived - The local endpoint of the TCP connection has sent and received a connection request and is waiting for an acknowledgment. 
	SynSent 	- The local endpoint of the TCP connection has sent the remote endpoint a segment header with the synchronize (SYN) control bit set and is waiting for a matching connection request. 
	TimeWait	- The local endpoint of the TCP connection is waiting for enough time to pass to ensure that the remote endpoint received the acknowledgement of its connection termination request. 
	Unknown		- The TCP connection state is unknown.
	
	Values are based on the TcpState Enumeration:
	http://msdn.microsoft.com/en-us/library/system.net.networkinformation.tcpstate%28VS.85%29.aspx

.EXAMPLE
	Get-NetworkStatistics

.EXAMPLE
	Get-NetworkStatistics iexplore

.EXAMPLE
	Get-NetworkStatistics -ProcessName md* -Protocol tcp 

.EXAMPLE
	Get-NetworkStatistics -Address 192* -State LISTENING 

.EXAMPLE
	Get-NetworkStatistics -State LISTENING -Protocol tcp

.OUTPUTS
	System.Management.Automation.PSObject
#>	
}

The output will look like:

PS C:\Users\Administrator> Get-NetworkStatistics | ft *

Protocol        LocalAddress    LocalPort       RemoteAddress  RemotePort     State          ProcessName    PID           
--------        ------------    ---------       -------------  ----------     -----          -----------    ---           
TCP             0.0.0.0         135             0.0.0.0        0              LISTENING      svchost        656           
TCP             0.0.0.0         445             0.0.0.0        0              LISTENING      System         4             
TCP             0.0.0.0         1025            0.0.0.0        0              LISTENING      wininit        476           
TCP             0.0.0.0         1026            0.0.0.0        0              LISTENING      lsass          544           
TCP             0.0.0.0         1027            0.0.0.0        0              LISTENING      svchost        784           
TCP             0.0.0.0         1028            0.0.0.0        0              LISTENING      svchost        820           
TCP             0.0.0.0         1029            0.0.0.0        0              LISTENING      spoolsv        940           
TCP             0.0.0.0         1030            0.0.0.0        0              LISTENING      services       536           
TCP             0.0.0.0         3389            0.0.0.0        0              LISTENING      svchost        1576          
TCP             0.0.0.0         5985            0.0.0.0        0              LISTENING      System         4             
TCP             0.0.0.0         47001           0.0.0.0        0              LISTENING      System         4             
TCP             10.0.0.14       139             0.0.0.0        0              LISTENING      System         4             
TCP             192.168.1.159   139             0.0.0.0        0              LISTENING      System         4             
TCP             192.168.1.159   1389            189.99.255.140 443            ESTABLISHED    MySecureAPP    2724          
TCP             192.168.1.159   3389            10.121.8.79    54785          ESTABLISHED    svchost        1576          
UDP             0.0.0.0         3389            *              *                             svchost        1576          
UDP             0.0.0.0         5355            *              *                             svchost        952           
UDP             10.0.0.14       137             *              *                             System         4             
UDP             10.0.0.14       138             *              *                             System         4             
UDP             192.168.1.159   137             *              *                             System         4             
UDP             192.168.1.159   138             *              *                             System         4             

We can also use parameters:

PS C:\Users\Administrator> Get-NetworkStatistics  -ProcessName svchost| ft *

Protocol        LocalAddress    LocalPort       RemoteAddress  RemotePort     State          ProcessName    PID           
--------        ------------    ---------       -------------  ----------     -----          -----------    ---           
TCP             0.0.0.0         135             0.0.0.0        0              LISTENING      svchost        656           
TCP             0.0.0.0         1027            0.0.0.0        0              LISTENING      svchost        784           
TCP             0.0.0.0         1028            0.0.0.0        0              LISTENING      svchost        820           
TCP             0.0.0.0         3389            0.0.0.0        0              LISTENING      svchost        1576          
TCP             192.168.1.159   3389            10.255.2.79    54785          ESTABLISHED    svchost        1576          
UDP             0.0.0.0         3389            *              *                             svchost        1576          
UDP             0.0.0.0         5355            *              *                             svchost        952           
UDP             fe80::2066:4... 546             *              *                             svchost        784

The above function will be useful in case we want to trace broken connections or we want to see when our computer is generating request on web or trace specific protocols. I’m usual using this to monitor open ports and listen for incoming connection when debugging some applications.

Active Directory health check – DCDIAG friendly view

We can monitor all Domain Controllers events using different monitoring tools like System Center Operation Manager or any other tools from any other 3rd parties. But how about automating a health check of a domain controller and instead of having a text document with all the information inside where we need to look after  errors and warning we can create a user friendly table easy to read and interpret. We just need somehow to run a DCDIAG and then parse the output using Regex and display it.

DCDIAG is as an end-user reporting program, a command-line tool that encapsulates detailed knowledge of how to identify abnormal behavior in the system. Dcdiag displays command output at the command prompt.

What we need now is to create a Powershell function that takes the output from dcdiag, analyze the output and then present it in a easy readable format.

The function will be:

Function Test-ADInfrastructure {
    
	Param (
		[Parameter(Mandatory=$true)]
        $domain,                                       # Domain

        [Parameter(Mandatory=$false)]
        [string]$namingconvention = "*",                # Server naming convention

        [Parameter(Mandatory=$false)]
        $tmpFile = [System.Io.Path]::GetTempFileName(),  # temp file
        
        [String]$OutputFile
	)
	
    # run dcdiag and get the output

    dcdiag.exe /e /n:$domain | Out-File $tmpFile

    $dcDiagOutput = Get-Content $tmpFile
    Copy-Item $tmpFile $outputFile
    Remove-Item $tmpFile


    $regex = "\.{25}\s(.*?)\s(passed|failed)\stest\s(.*?)$"

    # Obtain am array with elements
    # of type @{Server, Test, Passed}
    $testResults = @()
    $dcDiagOutput | Foreach-Object {

    	if ( $_ -match $regex) {
    		#$matches
    		
    		$testResults += New-Object PSObject -Property @{
    			'Server' = $matches[1] -as [String];
    			'Test' = $matches[3] -as [String];
    			'Passed' = ($matches[2] -eq 'passed')
    		} | Select Server,Test,Passed
    	}
    }
    # group elements by Server
    $dcDiagTmp = $testResults | Where { $_.Server -like $namingconvention } | Group-Object -Property Server -AsHashTable


    # return elements of type
    # @{Server, TestName1, TestName2, FrsEvent, ... , VerifyReferences}
    #
    # e.g.
    #
    #   Server             : SRV-DC-01
    #   Connectivity       : True
    #   Advertising        : True
    #   FrsEvent           : True
    #   DFSREvent          : True
    #   SysVolCheck        : True
    #   KccEvent           : True
    #   KnowsOfRoleHolders : True
    #   MachineAccount     : True
    #   NCSecDesc          : True
    #   NetLogons          : True
    #   ObjectsReplicated  : True
    #   Replications       : True
    #   RidManager         : True
    #   Services           : False
    #   SystemLog          : True
    #   VerifyReferences   : True
    #
    foreach ($server in $dcDiagTmp.Keys) {
    	
    	$tmp = New-Object PSObject -Property @{'Server' = $server} 

    	$dcDiagTmp.$($server) | Foreach-Object {
    		$tmp | Add-Member NoteProperty -Name $($_.Test) -Value $($_.Passed)
    	}
    	
    	Write-Output $tmp
    }
}

There is just one mandatory parameter which is -domain. Also this function has other non-mandatory parameters like: -namingconvention and -tmpfile. Domain was set as mandatory parameter in order to be able to run this function on multiple domains in one go. We can set it as Mandatory=$false and replace the null value with $env:userdomain. This variable will pick up the domain from where you run the command.

In order to have a easy readable output I’ve used a regex syntax, Regex stands for regular expression.

$regex = "\.{25}\s(.*?)\s(passed|failed)\stest\s(.*?)$"

All about Regular expresion can be found here:https://en.wikipedia.org/wiki/Regular_expression

For all the servers we will get an output like:

# return elements of type
    # @{Server, TestName1, TestName2, FrsEvent, ... , VerifyReferences}
    #
    # e.g.
    #
    #   Server             : SRV-DC-01
    #   Connectivity       : True
    #   Advertising        : True
    #   FrsEvent           : True
    #   DFSREvent          : True
    #   SysVolCheck        : True
    #   KccEvent           : True
    #   KnowsOfRoleHolders : True
    #   MachineAccount     : True
    #   NCSecDesc          : True
    #   NetLogons          : True
    #   ObjectsReplicated  : True
    #   Replications       : True
    #   RidManager         : True
    #   Services           : False
    #   SystemLog          : True
    #   VerifyReferences   : True
    #

Based on the above output we can create a table and read it easily. Also we can colour the output so in case some test failed we can spot it quickly.

dcdiag2

Powershell & C# Notifyicon

Some time ago I’ve managed to create a tool to show me in real time when a new high priority incident is being logged and when one of the priority incidents was closed. In the background the tool was querying Service Now and extracting some XML file which then were stored in memory and displayed in a grid. To make some room on my desktop, the tool was configured to run in the background and display small notification balloons in the corned with what’s changed in the database.

Based on the above, I’ve tried replicating the same in Powershell. This may help when running time consuming scripts and  when we want periodic notification regarding the status of the script.

In C# the code is pretty simple:

notifyIcon1 = new NotifyIcon()


if (notifyIcon1.Visible != true)
{
   notifyIcon1.Visible = true;
}                                                                 // balloon made visibil
notifyIcon1.BalloonTipText = StringNotification;                  // set text
notifyIcon1.BalloonTipTitle = Convert.ToString("NEW: " + newIncidentList.Count + " || CLOSED: " + oldIncidentList.Count);
notifyIcon1.ShowBalloonTip(60000);                                 // How long

The result:

CsharpCapturePNG

 

In PowerShell in order to use a Notifyicon we will need to create an object type System.Windows.Forms.NotifyIcon. After that, the code is almost as the C# one.

The full function will be:

function Show-BalloonTip  
{
 
  [CmdletBinding(SupportsShouldProcess = $true)]
  param
  (
    [Parameter(Mandatory=$true)]
    $Text,
   
    [Parameter(Mandatory=$true)]
    $Title,
   
    [ValidateSet('None', 'Info', 'Warning', 'Error')]
    $Icon = 'Info',
    $Timeout = 10000
  )
 
  Add-Type -AssemblyName System.Windows.Forms

  if ($script:balloon -eq $null)
  {
    $script:balloon = New-Object System.Windows.Forms.NotifyIcon
  }

  $path                    = Get-Process -id $pid | Select-Object -ExpandProperty Path
  $balloon.Icon            = [System.Drawing.Icon]::ExtractAssociatedIcon($path)
  $balloon.BalloonTipIcon  = $Icon
  $balloon.BalloonTipText  = $Text
  $balloon.BalloonTipTitle = $Title
  $balloon.Visible         = $true

  $balloon.ShowBalloonTip($Timeout)
}

We can then use this function in a time consuming script and when we need something to be displayed we will append this function. The mandatory parameters are -text and -title

Show-BalloonTip -text "Some text here" -title "Powershell..... 64.... char"

The output:

notifyIcon

Management – Different methodologies

Do you want to be successful in your business? Are you applying any methodologies in your day to day work?

The tools and techniques we’ve all been using have been only partially successful. Activities or tasks are still experience significant problems like poor quality, large backlogs, missed deadlines and overruns, inadequate management, inaccurate estimating methods, and an unacceptable level of delivered task.

Many IT managers have all but surrendered, but IT companies should instead begin evaluating maturing methodologies and wide related tools; good planning simply makes activities of any kind more productive. Development organizations should embrace and implement integrated methodologies now widely available, but little used or even understood.

There are numerous methodologies on the market as Kanban (a former inventory and control system used by Toyota which creates a wide overview about the input and output of task), Scrum (which is widely used in software development as a strategy framework), ITIL (a set of practices for IT service management with the role of aligning IT services with the needs of businesses), Agile (used in project management for optimization), Six Sigma ( a data-driven approach for eliminating defects or errors in any process).

All the methodologies are just frameworks and can be applied to all domain activities from manufactures to sales. The below table will just highlight some of the key items that we need to accomplish in order to be more successful.

scrum3