Active Directory health check – DCDIAG friendly view

We can monitor all Domain Controllers events using different monitoring tools like System Center Operation Manager or any other tools from any other 3rd parties. But how about automating a health check of a domain controller and instead of having a text document with all the information inside where we need to look after  errors and warning we can create a user friendly table easy to read and interpret. We just need somehow to run a DCDIAG and then parse the output using Regex and display it.

DCDIAG is as an end-user reporting program, a command-line tool that encapsulates detailed knowledge of how to identify abnormal behavior in the system. Dcdiag displays command output at the command prompt.

What we need now is to create a Powershell function that takes the output from dcdiag, analyze the output and then present it in a easy readable format.

The function will be:

Function Test-ADInfrastructure {
    
	Param (
		[Parameter(Mandatory=$true)]
        $domain,                                       # Domain

        [Parameter(Mandatory=$false)]
        [string]$namingconvention = "*",                # Server naming convention

        [Parameter(Mandatory=$false)]
        $tmpFile = [System.Io.Path]::GetTempFileName(),  # temp file
        
        [String]$OutputFile
	)
	
    # run dcdiag and get the output

    dcdiag.exe /e /n:$domain | Out-File $tmpFile

    $dcDiagOutput = Get-Content $tmpFile
    Copy-Item $tmpFile $outputFile
    Remove-Item $tmpFile


    $regex = "\.{25}\s(.*?)\s(passed|failed)\stest\s(.*?)$"

    # Obtain am array with elements
    # of type @{Server, Test, Passed}
    $testResults = @()
    $dcDiagOutput | Foreach-Object {

    	if ( $_ -match $regex) {
    		#$matches
    		
    		$testResults += New-Object PSObject -Property @{
    			'Server' = $matches[1] -as [String];
    			'Test' = $matches[3] -as [String];
    			'Passed' = ($matches[2] -eq 'passed')
    		} | Select Server,Test,Passed
    	}
    }
    # group elements by Server
    $dcDiagTmp = $testResults | Where { $_.Server -like $namingconvention } | Group-Object -Property Server -AsHashTable


    # return elements of type
    # @{Server, TestName1, TestName2, FrsEvent, ... , VerifyReferences}
    #
    # e.g.
    #
    #   Server             : SRV-DC-01
    #   Connectivity       : True
    #   Advertising        : True
    #   FrsEvent           : True
    #   DFSREvent          : True
    #   SysVolCheck        : True
    #   KccEvent           : True
    #   KnowsOfRoleHolders : True
    #   MachineAccount     : True
    #   NCSecDesc          : True
    #   NetLogons          : True
    #   ObjectsReplicated  : True
    #   Replications       : True
    #   RidManager         : True
    #   Services           : False
    #   SystemLog          : True
    #   VerifyReferences   : True
    #
    foreach ($server in $dcDiagTmp.Keys) {
    	
    	$tmp = New-Object PSObject -Property @{'Server' = $server} 

    	$dcDiagTmp.$($server) | Foreach-Object {
    		$tmp | Add-Member NoteProperty -Name $($_.Test) -Value $($_.Passed)
    	}
    	
    	Write-Output $tmp
    }
}

There is just one mandatory parameter which is -domain. Also this function has other non-mandatory parameters like: -namingconvention and -tmpfile. Domain was set as mandatory parameter in order to be able to run this function on multiple domains in one go. We can set it as Mandatory=$false and replace the null value with $env:userdomain. This variable will pick up the domain from where you run the command.

In order to have a easy readable output I’ve used a regex syntax, Regex stands for regular expression.

$regex = "\.{25}\s(.*?)\s(passed|failed)\stest\s(.*?)$"

All about Regular expresion can be found here:https://en.wikipedia.org/wiki/Regular_expression

For all the servers we will get an output like:

# return elements of type
    # @{Server, TestName1, TestName2, FrsEvent, ... , VerifyReferences}
    #
    # e.g.
    #
    #   Server             : SRV-DC-01
    #   Connectivity       : True
    #   Advertising        : True
    #   FrsEvent           : True
    #   DFSREvent          : True
    #   SysVolCheck        : True
    #   KccEvent           : True
    #   KnowsOfRoleHolders : True
    #   MachineAccount     : True
    #   NCSecDesc          : True
    #   NetLogons          : True
    #   ObjectsReplicated  : True
    #   Replications       : True
    #   RidManager         : True
    #   Services           : False
    #   SystemLog          : True
    #   VerifyReferences   : True
    #

Based on the above output we can create a table and read it easily. Also we can colour the output so in case some test failed we can spot it quickly.

dcdiag2

Powershell & C# Notifyicon

Some time ago I’ve managed to create a tool to show me in real time when a new high priority incident is being logged and when one of the priority incidents was closed. In the background the tool was querying Service Now and extracting some XML file which then were stored in memory and displayed in a grid. To make some room on my desktop, the tool was configured to run in the background and display small notification balloons in the corned with what’s changed in the database.

Based on the above, I’ve tried replicating the same in Powershell. This may help when running time consuming scripts and  when we want periodic notification regarding the status of the script.

In C# the code is pretty simple:

notifyIcon1 = new NotifyIcon()


if (notifyIcon1.Visible != true)
{
   notifyIcon1.Visible = true;
}                                                                 // balloon made visibil
notifyIcon1.BalloonTipText = StringNotification;                  // set text
notifyIcon1.BalloonTipTitle = Convert.ToString("NEW: " + newIncidentList.Count + " || CLOSED: " + oldIncidentList.Count);
notifyIcon1.ShowBalloonTip(60000);                                 // How long

The result:

CsharpCapturePNG

 

In PowerShell in order to use a Notifyicon we will need to create an object type System.Windows.Forms.NotifyIcon. After that, the code is almost as the C# one.

The full function will be:

function Show-BalloonTip  
{
 
  [CmdletBinding(SupportsShouldProcess = $true)]
  param
  (
    [Parameter(Mandatory=$true)]
    $Text,
   
    [Parameter(Mandatory=$true)]
    $Title,
   
    [ValidateSet('None', 'Info', 'Warning', 'Error')]
    $Icon = 'Info',
    $Timeout = 10000
  )
 
  Add-Type -AssemblyName System.Windows.Forms

  if ($script:balloon -eq $null)
  {
    $script:balloon = New-Object System.Windows.Forms.NotifyIcon
  }

  $path                    = Get-Process -id $pid | Select-Object -ExpandProperty Path
  $balloon.Icon            = [System.Drawing.Icon]::ExtractAssociatedIcon($path)
  $balloon.BalloonTipIcon  = $Icon
  $balloon.BalloonTipText  = $Text
  $balloon.BalloonTipTitle = $Title
  $balloon.Visible         = $true

  $balloon.ShowBalloonTip($Timeout)
}

We can then use this function in a time consuming script and when we need something to be displayed we will append this function. The mandatory parameters are -text and -title

Show-BalloonTip -text "Some text here" -title "Powershell..... 64.... char"

The output:

notifyIcon

Management – Different methodologies

Do you want to be successful in your business? Are you applying any methodologies in your day to day work?

The tools and techniques we’ve all been using have been only partially successful. Activities or tasks are still experience significant problems like poor quality, large backlogs, missed deadlines and overruns, inadequate management, inaccurate estimating methods, and an unacceptable level of delivered task.

Many IT managers have all but surrendered, but IT companies should instead begin evaluating maturing methodologies and wide related tools; good planning simply makes activities of any kind more productive. Development organizations should embrace and implement integrated methodologies now widely available, but little used or even understood.

There are numerous methodologies on the market as Kanban (a former inventory and control system used by Toyota which creates a wide overview about the input and output of task), Scrum (which is widely used in software development as a strategy framework), ITIL (a set of practices for IT service management with the role of aligning IT services with the needs of businesses), Agile (used in project management for optimization), Six Sigma ( a data-driven approach for eliminating defects or errors in any process).

All the methodologies are just frameworks and can be applied to all domain activities from manufactures to sales. The below table will just highlight some of the key items that we need to accomplish in order to be more successful.

scrum3

Events – Parsing the Event Viewer

The plan for today was to speak about Regular expression and how we can use regex in scripts, but I’ve been diverted to look at some logins and how can I track who’s connecting to some servers and from where.

The output:

07:02:06
=================================
VERBOSE: Connecting to computer: tst-server1
VERBOSE: ..WMI status: OK
VERBOSE: ...Found 168 events.
VERBOSE: ....Logon:         86
VERBOSE: ....Logoff:        82
07:02:24
=========================================
SELECT * FROM Win32_NTLogEvent WHERE (LogFile = 'Security') AND (EventIdentifier = 4634 OR EventIdentifier = 4624) 
AND (Message LIKE '%super-admin%')

ComputerName EventType LogonID    Timestamp             Account     SourceIP      
------------ --------- -------    ---------             -------     --------      
tst-server1  Logon     0xc4542909 4/22/2016 6:31:33 AM  super-admin 192.168.177.9 
tst-server1  Logon     0xc453df73 4/22/2016 6:31:31 AM  super-admin -             
tst-server1  Logon     0xc453a794 4/22/2016 6:31:29 AM  super-admin -             
tst-server1  Logon     0xa6785b38 4/21/2016 1:11:05 PM  super-admin 172.16.101.16 
tst-server1  Logon     0xa66d99e4 4/21/2016 1:09:24 PM  super-admin 172.16.101.16 
tst-server1  Logon     0xa2e04ec2 4/21/2016 11:16:05 AM super-admin 172.16.101.16 
tst-server1  Logon     0x9ee9524c 4/21/2016 9:27:47 AM  super-admin 192.168.177.9 
tst-server1  Logon     0x9ee8e0f3 4/21/2016 9:27:44 AM  super-admin -             
tst-server1  Logon     0x9ee86f5d 4/21/2016 9:27:41 AM  super-admin -             
tst-server1  Logon     0x784c9158 4/20/2016 11:16:46 AM super-admin 172.16.110.154
tst-server1  Logon     0x753a3945 4/20/2016 9:06:45 AM  super-admin 192.168.177.9

The script that was created to get the above data is pretty simple.  The parameters set for this are: computername which is not mandatory as we can run it locally on a server and the username we are after as a mandatory parameter.

Function Get-LogOnOffEvents
{

    [CmdletBinding()]
    
    Param (
        [Parameter(Mandatory=$false)]
        [String]$ComputerName = $Env:ComputerName,

        [Parameter(Mandatory=$true)]
        [String]$userAccount = $Env:username
    )

… then we get the data by using a select in the NTLogEvent table through a WQL query. Instead of parsing the Event viewer using gwmi -class Win32_NTlogEvent we are searching directly in the namespace table as the query is around 8 times faster that regular query using Get-WMIObject -class.. 

$query =    "SELECT * FROM Win32_NTLogEvent "
$query +=   "WHERE (LogFile = 'Security') "
$query +=   "AND (EventIdentifier = 4634 OR EventIdentifier = 4624) "
$query +=   "AND (Message LIKE '%$userAccount%')"

We can query any type of log files and get any type of information from the log file. We just need to know what we are after.

Depending if there is a firewall between the computer where we run the query and the remote one, the script can run successful or fail. This may be due to a WMI job that is hanged or not feeding back any information. As all knows, WMI query doesn’t have a time out in place so we need to define one.

The way I’ve done it here is by using the query as a job and setting a time windows until the job should return some output. If the Job state will be timeout or failed then the next query will not be initiated.

The full function will be:

Function Get-LogOnOffEvents
{

    [CmdletBinding()]
    
    Param (
        [Parameter(Mandatory=$false)]
        [String]$ComputerName = $Env:ComputerName,

        [Parameter(Mandatory=$true)]
        [String]$userAccount = $Env:username
    )
    $results = @()


    $query =    "SELECT * FROM Win32_NTLogEvent "
    $query +=   "WHERE (LogFile = 'Security') "
    $query +=   "AND (EventIdentifier = 4634 OR EventIdentifier = 4624) "
    $query +=   "AND (Message LIKE '%$userAccount%')"

    $query
    Write-Host '================================='

    Write-Verbose ("Connecting to computer: {0}" -f $ComputerName)
    
    # WMI related SYSTEM variables
    $wmiStatus = $null
    $wmiJob = $null
    $wmiResult = $null
    
    # Init the WMI query AS A JOB
    $wmiJob = Get-WmiObject -Query $query -ComputerName $ComputerName -AsJob
    
    $wmiJob | Wait-Job -Timeout 600 | Out-Null
    
    if ($wmiJob.JobStateInfo.State -eq 'Running')
    {
        $wmiStatus = 'Timeout'
        Stop-Job -Job $wmiJob
    }
    elseif ($wmiJob.JobStateInfo.State -eq 'Failed')
    {
        $wmiStatus = 'Failed'
    }
    else
    {
        $wmiStatus = 'OK'
        $wmiResult = Receive-Job $wmiJob
    }
    Remove-Job -Job $wmiJob
    
    Write-Verbose ("..WMI status: {0}" -f $wmiStatus)
    
    
    if ($wmiStatus -eq 'OK')
    {
        Write-Verbose ("...Found {0} events." -f $wmiResult.Count)

        $logonEvents = $wmiResult | ? { $_.CategoryString -eq 'Logon' }
        $logoffEvents = $wmiResult | ? { $_.CategoryString -eq 'Logoff' }
        
        Write-Verbose ("....Logon: {0, 10}" -f $logonEvents.Count)
        Write-Verbose ("....Logoff: {0, 9}" -f $logoffEvents.Count)

        foreach ($event in $logonEvents)
        {
            $results += New-Object PSObject -Property @{
                'ComputerName' = $ComputerName;
                'EventType' = 'Logon';
                'LogonID' = $event.InsertionStrings[7];
                'Timestamp' = $event.ConvertToDatetime($event.TimeGenerated);
                'Account' = $event.InsertionStrings[5];
                'SourceIP' = $event.InsertionStrings[18];
            } | Select ComputerName,EventType,LogonID,Timestamp,Account,SourceIP
        }

        #$logoffEventsResults = @()
        foreach($event in $logoffEvents)
        {
            $results += New-Object PSObject -Property @{
                'ComputerName' = $ComputerName;
                'EventType' = 'Logoff';
                'LogonID' = $event.InsertionStrings[3];
                'Timestamp' = $event.ConvertToDateTime($event.TimeGenerated);
                'Account' = $event.InsertionStrings[1];
                'SourceIP' = 'N/A'
            } | Select ComputerName,EventType,LogonID,Timestamp,Account,SourceIP
        }
    }
    
    Write-Output $results
}

Now we just need to run it:

Write-Host (Get-Date -Format 'HH:mm:ss')
$tmp = GetLogOnOffEvents -ComputerName 'tst-server1' -userAccount "super-admin" -Verbose
Write-Host (Get-Date -Format 'HH:mm:ss')

Write-Host '========================================='



$tmp | ft * -AutoSize

 

 

Monitoring virtual machines through scripts

How about monitoring VM’s using scripts instead of going to vCenter client? Is this posible?

As an administrator you can select a VM or an ESX and go to Performance tab and see the stats.

vc-perf-001

The stats provided by vCenter will look something similar with the ones below:

vc-perf

But how about creating your own charts based on some stats you are setting manually?

The first step will be to extract all the info you need by querying the virtual machines.

# import the necessary snapins

Add-PSSnapin VMware.VimAutomation.Core
Add-PSSnapin VMware.VimAutomation.Vds

# define the input file
# this will be the list of servers based on which the script will gather data
$inputFile = "C:\somepathhere\stats-vcenter.csv"

# define & create the output location
$start = Get-Date
$outputLocation = "C:\somepathhere\Get-VMPerfRealtime\{0}-{1:D2}-{2:D2} {3:D2}{4:D2}\" -f $start.Year, $start.Month, $start.Day, $start.Hour, $start.Minute
New-Item -ItemType Directory -Path $outputLocation | Out-Null

# connect to vCenter
Connect-VIServer My-vCenter-server -WarningAction SilentlyContinue | Out-Null

# get the VM objects
# get the list of servers either from an imput document - *.txt or *.csv or by using a query directly pointed to vCenter.
$vms = Get-VM -Name (import-csv $inputFile)


$vms | Foreach-Object {

	$serverStats = $_ | Get-Stat -Realtime -Stat (($_ | Get-StatType -Realtime) | Where { ($_ -like "cpu.entitlement*") -or ($_ -like "cpu.demand*") }) | Sort Timestamp,MetricId,Instance | Select Timestamp,MetricId,Value,Unit | Group Timestamp
	
	$results = @()
	
	$serverStats | Foreach-Object {

		$cpuDemand = $_.Group | Where { $_.MetricId -like "cpu.demand*" } | Select -ExpandProperty Value
		$cpuEntitlement = $_.Group | Where { $_.MetricId -like "cpu.entitlement*" } | Select -ExpandProperty Value

		$results += New-Object PSObject -Property @{
			'Timestamp' = ($_.Name -split " ")[1].Substring(0,5);
			'CPU Demand (MHz)' = $cpuDemand;
			'CPU Entitlement (MHz)' = $cpuEntitlement
		}
	}
	
	$filename = "{0}{1}.csv" -f $outputLocation, $_.Name
	
	$results | Export-Csv $filename -NoTypeInformation
}

Disconnect-VIServer My-vCenter-server -Confirm:$false

The above script is getting the CPU demand and CPU Entitlement (in Mhz) stats and store them in a *.csv format somewhere on disk. The *.csv file will contain some stats for each server we’ve queried.

"CPU Demand (MHz)","Timestamp","CPU Entitlement (MHz)"
"7095","8:55:","5845"
"7258","8:55:","5845"
"5289","8:56:","5845"
"4053","8:56:","5845"
"3029","8:56:","5845"
"4210","8:57:","5845"
"5019","8:57:","5845"
"4821","8:57:","5845"
"3599","8:58:","5845"
"2411","8:58:","5845"
"2736","8:58:","5845"
"2331","8:59:","5845"
"2723","8:59:","5845"
"2326","8:59:","5845"
"3221","9:00:","5845"

… then based on the stats we already captured we can create our own charts. This is not something very easy to do in Powershell, but as Powershell is very well integrated with all C# classes we can easily use the System.Windows.Forms namespace. The System.Windows.Forms namespace contains classes for creating Windows-based applications that take full advantage of the rich user interface features available in the Microsoft Windows operating system. (https://msdn.microsoft.com/en-us/library/system.windows.forms(v=vs.110).aspx)

We will need to also use “Reflection” to add the necessary assemblies in Powershell. You may refer Scripting Guy! Blog.

# load the appropriate assemblies 
[Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms") | Out-Null
[Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms.DataVisualization") | Out-Null

Now that we added the assemblies we will need to create a window for type chart, declare the chart area, create the chart legend and draw all necessarily lines.

We will end with a script like the below one:

########################################
# Plot the data
########################################



# load the appropriate assemblies 
[Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms") | Out-Null
[Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms.DataVisualization") | Out-Null

foreach ($file in (Get-ChildItem $outputLocation -Filter *.csv)) {

	$chart1 = New-object System.Windows.Forms.DataVisualization.Charting.Chart
	$chart1.Width = 1200
	$chart1.Height = 500
	$chart1.BackColor = [System.Drawing.Color]::White

	# title 
	$chart1.Titles.Add($file.BaseName) | Out-Null
	$chart1.Titles[0].Font = "Arial,13pt"
	$chart1.Titles[0].Alignment = "topLeft"

	# chart area 
	$chartarea = New-Object System.Windows.Forms.DataVisualization.Charting.ChartArea
	$chartarea.Name = "ChartArea1"
	$chartarea.AxisY.Title = "MHz"
	$chartarea.AxisX.Title = "Time"
	$chartarea.AxisY.Interval = 1000
	$chartarea.AxisX.Interval = 10
	$chartarea.AxisY.Maximum = 10000
	$chartarea.AxisY.IsStartedFromZero = $true
	$chartarea.AxisX.MinorGrid.Enabled = $false
	$chartarea.AxisX.MajorGrid.Enabled = $false

	$chart1.ChartAreas.Add($chartarea)

	# legend 
	$legend = New-Object system.Windows.Forms.DataVisualization.Charting.Legend
	$legend.name = "Legend1"
	$chart1.Legends.Add($legend)

	$datasource = Import-Csv $file.FullName

	[void]$chart1.Series.Add("CpuDemand")
	$chart1.Series["CpuDemand"].ChartType = "Line"
	$chart1.Series["CpuDemand"].BorderWidth  = 2
	$chart1.Series["CpuDemand"].IsVisibleInLegend = $true
	$chart1.Series["CpuDemand"].chartarea = "ChartArea1"
	$chart1.Series["CpuDemand"].Legend = "Legend1"
	$chart1.Series["CpuDemand"].color = "#62B5CC"
	$datasource | ForEach-Object {$chart1.Series["CpuDemand"].Points.addxy( $_.Timestamp , $_.'CPU Demand (MHz)') | Out-Null }

	[void]$chart1.Series.Add("CpuEntitlement")
	$chart1.Series["CpuEntitlement"].ChartType = "Line"
	$chart1.Series["CpuEntitlement"].BorderWidth  = 2
	$chart1.Series["CpuEntitlement"].IsVisibleInLegend = $true
	$chart1.Series["CpuEntitlement"].chartarea = "ChartArea1"
	$chart1.Series["CpuEntitlement"].Legend = "Legend1"
	$chart1.Series["CpuEntitlement"].color = "#FF0000"
	$datasource | ForEach-Object { $chart1.Series["CpuEntitlement"].Points.addxy( $_.Timestamp , $_.'CPU Entitlement (MHz)') | Out-Null }

	$chart1.SaveImage("$outputLocation$($file.BaseName).png","png")


	
	
}

The $outputLocation variable  will have the same path declared as in the 1st script where the *.csv’s are stored. Of course the *.csv files can be populated with any type of information. The only thing we need to take in consideration is that for each column in the *.csv  file we need to create a Chart series in the 2nd script like the one below.

[void]$chart1.Series.Add("CpuDemand")
	$chart1.Series["CpuDemand"].ChartType = "Line"
	$chart1.Series["CpuDemand"].BorderWidth  = 2
	$chart1.Series["CpuDemand"].IsVisibleInLegend = $true
	$chart1.Series["CpuDemand"].chartarea = "ChartArea1"
	$chart1.Series["CpuDemand"].Legend = "Legend1"
	$chart1.Series["CpuDemand"].color = "#62B5CC"
	$datasource | ForEach-Object {$chart1.Series["CpuDemand"].Points.addxy( $_.Timestamp , $_.'CPU Demand (MHz)') | Out-Null }

The System.Windows.Forms namespace is just an example of namespaces that can be added using Reflection into an Powershell script. Of course, not all namespaces from C# will work with Powershell, but most of them will.

In the end, the charts will look something like the below ones:

vc-perf-002

 

vc-perf-003

 

AntiMalware Health State Error 0x800106f7

If your Event viewer is reporting “There was an error 0x800106f7 in creating the Antimalware Health State WMI instance” this may be due to a corrupted WMI namespace. In this case the namespace belongs to System Center Endpoint Protection  and the namespace is called root\Microsoft\SecurityClient. The error in Event Viewer will look like the below one.

SCEP-Pic1

If you want to see the WMI query and who;s generating that query you can go to Operational log withing EventViewer\Applications and Services Logs\Microsoft\Windows\WMI-Activity.

The full error:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Microsoft-Windows-WMI-Activity" Guid="{1418EF04-B0B4-4623-BF7E-D74AB47BBDAA}" /> 
  <EventID>5858</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x4000000000000000</Keywords> 
  <TimeCreated SystemTime="2016-04-20T00:11:00.151879300Z" /> 
  <EventRecordID>869221</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="560" ThreadID="10136" /> 
  <Channel>Microsoft-Windows-WMI-Activity/Operational</Channel> 
  <Computer>mycomputer.mydomain.com</Computer> 
  <Security UserID="S-1-5-18" /> 
  </System>
- <UserData>
- <Operation_ClientFailure xmlns="http://manifests.microsoft.com/win/2006/windows/WMI">
  <Id>{8AE74E49-9589-000E-A443-ED8A8995D001}</Id> 
  <ClientMachine>mycomputer</ClientMachine> 
  <User>NT AUTHORITY\SYSTEM</User> 
  <ClientProcessId>3156</ClientProcessId> 
  <Component>Unknown</Component> 
  <Operation>Start IWbemServices::ExecNotificationQuery - root\Microsoft\SecurityClient : SELECT * FROM __InstanceOperationEvent WITHIN 30 WHERE TargetInstance ISA "AntimalwareInfectionStatus"</Operation> 
  <ResultCode>0x80041032</ResultCode> 
  <PossibleCause>Unknown</PossibleCause> 
  </Operation_ClientFailure>
  </UserData>
  </Event>

Next step was to look and see who’s ClientProcessId belongs to.

scep-pid

SMS Agent Host belongs to SCCM and this is why SCCM is unable to display the correct “Endpoint Protection Definition Last update time” and “Endpoint Protection Definition Last Version”.

Next step was to run the same query that was initiated by SMS Agent and see the result. This is when I’ve noticed that despite the fact that the namaspace and class “AntiMalwareHealthStatus” is present on the affected computer, the query is not returning anything. In fact it was returning an error –> Invalid Operation.

PS C:\> gwmi  -Query "select * from AntiMalwareHealthStatus" -namespace "root\Microsoft\SecurityClient"
gwmi :
At line:1 char:1
+ gwmi  -Query "select * from AntiMalwareHealthStatus" ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Get-WmiObject], COMException
    + FullyQualifiedErrorId : GetWMICOMException,Microsoft.PowerShell.Commands.GetWmiObjectCommand

PS C:\>

Running the above query is generating the same event ID in event viewer so this is how you replicate the error.

The fix can be simple at this stage: restart WMI, but in case we have numerous services depending on WMI this will be impossible to happen during normal business hours.

taskkill /im WmiPrvSE.exe /f

Let’s see which WmiPrvSE service was created by SCEP. I’ve found out this using Sysinternals Process Explorer.

scep-proc

The fix will be to recycle the process. If we have more than one server affected – in my case there were 50+ affected by this issue – you will need to use some sort of automation.

Step 1. –> Killing the affected process and releasing the namespace

(Get-Process | where {$_.Modules.filename -like "*mpprovider*"}).kill()

The above will kill the right WmiPrvSE process sa there is jut one that has the MpProvider.dll in use.

Step 2. –> testing if the query is now successful.

PS C:\> gwmi -query "select * from AntiMalwareHealthStatus" -namespace "root\Microsoft\SecurityClient"


__GENUS                            : 2
__CLASS                            : AntimalwareHealthStatus
__SUPERCLASS                       : ProtectionTechnologyStatus
__DYNASTY                          : SerializableToXml
__RELPATH                          : AntimalwareHealthStatus=@
__PROPERTY_COUNT                   : 31
__DERIVATION                       : {ProtectionTechnologyStatus, SerializableToXml}

[...]

When the query is initiated a new WmiPrvSE process is created.

scep-proc-2

 

If we check the WMI-Activity log we will see that the return code is 0x0

WmiPerfInst provider started with result code 0x0. HostProcess = wmiprvse.exe; ProcessID = 46344; ...

This method is non disruptive and can be completed any time. The root cause may be generated by some patches/updates as this can corrupt WMI repository.