Properly scoping get-inboxrule for a custom RBAC role group

After creating a custom RBAC role in Exchange for our help desk so that they could handle simpler end user requests like mailbox size, conference room permissions, etc. after deploying it we got reports that the get-inboxrule command was not working as expected and was throwing a “You may need elevated permissions. isn’t within your current write scopes. Can’t perform save operation.” Error whenever a member of the help desk ran that command against another user. At first we weren’t exactly sure what the issue was because roles like “View-Only recipients” (which was one of the roles that was used to create the custom role group) seemed to be in the right scope for other commands. After some searching we came across a blog post by Pawet Jarosz explaining the problem. So we needed to recreate the role group with a new management role for get-inboxrule that was in the proper scope. Once we did that our help desk was now able to properly view other user’s inbox rules. Below is the custom role group we created

Add all the Get and test commands from “View-Only Configuration”
New-ManagementRole –Name “Helpdesk View-Only Configuration” –Parent “View-Only Configuration”
Get-ManagementRoleEntry “Helpdesk View-Only Configuration*” |
Where Name -notmatch ‘(Get)|(Test)|(Write-AdminAuditLog)|(Start-AuditAssistant)(Get-InboxRule)’ |
Remove-ManagementRoleEntry -Confirm:$False
Add all the Get and test commands from “View-Only recipients”
New-ManagementRole –Name “Helpdesk View-Only recipients” –Parent “View-Only recipients”
Get-ManagementRoleEntry ” Helpdesk View-Only recipients*” |
Where Name -notmatch ‘(Get)|(Test)|(Write-AdminAuditLog)|(Start-AuditAssistant)(Get-InboxRule)’ |
Remove-ManagementRoleEntry -Confirm:$False
Add all the Get and test commands from “Monitoring”
New-ManagementRole –Name “Helpdesk Monitoring” –Parent “Monitoring”
Get-ManagementRoleEntry ” Helpdesk Monitoring*” |
Where Name -notmatch ‘(Get)|(Test)|(Write-AdminAuditLog)|(Start-AuditAssistant)(Get-InboxRule)’ |
Remove-ManagementRoleEntry -Confirm:$False
Get-Inbox rules has scope issues so we need to create a separate role for that
New-ManagementRole –Name “Helpdesk View Inbox Rules” –Parent ‘Mail Recipients’
Get-ManagementRoleEntry ” Helpdesk View Inbox Rules*” |
Where Name -notmatch ‘Get-InboxRule’ |
Remove-ManagementRoleEntry -Confirm:$False
$GroupSplat = @{
Name = ‘Helpdesk Exchange Tasks’
Roles = @(” Helpdesk View-Only Configuration”, ” Helpdesk View-Only recipients”, ” Helpdesk Monitoring”, ” Helpdesk View Inbox Rules”)
Description = ” specific collection of cmdlets needed for view only Exchange management”
members = @(“Global Helpdesk”)
New-RoleGroup @GroupSplat

Posted in Exchange | Leave a comment

Checking Exchange IIS logs for users who downloaded files via OWA

At my work we use a 3rd party application that rides ontop of OWA in Exchange in order to block attachment downloads across certain IP ranges (namely all external access). We recently had an issue with it silently failing and we needed to figure out who might have downloaded attachments via OWA while the application was down. In looking over the IIS logs we determined that we could accurately figure this out by looking for

  1. Any cs-uri-stem like ‘/owa*Get*Attachment*’. In particualr these were the ones we want to capture
    1. /owa/service.svc/s/GetFileAttachment
    2. /owa/service.svc/s/GetAllAttachmentsAsZip
    3. We did not want /owa/service.svc/s/GetAttachmentThumbnail
  2. Any cs-uri-query like ‘*&ClientRequestId=*&encoding=*’
  3. In our parsing of the logs the same cs-uri-stem was used for uploading and downloading files, but downloads had that near the of the query while uploads had ‘*&ClientId=*’

With this info we used the following log parser query to look for any successful file downloads from the IP ranges we considered external as well as any non internal IP range. We also included some other items to weed out any actions a health mailbox performed

SELECT Distinct
TO_LOCALTIME(TO_TIMESTAMP(date, time)) AS [DateTime],
s-ip as [Server-IP],
REVERSEDNS(s-ip) as [ExchangeServer],
cs-uri-stem as [OWAUrl],
cs-username as [UserName],
c-ip as [Client-IP],
REVERSEDNS(c-ip) as [RemoteHostName],
cs(User-Agent) as [Browser]
Where cs-uri-stem Like '/owa%Get%Attachment%'
and cs-uri-stem <> '/owa/service.svc/s/GetAttachmentThumbnail'
and cs-uri-query like '%&ClientRequestId=%&encoding=%'
and sc-status < 400 and cs-username NOT LIKE 'HealthMailbox%' and cs-method = 'GET' and (c-ip like '192.172.%' or c-ip like '189.18.5.%') and (c-ip not like '10.%' or c-ip not like '172.20.2%') and cs(User-Agent) <> 'AMProbe/Local/ClientAccess'
Posted in Exchange, LogParser | Leave a comment

A single server can serve as a witness for multiple DAGs. However, each DAG requires its own witness directory.

This recently trip us up when bringing up new 2016 servers in our existing 2013 environment. We and used the same witness directory for both DAGS (e.g. \\MGMT500\C$\DAGFileShareWitnesses) under the assumption that the witness directories would be different.  While it does create a folder with a different GUID for each DAG. it treats the full path as owned by either DAG. So if any issues occur and the DAG decides to remove the fileshare witness directory it will remove the whole path. So if you are going to use the same server for your fileshare witness make sure to specify a different path for each.

Posted in Uncategorized | Tagged , | Leave a comment

Sending an email with X-Headers in EWS via powershell

I recently needed to test sending an email via EWS with a bunch of custom X-headers. While I have worked with EWS in PowerShell before I couldn’t find any examples of how to do so but found plenty in Java and .Net. So, after stumbling through trying to recreate what I found in PowerShell I came with up with the following and inserted into this script for sending EWS emails.

In the code below, I’m

  1. Creating a new property set so I have access to edit all the parts of the new email. I think all the needed property sets must be loaded at the same time. That is you can’t load one set and add another without it overwriting the first property set
  2. I then save the email since I learned in my testing you can’t add a property to the email without an instance key associated with it, which it doesn’t get until it’s saved.
  3. Create a internet header object and it’s name
  4. Load the property set
  5. Insert the X-Header and its value in the headers of the email
  6. I then update the saved email (though I don’t think this necessary)


#Tested by using this :

#TO view more properties


$propertyset = New-Object Microsoft.Exchange.WebServices.Data.PropertySet (














#You need to save to load more properties (this creates a draft)


#Create New Xheader object

$XHeader = New-object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition([Microsoft.Exchange.WebServices.Data.DefaultExtendedPropertySet]::InternetHeaders,'X-CustomHeader',[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::String)


#Set the property

$message.SetExtendedProperty($XHeader,"Sent Via EWS")

#Update the draft version


Posted in EWS, PowerShell | Leave a comment

Undocumented Change in 2016 MessageID header behavior

At my company, we recently migrated from 2013 to 2016, but had to keep a 2013 footprint around for support of older Blackberry 5 devices. In our environment, we have a SendMail farm that is an internal open relay as well as being responsible for the send and receiving of external mail. We used various config files on this SendMail farm to distinguish if an email came from Exchange and not one of the various internal application servers that relays off this farm. One of those methods was using the MessageID header of the email. In 2013 that message header would look something like GUID@ServerFQDN, as explained in this technet article. The same article also states this is true for 2016, but this does not appear to the case in our mixed environment.  While 2013 still follows this pattern, 2016 instead uses the primary SMTP address of the sending object for its domain on the MessageID header, so GUID@Primary_SMTP_Address_of_Sending_Object. Whenever the primary SMTP address of an Exchange object changes, and that change is reflected in the Global Address Book, any message sent by that object will be in that format. I’m guessing that this was a change meant to help better track messages in Office 365, but I’m curious as to why this behavior change hasn’t been called out in any of the Cumulative updates for 2016 or the official documentation.

Posted in Exchange | Leave a comment

How to remove a user from a security group in a different domain in PowerShell

Recently I ran into an issue at my company removing a user in our primary domain from a group in our root domain using the AD cmdlets in PowerShell. All my company’s user, computer, and group objects are in our primary domain and our root domain is more of a resource forest. The group in question was an Exchange RBAC role in the resource forest. So, when I first attempted the removal as such

Remove-ADGroupMember -Identity “HelpDesk Exchange Tasks” -members doej

I got the following error

Remove-ADGroupMember : Cannot find an object with the Identity: ‘HelpDesk Exchange Tasks’ under: ‘DC=corp,DC=contoso,DC=com’.

At first it seemed obvious that the solution was to use a domain controller in our resource domain to perform the task. So, I tried referencing a DC in the resource domain

Remove-ADGroupMember -Identity “HelpDesk Exchange Tasks” -members doej -server

But got the following error

Remove-ADGroupMember : Cannot find an object with the Identity: ‘CN=doel,OU=US,OU=CORP,DC=corp,DC=contoso,DC=com’’ under: ‘DC=root,DC=contoso,DC=com’.

At that point I didn’t know how to proceed so I did some searching on the internet and came across an MS blog entry entitled Adding/removing members from another forest or domain to groups in Active Directory

Basically, you need to

  1. Choose against what domain server you want to run the command against.
  2. Get the default returned property set of the object in the other domain, referencing a domain controller in that domain if needed
  3. Run the command referencing just the name/samaccountname/CN/DN of the object that will be referenced by the server in the command and for the object in the other domain use the full object
    1. Referencing just the name/samaccountname/CN/DN OR even just selecting those properties on the object will not work. It needs to be the full default object as returned by the get-AD* command you are using to get the object

So, in my example I pulled the PDCEmulator from the resource domain (where the group was) and the default domain (where the user object was)

$DC_In_Root = (Get-ADDomain
$DC_In_Default = (Get-ADDomain

Then I saved the default returned property set of the user object in the current domain (I didn’t need to reference a DC in this domain since it was my default working domain, but it’s done here for clarity’s sake)

$Default_Domain_User = Get-Aduser doej -server $DC_In_Default

In my example, I’m going to use the DC in my root domain to remove the user from the group. So, I only need to reference the group in this domain by name/samaccountname/CN/DN BUT the user needs to be referenced as an object with it’s complete default returned property set. The opposite can be done if needed

Remove-ADGroupMember -Identity “HelpDesk Exchange Tasks” -members $Default_Domain_User -server $DC_In_Root

I’m not sure why it needs to be the complete default property set. In my limited testing, removing just one of the properties caused it to fail.

Posted in Exchange, PowerShell | Leave a comment

Getting Exchange 2013/2016 Add-ins (Outlook Apps) working through a proxy

Like most companies, my organization uses a proxy for all internet traffic. This presented a problem when we tried using Add-ins for Exchange 2013. At the time we could not figure out how to get the subsystem that pulled down apps in Exchange to use the proxy server despite trying the following methods

·        Configuring the proxy in IE

·        Setting the proxy at the Exchange server level via set-exchangeserver -internetwebproxy

·        Using netsh or proxycfg

Since it was not needed at the time we migrated to Exchange 2013, I dropped the effort. Recently though, after we migrated to 2016, an actual request came in for an app from the Office Outlook app store. Since the servers could not get through the proxy we would see errors in the applications logs (Event ID 3018, see below for an example) and we would get errors every time we tried to add an app via the EMS or EAC. In regard to the Event log error, we would see a different URL referenced each time. When we logged into the Exchange host we could easily get to the URL in Internet Explorer (as along as long as our company’s  proxy settings were in place) but the Exchange server could not reach it

Log Name:      Application
Source:        MSExchangeApplicationLogic
Date:          11/19/2017 1:13:29 AM
Event ID:      3018
Task Category: Extension
Level:         Error
Keywords:      Classic
User:          N/A
Scenario[ServiceHealth]: GetConfig. CorrelationId: e0bc58ff-f87e-4f73-a3df-814b4681bbfb. The request failed. Mailbox:  Url: Exception: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.Exchange.Data.ApplicationLogic.Extension.BaseAsyncOmexCommand.<>c__DisplayClass2.<EndGetResponseCallback>b__1()
Event Xml:
<Event xmlns="">
    <Provider Name="MSExchangeApplicationLogic" />
    <EventID Qualifiers="49156">3018</EventID>
    <TimeCreated SystemTime="2017-11-19T06:13:29.933295500Z" />
    <Security />
    <Data>System.Net.WebException: Unable to connect to the remote server ---&gt; System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket&amp; socket, IPAddress&amp; address, ConnectSocketState state, IAsyncResult asyncResult, Exception&amp; exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.Exchange.Data.ApplicationLogic.Extension.BaseAsyncOmexCommand.&lt;&gt;c__DisplayClass2.&lt;EndGetResponseCallback&gt;b__1()</Data>

After some digging we found out that we needed to set the proxy for the account that is running the app pools for Exchange (which in most cases is LOCALSYSTEM) and that it needed to be set with bitsadmin /setproxysetting. When using this command you will be given a message that it is deprecated but I couldn’t find another method to set the proxy for the LOCALYSTSTEM account. Using bitsadmin You can configure the proxy either manually, like so

bitsadmin /util /setieproxy localsystem MANUAL_PROXY "*.corp,; <local>"

Or using a PAC file

bitsadmin /util /setieproxy localsystem AUTOSCRIPT http://security/webproxy/BalaPAC.pac

We had trouble in our environment getting the PAC file to work with Windows Server 2012R2 and it worked half the time with Windows Server 2016. So, we stuck with the manual method. Our exclusions list was really long and apparently was too big for the buffer to read the settings back using

bitsadmin /util /getieproxy localsystem

Or you can check the following registry entry for to verify the setting took: HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\DefaultConnectionSettings

After setting the proxy and restarting IIS, we could install Add-ins via PowerShell and the EAC. Though occasionally we still get the 3018 Application log errors for some URLs, but at least we can now install Add-ins.

Posted in Exchange | Leave a comment

Log parser query to get Exchange clients below a certain patch level

At my company we are currently in the early stages of an Exchange 2013 to Exchange 2016 migration and we needed to identify any Outlook clients below a certain patch level (ones we identified as having issues with Mapi over HTTP via a proxy). So we used the following log parser query to gather a list of all clients past a certain patch level after a certain date and ran it against the RPC and MAPI logs on all our Exchange servers.

SELECT EXTRACT_SUFFIX(client-name,0,'=') as User,
client-name as DN,client-software,
client-software-version as Version,
REVERSEDNS(client-ip) as ClientName, protocol,
TO_LOCALTIME(TO_TIMESTAMP(EXTRACT_PREFIX(TO_STRING([#Fields: date-time]),0,'T'), 'yyyy-MM-dd')) AS [Day]
WHERE (operation='Connect')
And Day > TimeStamp('2017-07-11','yyyy-MM-dd')
And (Version between '15.0.0000' and '15.0.4849.0000') OR (Version between '14.0.0000' and '14.0.7172.4000')
GROUP BY User,DN,client-software,Version,client-mode,client-ip,ClientName,protocol,Day
Posted in Exchange | Leave a comment

“Message Trace” option missing from the Exchange Admin Center in Office 365

Recently I created an RBAC role group for some of my team members so that they could manage a subset of Exchange features in one of our O365 instance. While the Role group I created had the following roles

  • Distribution Groups
  • HistoricalSearch
  • Mail Enabled Public Folders
  • Mail Recipient Creation
  •  Mail Recipients
  • Message Tracking
  • Public Folders
  • Security Group Creation and Membership
  • Security Reader
  • User Options
  • View-Only Audit Logs
  • View-Only Configuration

The “message trace” option was not available under “Mail flow” in the EAC for the members of this role group even though they had access to the get-messagetrace cmdlet when connecting to this Instance via PowerShell. After a call to Microsoft we discovered that we had to add the ‘View Only Recipients’ role to reveal that option in the EAC. This was odd seeing that they had the ‘Mail Recipients’ role already, but it worked.

Posted in Office 365 | Leave a comment

Unable to delete items in modern public folders : “Some Items cannot be deleted. They were either moved or already deleted, or access was denied”

Recently we had reports from a subset of users that they received the following error when they tried to deleted items from a specific set of public folders using their Outlook client: “Some Items cannot be deleted. They were either moved or already deleted, or access was denied”

A quick google search revealed that this happens to a mailbox whenever it goes over its RecoverableItemsQuota. Seeing that this was an Exchange 2013 environment, and starting in Exchange 2013 public folders are now stored in mailboxes, I assumed the same symptom was occurring. Sure enough the public folder mailbox these folders resided in was over its deleted item limit.

Get-Mailbox pubfoldermbx01 -PublicFolder | Select Name, *recoverable*
Name            RecoverableItemsQuota         RecoverableItemsWarningQuota
----            ---------------------         ----------------------------
PubFolderMbx02  60 GB (64,424,509,440 bytes)  40 GB (42,949,672,960 bytes)
Get-Mailbox pubfoldermbx0q -PublicFolder | Get-MailboxStatistics | Select DisplayName, TotalDeletedItemSize
DisplayName     TotalDeletedItemSize
-----------     --------------------
PubFolderMbx01  60 GB (64,424,509,440 bytes)

Another folder in the same public folder mailbox was taking up most the deleted item space of the mailbox. You can find this out by running the following

$PFStats = Get-PublicFolder -ResidentFolders -Mailbox pubfoldermbx01 -Recurse | Get-PublicFolderStatistics
$PFStats | Select Name, FOlderPath, TotalDeletedItemSize | Sort TotalDeletedItemSize -Descending | select -First 3
Name              FolderPath           TotalDeletedItemSize
----              ----------           --------------------
Offending Folder  {Offending Folder}   59.10 GB (63,458,141,798 bytes)
CLEAN             {CLEAN}	       3.517 GB (3,776,367,512 bytes)
Inbox             {Inbox}	       1.715 GB (1,841,476,727 bytes)

At this point we temporarily set the RecoverableItemsQuota on this public folder mailbox to unlimited reached out to the owners of the offending public folder.

Get-Mailbox -PublicFolder pubfoldermbx01 | Set-Mailbox -UseDatabaseRetentionDefaults $FALSE -PublicFolder
Get-Mailbox -PublicFolder pubfoldermbx01 | Set-Mailbox -RetainDeletedItemsFor $NULL –PublicFolder

We learned they were using the folder as a dumping group for alert messages from a system in their development environment, which was generating close to 10,000 messages a day. After explaining the situation and its impact on other users, the public folder owners agreed to a shorter item age and deleted item retention period.

Set-PublicFolder "\Offending Folder" -RetainDeletedItemsFor 0 -AgeLimit 5

We could have also moved the public folder to its own mailbox, but we decided that it would be best to try to limit how long the data was being held instead of continuing to accommodate a large volume of non-critical data.  After about 48 hours the new retention policy kicked in. This is due to the Managed folder assistant needing to first stamp the items with the new retention settings during the first past and then to act on the new stamp after the second pass. This process usually happens every 24 hours in Exchange 2013+. You can manually kick it off using Start-ManagedFolderAssistant like so

Start-ManagedFolderAssistant -Identity pubfoldermbx01
Posted in Exchange, Exchange 2013, Public Folders | Leave a comment