Custom RBAC roles in Exchange Online do not work as expected when compared to on premises Exchange servers

At our company we have created multiple paired down RBAC roles for various business units like Information security and our various regional helpdesks. As we started our journey into Office 365 in 2019, we replicated these paired down RBAC roles in Exchange Online as well. This is mostly accomplished by creating new management role from an native parent role via the new-managementrole exchange PowerShell cmdlet and then using the Remove-ManagementRoleEntry exchange PowerShell cmdlet to remove any cmdlets in that role we don’t feel fit the custom role. One of the main reasons for doing so is providing just the needed cmdlets for a given role as opposed to overpromising. MS has a few articles on how to do this and here is an example of a role we created in our on premises Exchange environment for 1^st level Exchange Server Support

New-ManagementRole –Name “EXCH 1st Level Support Database Copies” –Parent “Database Copies”
Remove-ManagementRoleEntry “EXCH 1st Level Support Database Copies\Remove-UMMailboxPolicy” -confirm:$FALSE
Remove-ManagementRoleEntry “EXCH 1st Level Support Database Copies\Remove-MailboxDatabaseCopy”-confirm:$FALSE

New-ManagementRole –Name “EXCH 1st Level Support Databases” –Parent “Databases”
Remove-ManagementRoleEntry “EXCH 1st Level Support Databases\Remove-StoreMailbox” -confirm:$FALSE
Remove-ManagementRoleEntry “EXCH 1st Level Support Databases\Remove-PublicFolderDatabase” -confirm:$FALSE
Remove-ManagementRoleEntry “EXCH 1st Level Support Databases\Remove-MailboxDatabase” -confirm:$FALSE
Remove-ManagementRoleEntry “EXCH 1st Level Support Databases\New-MailboxDatabase” -confirm:$FALSE
Remove-ManagementRoleEntry “EXCH 1st Level Support Databases\New-PublicFolderDatabase” -confirm:$FALSE

New-ManagementRole –Name “EXCH 1st Level Support Database Availability Groups” –Parent “Database Availability Groups"
Remove-ManagementRoleEntry “EXCH 1st Level Support Database Availability Groups\Remove-DatabaseAvailabilityGroupNetwork” -confirm:$FALSE
Remove-ManagementRoleEntry “EXCH 1st Level Support Database Availability Groups\Remove-DatabaseAvailabilityGroup” -confirm:$FALSE
Remove-ManagementRoleEntry “EXCH 1st Level Support Database Availability Groups\New-DatabaseAvailabilityGroupNetwork” -confirm:$FALSE
Remove-ManagementRoleEntry “EXCH 1st Level Support Database Availability Groups\New-DatabaseAvailabilityGroup” -confirm:$FALSE

New-ManagementRole –Name “EXCH 1st Level Support Exchange Servers” –Parent “Exchange Servers”
Remove-ManagementRoleEntry “EXCH 1st Level Support Exchange Servers\Remove-DatabaseAvailabilityGroupServer” -confirm:$FALSE

New-ManagementRole –Name “EXCH 1st Level Support Send Connectors” –Parent “Send Connectors”
Remove-ManagementRoleEntry “EXCH 1st Level Support Send Connectors\new-SendConnector” -confirm:$FALSE
Remove-ManagementRoleEntry “EXCH 1st Level Support Send Connectors\Remove-SendConnector” -confirm:$FALSE

New-RoleGroup –Name "EXCH 1st Level Support Exchange Tasks" –Roles “EXCH 1st Level Support Database Copies”, “EXCH 1st Level Support Databases”, “EXCH 1st Level Support Database Availability Groups”, “EXCH 1st Level Support Exchange Servers”, “EXCH 1st Level Support Send Connectors”, "Active Directory Permissions", "Transport Queues", "Monitoring", "Public Folder Replication", "Public Folders", "Disaster Recovery", "Address Lists", "Recipient Policies", "Mail Recipients", "Message Tracking", "View-Only Configuration", "Mail Enabled Public Folders", "Mail Recipient Creation", "Distribution Groups", "Security Group Creation and Membership", "Migration", "Move Mailboxes", "Mailbox Import Export", "Mailbox Search", "Legal Hold", "Retention Management" -ManagedBy EXCHAdmins -Description "EXCH 1st Level Support specific collection of cmdelts needed for Exchange management" -members "TechSupport-MSExchange"

Recently we ran into an issue with our helpdesk support staff not being able to download extended message trace reports from Exchange online after granting them ability to do so via a custom RBAC role group. When doing so they would receive a HTTP 403 error from Microsoft 365. The role group in question we created appeared to have all the necessary cmdlets to run standard and historical searches. Example of the role we created.

#Add all non destructive cmdlets from "HistoricalSearch"
New-ManagementRole -Name "US Helpdesk HistoricalSearch" -Parent "HistoricalSearch"
Get-ManagementRoleEntry "US Helpdesk HistoricalSearch\*" |Where-Object Name -in @('New-TenantExemptionInfo','Remove-PublicFolderMailboxMigrationRequest','Reset-EventsFromEmailBlockStatus','Set-EventsFromEmailConfiguration','Update-License') |
Foreach-object { Remove-ManagementRoleEntry -Identity "$($_.Role)\$($_.Name)" -Confirm:$False }

#Add all non destructive cmdlets from "Message Tracking"
New-ManagementRole -Name "US Helpdesk Message Tracking" -Parent "Message Tracking"
Get-ManagementRoleEntry "US Helpdesk Message Tracking\*" |Where-Object Name -in @('Add-AvailabilityAddressSpace','New-IntraOrganizationConnector','New-OrganizationRelationship','Remove-AvailabilityAddressSpace','Set-AvailabilityConfig','Set-IntraOrganizationConnector','Set-OrganizationRelationship','Set-UnifiedAuditSetting') |
Foreach-object { Remove-ManagementRoleEntry -Identity "$($_.Role)\$($_.Name)" -Confirm:$False }

#Add all the Get and test commands from "View-Only recipients"
New-ManagementRole -Name "US Helpdesk View-Only recipients" -Parent "View-Only recipients"
Get-ManagementRoleEntry "US Helpdesk View-Only recipients\*" |Where-Object Name -notmatch '(Get)|(Test)|(Write-AdminAuditLog)|(Start-AuditAssistant)|(Get-InboxRule)|(Start-HistoricalSearch)|(Stop-HistoricalSearch)|( Get-HistoricalSearch)'|
Foreach-object { Remove-ManagementRoleEntry -Identity "$($_.Role)\$($_.Name)" -Confirm:$False }

#Create Role Group contain all needed roles
$GroupSplat = @{
     Name = 'US Helpdesk Exchange Tasks'
     Roles = @("US Helpdesk View-Only Configuration", "US Helpdesk View-Only recipients", "US Helpdesk View Inbox Rules","US Helpdesk HistoricalSearch","US Helpdesk Message Tracking")
     ManagedBy = 'EXOAdmins'
     Description = "US specific collection of cmdlets needed for view only Exchange management"
     members = @("HelpDesk-NYC","HelpDesk-SFO","Helpdesk-BOS")
 }  New-RoleGroup @GroupSplat

In troubleshooting we created a test role with the same native roles with no cmdlets removed and the 403 issue went away. So we opened a ticket with Microsoft to assist us in finding the minimum number of cmdlets needed to download extended messages trace reports. After much back and forth what we found out was that

1. The Historical Search and Message tracking Role in Exchange Online should allow all the needed functionality, but it their current state they do not due to technical limitations when it comes propagation of all the needed permission throughout the various components of Exchange Online in order to enable the matching functionality.
   • At some point in the future, this might work as expected, but for now it does not and there are significant technical barriers to getting it to work as expected.
2. Similarly, the “View-Only Recipients” role has technical limitations if the role is paired down in any fashion. For any of the cmdlets contained in this role to work as expected it should be assigned wholly and not in a custom faction with a reduced cmdlet set.
3. Furthermore, no role in Exchange Online should be paired down as Microsoft cannot guarantee it will work as expected in Exchange Online   

So in summary, you should only use the native roles in Exchange Online and any custom versions that remove cmdlets may or may not work as expected due to the distributed nature of Exchange Online