Though we completed our Exchange 2010 migration over a year ago we still kept our Exchange 2003 server around due to some hard-coded references in internally developed apps. Once those were taken care of we shutdown our Exchange 2003 server for a few months to make sure nothing else was still referencing it. After 6 months we decided it was safe to remove it from the domain and we used the following TechNet article as a guide. Now that Exchange 2003 was gone and after upgrading some other services that were dependent on a 2003 Native Domain Functional level we decided raise the level to 2008 R2. But immediately after doing so our Exchange 2010 server started locking up every 24-36 hours. We tried restarting the Exchange services during the lock up but only a reboot would fix the issue.
When reviewing the event log on the Exchange 2010 server it appears that it could not reach either of our domain controllers (DC01 and DC02), both of which held the Global Catalog server role. After multiple attempts to reach a DC or a GC the Exchange services would shut down causing all client protocols to go down with it. The most prevalent event log entry was as follows:
Log Name: Application Source: MSExchange Autodiscover Date: 8/2/2012 6:27:01 PM Event ID: 1 Task Category: Web Level: Error Keywords: Classic User: N/A Computer: EX2010.company.net Description: Unhandled Exception "Could not find any available Global Catalog in forest company.net."
Other events that appeared related to the problem as are as follows:
Log Name: System Source: Microsoft-Windows-WAS Date: 8/2/2012 6:18:26 PM Event ID: 5011 Task Category: None Level: Warning Keywords: Classic User: N/A Computer: EX2010.company.net Description: A process serving application pool 'MSExchangeSyncAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4976'. The data field contains the error number.
Log Name: Application Source: MSExchange ADAccess Date: 8/2/2012 6:29:03 PM Event ID: 2103 Task Category: Topology Level: Error Keywords: Classic User: N/A Computer: EX2010.company.net Description: Process MAD.EXE (PID=5820). All Global Catalog Servers in forest DC=company,DC=net are not responding: DC01.company.net DC02.company.net
Log Name: Application Source: MSExchangeTransport Date: 8/2/2012 1:13:32 PM Event ID: 5020 Task Category: Routing Level: Warning Keywords: Classic User: N/A Computer: EX2010.company.net Description: The topology doesn’t contain a route to Exchange 2000 server or Exchange 2003 EX2003.company.com in Routing Group CN=First Routing Group,CN=Routing Groups,CN=First Administrative Group,CN=Administrative Groups,CN=Email,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=company,DC=net in routing tables with the timestamp 8/2/2012 5:13:32 PM DC02.company.net
Log Name: Application Source: MSExchangeTransport Date: 8/2/2012 1:13:32 PM Event ID: 5006 Task Category: Routing Level: Warning Keywords: Classic User: N/A Computer: EX2010.company.net Description: Cannot find route to Mailbox Server CN=EX2003,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=Email,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=company,DC=net CN=Test,CN=First Storage Group (EX2003),CN=InformationStore,CN=EX2003,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=Email,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=company,DC=net in routing tables with timestamp 8/2/2012 5:13:32 PM. Recipients will not be routed to this store
Log Name: Application Source: MSExchangeApplicationLog Date: 8/1/2012 1:10:42 PM Event ID: 9106 Task Category: ServicePicker Level: Error Keywords: Classic User: N/A Computer: EX2010.company.net Description: Service MSExchangeMailSubmission. Exchange topology discovery encountered an exception. Microsoft.Exchange.Data.Directory.ADTransientException: Could not find any available Domain Controller
Log Name: Application Source: MSExchange ADAccess Date: 8/1/2012 12:59:24 PM Event ID: 2501 Task Category: General Level: Error Keywords: Classic User: N/A Computer: EX2010.company.net Description: Process MSEXCHANGEADTOPOLOGY (PID=1520). The site monitor API was unable to verify the site name for this Exchange computer – Call=HrSearch Error code=80040934. Make sure that Exchange server is correctly registered on the DNS server
At first glance the issues appeared AD related, but there were no obvious issues with the domain controllers or any other services relying on AD. The only recent major AD change was raising the domain functional level. Knowing that Exchange 2003 and back cannot function when the domain functional level is higher than 2003 Native, the entries referencing our decommissioned Exchange 2003 server (EX2003) seemed to be the root cause. In searching the web for answers we stumbled across a TechNet article that lead us to the references to EX2003 in Active Directory Sites and Services under Services -> Email -> Administrative Groups –> First Administrative Group. Which needed to be deleted:
- \Email\Administrative Groups\First Administrative Group\Servers\
- \Email\Administrative Groups\First Administrative Group\Routing Groups\First Routing Group\
- \Email\Administrative Groups\First Administrative Group\Folder Hierarchies\Public Folders\
The actual removal was done in ADSI edit, which correlated to the following sections:
- CN=Configuration,DC=company,DC=NET\CN=Services\CN=Microsoft Exchange\CN=Email\CN=Administrative Groups\CN=First Administrative Group\Cn=Servers\
- CN=Configuration,DC=company,DC=NET\CN=Services\CN=Microsoft Exchange\CN=Email\CN=Administrative Groups\ CN=First Administrative Group\CN=Routing Groups\CN=First Routing Group\
- CN=Configuration,DC=company,DC=NET\CN=Services\CN=Microsoft Exchange\CN=Email\CN=Administrative Groups\CN=First Administrative Group\CN=Folder Hierarchies\Public Folders\
Once removed the issue ceased!
When I discussed the issue at the Philly Exchange User Group, resident Exchange Master Bhargav Shukla mentioned that if I had used the Exchange 2010 Deployment assistant it would have had a section detailing how to properly remove Exchange 2003 in my environment.
