Make sure your Exchange 2003 server is fully removed before raising your AD Domain Functional Level

Though we completed our Exchange 2010 migration over a year ago we still kept our Exchange 2003 server around due to some hard-coded references in internally developed apps. Once those were taken care of we shutdown our Exchange 2003 server for a few months to make sure nothing else was still referencing it. After 6 months we decided it was safe to remove it from the domain and we used the following TechNet article as a guide. Now that Exchange 2003 was gone and after upgrading some other services that were dependent on a 2003 Native Domain Functional level we decided raise the level to 2008 R2. But immediately after doing so our Exchange 2010 server started locking up every 24-36 hours. We tried restarting the Exchange services during the lock up but only a reboot would fix the issue.

When reviewing the event log on the Exchange 2010 server it appears that it could not reach either of our domain controllers (DC01 and DC02), both of which held the Global Catalog server role. After multiple attempts to reach a DC or a GC the Exchange services would shut down causing all client protocols to go down with it. The most prevalent event log entry was as follows:

Log Name:      Application
Source:        MSExchange Autodiscover
Date:          8/2/2012 6:27:01 PM
Event ID:      1
Task Category: Web
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX2010.company.net
Description:
Unhandled Exception "Could not find any available Global Catalog in forest company.net."

Other events that appeared related to the problem as are as follows:

Log Name:      System
Source:        Microsoft-Windows-WAS
Date:          8/2/2012 6:18:26 PM
Event ID:      5011
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      EX2010.company.net
Description:
A process serving application pool 'MSExchangeSyncAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4976'. The data field contains the error number.
Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/2/2012 6:29:03 PM
Event ID:      2103
Task Category: Topology
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX2010.company.net
Description:
Process MAD.EXE (PID=5820). All Global Catalog Servers in forest DC=company,DC=net are not responding:
DC01.company.net
DC02.company.net
Log Name:      Application
Source:        MSExchangeTransport
Date:          8/2/2012 1:13:32 PM
Event ID:      5020
Task Category: Routing
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      EX2010.company.net
Description:
The topology doesn’t contain a route to Exchange 2000 server or Exchange 2003 EX2003.company.com in Routing Group CN=First Routing Group,CN=Routing Groups,CN=First Administrative Group,CN=Administrative Groups,CN=Email,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=company,DC=net in routing tables with the timestamp 8/2/2012 5:13:32 PM DC02.company.net
Log Name:      Application
Source:        MSExchangeTransport
Date:          8/2/2012 1:13:32 PM
Event ID:      5006
Task Category: Routing
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      EX2010.company.net
Description:
Cannot find route to Mailbox Server CN=EX2003,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=Email,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=company,DC=net CN=Test,CN=First Storage Group (EX2003),CN=InformationStore,CN=EX2003,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=Email,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=company,DC=net in routing tables with timestamp 8/2/2012 5:13:32 PM. Recipients will not be routed to this store
Log Name:      Application
Source:        MSExchangeApplicationLog
Date:          8/1/2012 1:10:42 PM
Event ID:      9106
Task Category: ServicePicker
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX2010.company.net
Description:
Service MSExchangeMailSubmission. Exchange topology discovery encountered an exception. Microsoft.Exchange.Data.Directory.ADTransientException: Could not find any available Domain Controller
Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/1/2012 12:59:24 PM
Event ID:      2501
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX2010.company.net
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1520). The site monitor API was unable to verify the site name for this Exchange computer – Call=HrSearch Error code=80040934. Make sure that Exchange server is correctly registered on the DNS server

At first glance the issues appeared AD related, but there were no obvious issues with the domain controllers or any other services relying on AD. The only recent major AD change was raising the domain functional level. Knowing that Exchange 2003 and back cannot function when the domain functional level is higher than 2003 Native, the entries referencing our decommissioned Exchange 2003 server (EX2003) seemed to be the root cause. In searching the web for answers we stumbled across a TechNet article that lead us to the references to EX2003 in Active Directory Sites and Services under Services -> Email -> Administrative Groups –> First Administrative Group. Which needed to be deleted:

  • \Email\Administrative Groups\First Administrative Group\Servers\
  • \Email\Administrative Groups\First Administrative Group\Routing Groups\First Routing Group\
  • \Email\Administrative Groups\First Administrative Group\Folder Hierarchies\Public Folders\

The actual removal was done in ADSI edit, which correlated to the following sections:

  • CN=Configuration,DC=company,DC=NET\CN=Services\CN=Microsoft Exchange\CN=Email\CN=Administrative Groups\CN=First Administrative Group\Cn=Servers\
  • CN=Configuration,DC=company,DC=NET\CN=Services\CN=Microsoft Exchange\CN=Email\CN=Administrative Groups\ CN=First Administrative Group\CN=Routing Groups\CN=First Routing Group\
  • CN=Configuration,DC=company,DC=NET\CN=Services\CN=Microsoft Exchange\CN=Email\CN=Administrative Groups\CN=First Administrative Group\CN=Folder Hierarchies\Public Folders\

Once removed the issue ceased!

When I discussed the issue at the Philly Exchange User Group, resident Exchange Master Bhargav Shukla mentioned that if I had used the Exchange 2010 Deployment assistant it would have had a section detailing how to properly remove Exchange 2003 in my environment.

 

Posted in Active Directory, Exchange 2003, Exchange 2010, W2K8R2 | Leave a comment
  • Archives

  • December 2012
    M T W T F S S
    « Oct   Jan »
     12
    3456789
    10111213141516
    17181920212223
    24252627282930
    31  
  • Page list