ADWS Fails to Start With "Error 1067: The Process Terminated Unexpectedly"

Greetings all, it's Eric here again, wait nope...this is the first time posting to CB5! I'm a guest blogger on the Crandall brothers' site today and I wanted to talk about an interesting case that I had regarding the ADWS service not starting on most of the DC's at one of my customers' hub site. I want to cover what kind of impact that can have, so I'll talk about what ADWS does, requirements of ADWS, how to troubleshoot the issue, and some workarounds to fix the issue.

First of all, I'll give you a high level overview of ADWS. ADWS is the Active Directory Web Service and it's a new component that shipped with Windows 2008 R2 and is installed by default when adding the ADDS or ADLDS roles. It's a web service interface for managing your Active Directory Domains, database Instances, or database mounting tools . One great thing is that all management is done over a single (non configurable) port, TCP 9389. Also, just as a side note, this service runs independently of IIS.

Due to the fact that the Active Directory Module for PowerShell uses/requires ADWS to function, a lot of things can be rendered useless if those services don't start. Some examples of that are things like the AD-Recycle Bin, the Active Directory Administrative Center, PowerGUI, certain client applications, or anything else that uses PowerShell on the back end that manages the directory.

So, as far as requirements for ADWS, they are as follows:

< 2008 R2

• 2003 or 2008 DC's with the Active Directory Management Gateway Service
• .NetFramework 3.5 SP1 (need that to be able to install the ADMGS)
• Depending on the OS that ADMGS is being installed, hotfixes from the following knowledge base articles:
  • KB969166
  • KB969429
  • KB967574

>= 2008 R2

• 2008 R2 OS
• Event Log Service.  It's a hidden dependency, but ADWS will not start without the "Windows Event Log" service being started

Both

• If certificates are involved, then ADWS needs to use a Server Authentication certificate for encryption purposes to be able to start.

In my customer's case, they had all of the basic requirements met, to include a valid Server Authentication certificate, but their ADWS service still didn't start. They got the following errors:

• When attempting to start the service they got "Error 1067: The process terminated unexpectedly."
• When rebooting, they saw ADWS Event ID 1002 in the ADWS Logs that said: "Active Directory Web Services could not initialize its endpoints. A networking error could have occurred."

OK, now what?? Now we turn on diagnostics logging for ADWS and see what's going on.

To do this we need to modify the Microsoft.ActiveDirectory.WebServices.exe.config file found in the %Windir%\ADWS directory. You'll need to add the following lines into the <AppSettings> section. Be sure that its between the <appSettings> and </appSettings> section boundaries…

<add key="DebugLevel" value="Info" />

<add key="DebugLogFile" value="c:\windows\debug\adws.log" />

Valid values for the DebugLevel value are:

0 – No logging

1 – Error (this logs critical errors only)

2 – Warn (this logs warning events as well as error events) – Recommended value to use unless you need full tracing

3 - Info (verbose)

Use strings rather than numbers, so just to be clear, type "Info" between the quotes instead of "3" for example. Once this is done, you'll see some new events trigger in the ADWS Event log, and then you'll see the ADWS.log start to populate with diagnostics info.

Note: So you don't gather too much data, just turn logging on, reproduce your issue by trying to start the service, wait for it to fail, and then turn logging back off.

When I did that I found the problem as seen in the snip below:

Well, again, the DC had a server authentication certificate, so it should have worked, so I compared the certificate store of the DC's where ADWS was failing to the one that was working. The difference was the OCSP Signing Cert… The three that were failing had the OCSP cert, and the one that was working did not.. The domain controllers that were unable to start the ADWS service also had the OCSP Responder role installed on them as well (this isn't recommended BTW). So I did a quick test by deleting the OCSP Signing cert and then trying to start the service…voila! It worked…until you rebooted the server or manually re-enrolled to pick up a new OCSP Signing cert and then tried to restart the service.. As a temporary work around what I did was delete the OCSP Signing cert, started the service and then re-enrolled, telling the customer that if they rebooted to follow my work around so everything was functional again.

Later on, I sent the customer two other more permanent work around that were better, but still not great. They are as follows:

1. Move OCSP Responders off of the DCs, so the OCSP Signing Certs go away. This is the best case scenario as it is not recommended to collocate the directory service and OCSP role.
2. Modify the “OCSP Response Signing” certificate template to add encryption to the certificates purpose. This can be done by going to the template and modifying the Request Handling.
   1. Open up the Certificate Authority, right click on Certificate Templates and select “Manage”
      1. Go to the properties of the “OCSP Response Signing” template and then to the “Request Handling” tab
      2. Change “signature” to “signature and encryption”.

In the end, the service is supposed to use the first valid server authentication certificate that it finds on startup, but I've found that in at least my one customers case, this isn't true. This appears to be an unintended behavior in the certificate selection process for the ADWS service and there's no way that I know of to manually select what it uses.

Note: In my customers case, they copied and modified the default template. Using the default template, I'm not able to reproduce this issue. Once I find the discrepancy in the modified template, I intend to update this blog entry.