Friday, June 21, 2013

The mysterious case of the missing VSS System Writer - fixed..

I am taking the time to blog this as after extensive googling for solutions, none of them worked and was nothing else for it but to troubleshoot till I finally got to the answer.. Read on Macduff..

So we had a Windows 2008 server (not R2) which upon running "vssadmin list writers" simply did not show the System Writer.. The backup team were complaining and needed it fixed.. Obviously 1st port of call was to research online and we tried all of that.. We re-registered the VSS writers, we reset the permissions as per here

We also looked at removing the number of temp files in the asp temp directories which was also identified as an issue.. None of this worked.. Stubborn mule that it was.

So at this point, we really had exhausted every online solution available and none worked.

The application event log was continuing to Event ID 8193

Volume Shadow Copy Service error: Unexpected error calling routine CreateVssExamineWriterMetadata.  hr = 0x80042311.
   Writer Exposing its Metadata
   Execution Context: Requestor
   Writer Instance ID: {83225E56-CD56-4FF3-93B6-E54CE9D3F44A}
   Writer Class Id: {E8132975-6F93-4464-A53E-1050253AE220}
   Writer Name: System Writer

And Event ID 8228

Fail to parse XML file.
Reason An invalid character was found in text content.
Line 0x00000174
Position 0x00000013
Errorcode 0xc00ce508

So as calling Microsoft was not an option, we installed the VSS SDK kit which has some excellent troubleshooting tools and creates some very handy dumps of the VSS service when it makes its calls.

So with diagnostics enabled, and logging in place, I again ran the fabled 'vssadmin list writers'...

Once the dump was complete, I was able to see this nugget in the log.

[0074504171,0x001248:0x0f30:0x1818a12e] EXIT  {CVssSafeCriticalSection::Init}: hr: 0x00000000
[0074504171,0x001248:0x0f30:0x1818a12e] ENTER {CVssExamineWriterMetadata::LoadDocument}:
[0074504203,0x001248:0x0f30:0x1818a12e] ENTER {CXMLDocument::LoadFromXML}:
[0074504203,0x001248:0x0f30:0x1818a12e] ENTER {CXMLDocument::Initialize}:
[0074504203,0x001248:0x0f30:0x1818a12e] EXIT  {CXMLDocument::Initialize}: hr: 0x00000000
[0074504312,0x001248:0x0f30:0x1818a12e] backupext\vsxml\vs_xml.cxx(0815): CXMLDocument::LoadFromXML: XML parse fails, error code 0xc00ce508, line 372 char 19, reason: An invalid character was found in text content.
[0074504312,0x001248:0x0f30:0x1818a12e] backupext\vsxml\vs_xml.cxx(0817): CXMLDocument::LoadFromXML: XML parse fails, Problem text: ilespec="mfcom.exe" filespecBackupType="3855"/>

Tucked away in the registry was an invalid path to a service which had been uninstalled at some point but never fully flushed from the registry..

Check out that illegal path filename.. :-(

As the path was invalid anyway, I removed the offending service entry from the registry, and lo and behold, the VSS System Writer sprang into life, appearing back as it should be.

So top tips learnt, use the VSS SDK kit to get diagnostics asap and the solution will hopefully reveal itself a lot quicker than it did to me.. Glad to say got there in the end though :-)