This time I ran out of luck. Last time what I did was just installing windows updates and then reboot the system, but that didn't fix the problem.
Here is what I was facing:
1. "Excel" works fine, but "Word" and "PowerPoint" got error.
2. "Edit in browser" works fine, but not the "View in browser".
3. SilverLight is installed and enabled on server and client side.
4. Test documents are fine.
5. Service account have db_owner rights of the content databases;
6. Test farm works fine. The error only happened on Dev and Production farm.
7. I tried to turn on and off the "Sandboxed" settings. But nothing changed.
8. The farms are all SharePoint 2010 SP2 + CU201308.
So, what else can I do?
Well, this is a really long journey of trouble shooting. I guess most of you don't have interest to read through it. Let me show you the result first.
The problem is caused by the permission settings of "C:\Windows\System32\t2embed.dll"
For unknown reason (I guess it's caused by windows updates or anti-virus program), no user account can access this DLL file.
We need to change its owner, and then recover the permission settings, for the same file on all SharePoint servers.
At first, because this problem came from nowhere, I guess some SharePoint file was corrupted, so I moved the service to another SharePoint server. The problem was still there.
In site collection "/sites/Office_Viewing_Service_Cache", I can see the temporary files are created, but the file size is 1KB.
In ULS log, the relevant errors I noticed are:
GetCachedItem() result: ItemNotGenerated for item p_1_10.xml, document F422807d7bc2b46a48bfd7fb5c9d21922m16e96d7982d545798e3e35dd01314f6cmd60ff25cdb52489e94e180a605f1d348m
AppWorker:be19d243-dccb-48d5-aa76-c5a47398a553 response UnexpectedError sent for request fee30614-5705-4d40-abad-df636a7fdad1. Worker name WordServer, Document F422807d7bc2b46a48bfd7fb5c9d21922m16e96d7982d545798e3e35dd01314f6cmd60ff25cdb52489e94e180a605f1d348m
AppWorker:be19d243-dccb-48d5-aa76-c5a47398a553 recycle worker process because the conversion failed with result UnexpectedError. Worker is WordServer
So I started to check the windows folder.
In "C:\Windows\Temp\waccache" (and "C:\Windows\Temp\powerpointcache" for PowerPoint service), the "output.docx" was retrieved from SharePoint library, but there was no ".bundle" file.
In the sub folder, a "docdata.xml" was created, but no other files.
And they should look like below:
Not sure why the files could not be created, but it seems that "Word Viewing Service" didn't work properly. So I compared it in Dev and Test farm, nothing wrong. (This is done with the famous tool "SharePoint Manager 2010")
Then I checked the farm configurations, all OK.
Then I tested it with the help of Fiddler. There were "404" errors, which mean "could not find resource files" which suppose to be generated.
Maybe the service account doesn't have rights to access some folder or file?
I checked IIS authentication settings, and didn't see any problem.
Then I turned back to SharePoint.
I removed the server from the farm, then rejoined it. Still same.
Then I rebuilt "Word Viewing Service Application". Same.
Then I ran the scripts below and compared the result between Dev and Test farm. Could not find any problem.
$e = Get-SPServiceApplication | where {$_.TypeName.Equals("Secure Store Service Application")}
$e | Format-List *
$e = Get-SPServiceApplication | where {$_.TypeName.Equals("Security Token Service Application")}
$e | Format-List *
$e = Get-SPServiceApplication | where {$_.TypeName.Equals("ConversionService")}
$e | Format-List *
$e = Get-SPServiceApplication | where {$_.TypeName.Equals("Word Viewing Service Application")}
$e | Format-List *
$e = Get-SPServiceApplicationProxy | where {$_.TypeName.Equals("Word Viewing Service Application Proxy")}
$e | Format-List *
$e = Get-SPServiceApplication | where {$_.TypeName.Equals("Application Discovery and Load Balancer Service Application")}
$e | Format-List *
$e = Get-SPServiceApplicationProxy | where {$_.TypeName.Equals("Application Discovery and Load Balancer Service Application Proxy")}
$e | Format-List *
$e = Get-SPServiceApplication | where {$_.TypeName.Equals("PowerPoint Service Application")}
$e | Format-List *
$e = Get-SPServiceApplicationProxy | where {$_.TypeName.Equals("PowerPoint Service Application Proxy")}
$e | Format-List *
Then I repaired the SharePoint and Office Web Apps installation. Same.
In desperate, I uninstalled and then reinstalled "Office Web Apps" on Dev farm. The farm looked like been stomped by an elephant, but the problem was still there.
No choice. I started to dig into the DLL with the help of dotPeek.
In "C:\Program Files\Microsoft Office Servers\14.0\WebServices\ConversionService\Bin\Converter" and "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\WebServices\PowerPoint\Bin\Converter", we can see the components.
Below is the code I guess threw out the exception.
However, I could not debug the code to see which parameter was incorrect. I think this is a dead end.
Luckily I got the same error with PowerPoint Service. When I tested it, the errors below were caught by ULS.
Loading of the server assembly failed System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Office.Server.PowerPoint.Core.WebConversion, Version=0.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies. Access is denied. File name: 'Microsoft.Office.Server.PowerPoint.Core.WebConversion, Version=0.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' ---> System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
at System.Reflection.Assembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection)
at System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
at System.Reflection.Assembly.LoadFrom(String assemblyFile, Evidence securityEvidence)
at System.Activator.CreateInstanceFrom(String assemblyFile, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)
at Microsoft.Office.Web.Conversion.Viewing.Host.AppServerWrapper.CreateServer()
AppWorker:fbba7a81-49b5-47cd-8ffc-49e010c2b281 worker call failed System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error. Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown
at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Office.Web.Conversion.Framework.Remoting.IAppChannelCallback.Initialize(WorkerRequest request, FileItem fileItem)
at Microsoft.Office.Web.Conversion.Framework.AppWorker.ProcessRequest(ConversionRequest request). Worker name PowerPointServer, Document F25f87b4e13a2482a9382563c9b2c1861m83d394644d014e599244ea83257397feme2de2aff20cc494b8770142f561aa80dm
I double checked it. The service account have "full control" rights over the components folders. So there must be a "dependencies" dll failed the whole process.
The rest is quite simple. I downloaded Process Monitor, then tried to capture "ACCESS DENIED" error. That's how I caught the evil "t2embed.dll".
Done.
PS:
Do you like trouble shooting? :-)
No comments:
Post a Comment