"AppFabric Caching Service" stopped working and the solution

I'd like to configure one SharePoint server to provide dedicated "Distributed Caching Service". However, during the test, the "AppFabric Caching Service" on all SharePoint servers stopped working. The error message is pasted at the end of this post, and it didn't help much.

From "windows services" management console, I noticed that "AppFabric Caching Service" was not running. Everytime I tried to start it, it stopped in a few seconds, and the same error message appeared in ULS and Windows Events log.

After lengthy trouble shooting (include re-installation of AppFabric), in the end, I think I found the solution.

The error message in Windows Events mentioned "UriFormatException", that is a good sign, which tells us that something wrong with the URL "net.tcp://spServer1:22233"

In "C:\Program Files\AppFabric 1.1 for Windows Server\DistributedCacheService.exe.config", in [configuration]->[dataCache]->[hosts]->[host], there is an attribute "name". This is the one which is used in the URL. By default, the server name is "localhost". "localhost" is supposed to be replaced by the real server name automatically. Unfortunately, in my case, the conversion didn't work well. So, I had to replace it with the real server name manually, and then remove the IP address definition in local windows hosts file, which is added by AutoSPInstaller

After that, I rebooted the SharePoint server, and "AppFabric Caching Service" works again.


========= Windows Events Log =========

Application: DistributedCacheService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.UriFormatException
Stack:
   at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartServiceCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Faulting application name: DistributedCacheService.exe, version: 1.0.4632.0, time stamp: 0x4eafeccf

Faulting module name: KERNELBASE.dll, version: 6.1.7601.18409, time stamp: 0x5315a05a

Exception code: 0xe0434352

Fault offset: 0x000000000000940d

Faulting process id: 0x20c0

Faulting application start time: 0x01d012913ac4d3e5

Faulting application path: C:\Program Files\AppFabric 1.1 for Windows Server\DistributedCacheService.exe

Faulting module path: C:\Windows\system32\KERNELBASE.dll

Report Id: 7a999bf3-7e84-11e4-8843-005056b40054

========= ULS Log =========

12/08/2014 10:02:40.06 w3wp.exe (0x1A1C)                       0x26D4 SharePoint Foundation         DistributedCache               ah24w Unexpected Unexpected Exception in SPDistributedCachePointerWrapper::InitializeDataCacheFactory for usage 'DistributedLogonTokenCache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.). Additional Information : The client was trying to co... a492d39c-784a-40a4-b74c-34bf8b195e0d
12/08/2014 10:02:40.06* w3wp.exe (0x1A1C)                       0x26D4 SharePoint Foundation         DistributedCache               ah24w Unexpected ...mmunicate with the server : net.tcp://spServer1:22233     at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, RequestBody reqBody)     at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCacheProperties(RequestBody request, IClientChannel channel)     at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName)     at Microsoft.SharePoint.DistributedCaching.SPDistributedCachePointerWrapper.InitializeDataCacheFactory()'. a492d39c-784a-40a4-b74c-34bf8b195e0d
12/08/2014 10:02:40.06 w3wp.exe (0x1A1C)                       0x26D4 SharePoint Foundation         DistributedCache               air4g Monitorable Token Cache: Failed to initialize SPDistributedSecurityTokenCache  Exception: 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.). Additional Information : The client was trying to communicate with the server : net.tcp://spServer1:22233     at... a492d39c-784a-40a4-b74c-34bf8b195e0d
12/08/2014 10:02:40.06* w3wp.exe (0x1A1C)                       0x26D4 SharePoint Foundation         DistributedCache               air4g Monitorable ... Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, RequestBody reqBody)     at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCacheProperties(RequestBody request, IClientChannel channel)     at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName)     at Microsoft.SharePoint.DistributedCaching.SPDistributedCachePointerWrapper.InitializeDataCacheFactory()     at Microsoft.SharePoint.DistributedCaching.SPDistributedCache..ctor(String name, TimeSpan timeToLive, SPDistributedCacheContainerType containerType, Boolean encryptData)     at Microsoft.SharePoint.IdentityModel.SPDistributedSecurityTokenCache..ctor(String name, TimeSpan timeToLive, SPDistributedCacheContainerType containerType, Boolean encrptyData, TimeSpan minimumTo... a492d39c-784a-40a4-b74c-34bf8b195e0d
12/08/2014 10:02:40.06* w3wp.exe (0x1A1C)                       0x26D4 SharePoint Foundation         DistributedCache               air4g Monitorable ...kenExpirationWindow)     at Microsoft.SharePoint.IdentityModel.SPDistributedSecurityTokenCacheInitializer.Init(Object state)'. a492d39c-784a-40a4-b74c-34bf8b195e0d