Purge huge list/library through PowerShell script quickly

If there are less than 20,000 items in the list, you can run the script below directly.

If there are more than 20,000 items......well, you can call the function multiple times :-)

Please let me know if you think there is a quicker way to clean up huge list.


Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction "SilentlyContinue" 

$WebUrl = "http://server/sites/site1/web1" 

$web = Get-SPWeb -Identity $WebUrl 

function PurgeLargeList([string]$ListName) 
{ 
    $list = $web.Lists[$ListName] 

    if ($list -ne $null) 
    {
        $spQuery = New-Object Microsoft.SharePoint.SPQuery
        $spQuery.RowLimit = 20000
        $spQuery.ViewFields = ""
        $spQuery.ViewFieldsOnly = $true
        $spQuery.ViewAttributes = "Scope='Recursive'"
        $spQuery.Query = "0"
        $items = $list.GetItems($spQuery)
        $myArrayList = New-object System.Collections.Generic.List[int]
        
        foreach($item in $items)
        {
            Write-Host -NoNewline ".";
            $myArrayList.add($item.ID)
        }   

        $arraycount = $myArrayList.Count;
        for ($i = 0; $i -lt $arraycount; $i++)
        {
            Write-Host "Deleting " myArrayList[$i] $myArrayList[$i]
            $list.GetItemById($myArrayList[$i]).Delete()
        }
        
        $myArrayList.Clear()
        
        foreach ($folder in $list.Folders)
        {
            Write-Host -NoNewline ".";
            $myArrayList.add($folder.ID)
        }
        Write-Host " -> complete.";
        
        $arraycount = $myArrayList.Count;
        $folders = $list.Folders
        for ($i = 0; $i -lt $arraycount; $i++)
        {
            try {
                Write-Host -NoNewline "Deleting folder, ID: " $myArrayList[$i] " -> "
                $folders.DeleteItemById($myArrayList[$i])
            }
            catch {
                Write-Host "fail." -foregroundcolor "Red"
            }
            Write-Host "succ."
        }
        
        $myArrayList.Clear()
    } 
} 

PurgeLargeList "list title" 

$web.Dispose() 

Write-Host "Completed.  Any key to exit..." 
Read-Host 

How to cancel a workflow from custom workflow activity

When things go wrong, quite often we don't want to let SharePoint workflow engine to handle the exception. Instead of the "Error Occurred" result, it's better to cancel the workflow instance so site owners don't need to get involved to cancel it manually.

But how? "return ActivityExecutionStatus.Canceling;" doesn't work, because workflow engine knows that something is wrong, and it still end the workflow instance with "Error Occurred" status.

Actually, we cannot cancel a workflow instance from custom workflow activity, and we should not do that.

This is easy to understand. Application should not "Log Off" or "Restart" Windows OS directly. If it's necessary, the application should "tell" the users to do that. The relationship between "Workflow Activity" and "Workflow Instance" is just like the one between "Application" and "Windows".

Once this is clear, it's easy to implement the feature properly.

We should add a column such as "ActivityStatus" to the target list. After executing a workflow activity, the workflow can check this field. If the value is "Cancelled", then the workflow can cancel the workflow instance. This normally can be done through SharePoint Designer.

        //this is done through RESTful API
        private void UpdateResult(WorkflowContext __Context, string strActivityStatus, string strComments)
        {
            try
            {
                SharePointAdministrationDataContext ctx = new SharePointAdministrationDataContext(new Uri(string.Format(@"{0}/_vti_bin/listdata.svc/", __Context.CurrentWebUrl)));
                ctx.MergeOption = MergeOption.OverwriteChanges;
                ctx.Credentials = CredentialCache.DefaultCredentials;
                ModulesItem oItem = (ModulesItem)ctx.Modules.Where(i => i.Id == __Context.ItemId).FirstOrDefault();
                oItem.Comments = strComments;
                oItem.ActivityStatus = strActivityStatus;

                ctx.UpdateObject(oItem);
                ctx.SaveChanges();

            }
            catch (Exception ex)
            {
                WriteInfoToHistoryLog(__Context.Web, __Context.WorkflowInstanceId, @"ex.Message=" + ex.Message);
                WriteInfoToHistoryLog(__Context.Web, __Context.WorkflowInstanceId, @"ex.StackTrace=" + ex.StackTrace);
            }
        }

        public static void WriteInfoToHistoryLog(SPWeb web, Guid workflow, string description)
        {
            TimeSpan ts = new TimeSpan();
            SPSecurity.RunWithElevatedPrivileges(delegate()
            {
                SPWorkflow.CreateHistoryEvent(web, workflow, 0, web.CurrentUser, ts, "information", description, string.Empty);
            });
        }

Now we can catch and process the exception in workflow activity properly.

            catch (Exception ex)
            {
                WriteInfoToHistoryLog(__Context.Web, __Context.WorkflowInstanceId, @"ex.Message=" + ex.Message);
                WriteInfoToHistoryLog(__Context.Web, __Context.WorkflowInstanceId, @"ex.StackTrace=" + ex.StackTrace);
                UpdateResult(__Context, _ActivityStatus_Canceled, @"ex.Message=" + ex.Message);
                return ActivityExecutionStatus.Canceling;
            }

Hyper-V + SSD + "Data Deduplication" on Windows Server 2012

As a SharePoint developer or administrator, we all have a number ( >10 ) of virtual machines for test and development environment. If your VMs are hosted on Windows Server 2012 and stored on SSD ...... don't forget to enable "Data Deduplication"!

Why? Please check the screenshot below. 340GB vhdx files are "shrinked" to 54GB!

There are some extra cost of disk space. Let's check it from OS level, as the screenshot below, those 340GB vhdx files still takes less than 125GB disk space! What does this mean? Even a 240GB SSD is enough for a SharePoint 2013 development environment.

I am not a virtual machine expert. But if Data Deduplication is as good as here said, I believe that for most of the companies, the ESX vs Hyper-V war is over. Hyper-V wins. Let's imagine how many SSD RAID it can save for 1000 virtual machines!

This feature is really impressive. I strongly recommend it to all SharePoint gurus!

Please let me know if you have different thoughts.

PowerShell script to fix SharePoint calendar overlay issue

There is a bug in SharePoint list view property "CalendarSettings". When a site collection is restored to another web application or farm, the web URL in "CalendarSettings" still points to the previous web. The URL should be server relative, but unfortunately it's an absolute path.

(My test environment is SharePoint 2010 SP2 + CU201308, and the problem is still there.)

Thanks to the PowerShell script from Paul Ewert, we can fix the problem easily. The scripts are pretty good, and I don't see any problem there.

Below is the scripts, just to make it easier to maintain:

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction "SilentlyContinue" 

cls 

# http://spsawyer.wordpress.com/2013/07/11/sharepoint-calendar-overlay-not-found-errors/

function FixCalendarSettings([string]$oldWebApplicationURL, [string]$currentWebApplicationURL) 
{
write-host "Updating CalendarSettings..."
Start-SPAssignment –Global

write-host $views.Count " views update start...";
$views = New-Object System.Collections.ArrayList

$webApp = Get-SPWebApplication $currentWebApplicationURL
foreach ($site in $webApp.Sites) 
{
Write-Host "site URL: " $site.Url
foreach($web in $site.AllWebs) 
{
Write-Host "web URL: " $web.Url
foreach($list in $web.Lists)
{
foreach($view in $list.Views)
{
if ($view.CalendarSettings -ne $null -and $view.CalendarSettings -like '*'+$oldWebApplicationURL+'*')
{
$views.Add($view)
}
else
{
Write-Host -NoNewline ".";
}
}
}
Write-Host "";
}
}

foreach($view in $views)
{
$view.CalendarSettings = $view.CalendarSettings.Replace($oldWebApplicationURL, $currentWebApplicationURL);
$view.Update();
Write-Host "View " $view.ServerRelativeUrl " updated." -ForegroundColor Yellow;
}
write-host $views.Count " views updated.";

Stop-SPAssignment –Global
}

FixCalendarSettings "http://webapp1" "http://webapp2"

Write-Host "Finished! Press enter key to exit." -ForegroundColor Green 
Read-Host 

PowerShell script - Back up all site collections of a web application

Below is the PowerShell script to back up all site collections of a web application. We can schedule it from windows task scheduler, to run it from Monday to Friday.

I feel it's handly, because, only the latest 5 copies will be kept, which save the housekeeping work.

================

Add-PSSnapin Microsoft.SharePoint.PowerShell –ErrorAction SilentlyContinue 

#For .net 4.5, we can compress backup files through build-in components.
#Add-Type -As System.IO.Compression.FileSystem
#or else, we can do it through 7-zip or winrar

# The compression level (defaults to Optimal):
#   Optimal - The compression operation should be optimally compressed, even if the operation takes a longer time to complete.
#   Fastest - The compression operation should complete as quickly as possible, even if the resulting file is not optimally compressed.
#   NoCompression - No compression should be performed on the file.
[System.IO.Compression.CompressionLevel]$Compression = "Optimal"
    
$FileNameTime=(get-date).DayOfWeek 

$FolderPath = "E:\EricFang\Scripts\Backup\" 

function BackupWebApplication([string]$webApplicationURL) 
{ 
    $SPWebApplication = Get-SPWebApplication $webApplicationURL 
$WebApplicationName = $webApplicationURL.Replace("https://","") 
$WebApplicationName = $WebApplicationName.Replace("http://","") 
$WebApplicationName = $WebApplicationName.Replace(".","-") 

    foreach ($Site in $SPWebApplication.Sites) 
    { 
        if ($Site.ServerRelativeUrl -notmatch "Office_Viewing_Service_Cache") 
        { 
            $SitePath = $Site.ServerRelativeUrl 
            if ($SitePath -eq "/") 
            { 
                $SitePath = "" 
            } 
            else 
            { 
                $SitePath = $Site.ServerRelativeUrl.Replace("/",".") 
            } 
             
            $Filename = $FolderPath + $WebApplicationName + $SitePath + "." + $FileNameTime + ".dat" 
            Write-Host "Back up site " $Site.Url -nonewline 
            backup-spsite -identity $Site.URL -path $FileName -Force 
            Write-Host "......completed." 
            
#WinRAR, shareware
            #&$WinRar a "$FileName.rar" $FileName -df -r -m1 -ilog
            
#.Net 4.5           
            #$Archive = [System.IO.Compression.ZipFile]::Open( "$FileName.zip", "Update" )
            #$relative = (Resolve-Path $FileName -Relative).TrimStart(".\")
            #$null = [System.IO.Compression.ZipFileExtensions]::CreateEntryFromFile($Archive, $FileName, $relative, $Compression)
            #$Archive.Dispose()
            
#### http://mats.gardstad.se/matscodemix/2009/02/05/calling-7-zip-from-powershell/  
#7-zip, freeware
            #if (-not (test-path "$env:ProgramFiles\7-Zip\7z.exe")) {throw "$env:ProgramFiles\7-Zip\7z.exe needed"} 
            #set-alias sz "$env:ProgramFiles\7-Zip\7z.exe"           
            #sz a -t7z "$FileName.7z" "$FileName"
            
            #Move-Item "$FileName" "\\AnotherServer\DailyBackup" -force
            #if (Test-Path "$FileName")
            #{
            #    Remove-Item "$FileName" -force
            #}
        } 
    } 
} 

BackupWebApplication http://webapp1.domain.local 

#Write-Host "Finished! Press enter key to exit." -ForegroundColor Green 
#Read-Host

Office Web Apps - Word Viewing Error - Happened again

This time I ran out of luck. Last time what I did was just installing windows updates and then reboot the system, but that didn't fix the problem.

Here is what I was facing:

1. "Excel" works fine, but "Word" and "PowerPoint" got error.

2. "Edit in browser" works fine, but not the "View in browser".

3. SilverLight is installed and enabled on server and client side.

4. Test documents are fine.

5. Service account have db_owner rights of the content databases;

6. Test farm works fine. The error only happened on Dev and Production farm.

7. I tried to turn on and off the "Sandboxed" settings. But nothing changed.

8. The farms are all SharePoint 2010 SP2 + CU201308.

So, what else can I do?

Well, this is a really long journey of trouble shooting. I guess most of you don't have interest to read through it. Let me show you the result first.

The problem is caused by the permission settings of "C:\Windows\System32\t2embed.dll"

For unknown reason (I guess it's caused by windows updates or anti-virus program), no user account can access this DLL file.

 We need to change its owner, and then recover the permission settings, for the same file on all SharePoint servers.

Now, let me show you what I did to figure that out.

At first, because this problem came from nowhere, I guess some SharePoint file was corrupted, so I moved the service to another SharePoint server. The problem was still there.

In site collection "/sites/Office_Viewing_Service_Cache", I can see the temporary files are created, but the file size is 1KB.

In ULS log, the relevant errors I noticed are:

GetCachedItem() result: ItemNotGenerated for item p_1_10.xml, document F422807d7bc2b46a48bfd7fb5c9d21922m16e96d7982d545798e3e35dd01314f6cmd60ff25cdb52489e94e180a605f1d348m

AppWorker:be19d243-dccb-48d5-aa76-c5a47398a553 response UnexpectedError sent for request fee30614-5705-4d40-abad-df636a7fdad1. Worker name WordServer, Document F422807d7bc2b46a48bfd7fb5c9d21922m16e96d7982d545798e3e35dd01314f6cmd60ff25cdb52489e94e180a605f1d348m

AppWorker:be19d243-dccb-48d5-aa76-c5a47398a553 recycle worker process because the conversion failed with result UnexpectedError. Worker is WordServer

So I started to check the windows folder.

In "C:\Windows\Temp\waccache" (and "C:\Windows\Temp\powerpointcache" for PowerPoint service), the "output.docx" was retrieved from SharePoint library, but there was no ".bundle" file.

In the sub folder, a "docdata.xml" was created, but no other files.

And they should look like below:

Not sure why the files could not be created, but it seems that "Word Viewing Service" didn't work properly. So I compared it in Dev and Test farm, nothing wrong. (This is done with the famous tool "SharePoint Manager 2010")

Then I checked the farm configurations, all OK.

Then I tested it with the help of Fiddler. There were "404" errors, which mean "could not find resource files" which suppose to be generated.

Maybe the service account doesn't have rights to access some folder or file?

I checked IIS authentication settings, and didn't see any problem.

Then I turned back to SharePoint.

I removed the server from the farm, then rejoined it. Still same.

Then I rebuilt "Word Viewing Service Application". Same.

Then I ran the scripts below and compared the result between Dev and Test farm. Could not find any problem.

$e = Get-SPServiceApplication | where {$_.TypeName.Equals("Secure Store Service Application")}
$e | Format-List *

$e = Get-SPServiceApplication | where {$_.TypeName.Equals("Security Token Service Application")}
$e | Format-List *

$e = Get-SPServiceApplication | where {$_.TypeName.Equals("ConversionService")}
$e | Format-List *

$e = Get-SPServiceApplication | where {$_.TypeName.Equals("Word Viewing Service Application")}
$e | Format-List *

$e = Get-SPServiceApplicationProxy | where {$_.TypeName.Equals("Word Viewing Service Application Proxy")}
$e | Format-List *

$e = Get-SPServiceApplication | where {$_.TypeName.Equals("Application Discovery and Load Balancer Service Application")}
$e | Format-List *

$e = Get-SPServiceApplicationProxy | where {$_.TypeName.Equals("Application Discovery and Load Balancer Service Application Proxy")}
$e | Format-List *

$e = Get-SPServiceApplication | where {$_.TypeName.Equals("PowerPoint Service Application")}
$e | Format-List *

$e = Get-SPServiceApplicationProxy | where {$_.TypeName.Equals("PowerPoint Service Application Proxy")}
$e | Format-List *

Then I repaired the SharePoint and Office Web Apps installation. Same.

In desperate, I uninstalled and then reinstalled "Office Web Apps" on Dev farm. The farm looked like been stomped by an elephant, but the problem was still there.

No choice. I started to dig into the DLL with the help of dotPeek.

In "C:\Program Files\Microsoft Office Servers\14.0\WebServices\ConversionService\Bin\Converter" and "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\WebServices\PowerPoint\Bin\Converter", we can see the components.

Below is the code I guess threw out the exception.

However, I could not debug the code to see which parameter was incorrect. I think this is a dead end.

Luckily I got the same error with PowerPoint Service. When I tested it, the errors below were caught by ULS.

Loading of the server assembly failed System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Office.Server.PowerPoint.Core.WebConversion, Version=0.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies. Access is denied.  File name: 'Microsoft.Office.Server.PowerPoint.Core.WebConversion, Version=0.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' ---> System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))   
 at System.Reflection.Assembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection)   
 at System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)   
 at System.Reflection.Assembly.LoadFrom(String assemblyFile, Evidence securityEvidence)   
 at System.Activator.CreateInstanceFrom(String assemblyFile, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)   
 at Microsoft.Office.Web.Conversion.Viewing.Host.AppServerWrapper.CreateServer()

AppWorker:fbba7a81-49b5-47cd-8ffc-49e010c2b281 worker call failed System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.    Server stack trace:    
 at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)   
 at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)   
 at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown
 at [0]:    
 at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)   
 at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)   
 at Microsoft.Office.Web.Conversion.Framework.Remoting.IAppChannelCallback.Initialize(WorkerRequest request, FileItem fileItem)   
 at Microsoft.Office.Web.Conversion.Framework.AppWorker.ProcessRequest(ConversionRequest request). Worker name PowerPointServer, Document F25f87b4e13a2482a9382563c9b2c1861m83d394644d014e599244ea83257397feme2de2aff20cc494b8770142f561aa80dm

I double checked it. The service account have "full control" rights over the components folders. So there must be a "dependencies" dll failed the whole process.

The rest is quite simple. I downloaded Process Monitor, then tried to capture "ACCESS DENIED" error. That's how I caught the evil "t2embed.dll".

Done.


PS:

Do you like trouble shooting? :-)

Error: Content Plugin can not be initialized - list of CSS addresses is not set

AutoSPInstaller released a new version last month. To test it, I reinstalled SharePoint 2013 farm (leave the farm, then delete all SharePoint databases). It's almost perfect, except one error:


topology activation failed. no system manager locations set, search application might not be ready yet


Dig into it, I found the error was thrown out by "$clone.Activate()" of "AutoSPInstallerFunctions.ps1".

However, I don't see anything wrong there.

In ULS, there are a lot of error message like below:


Content Plugin can not be initialized - list of CSS addresses is not set

Failed to extract required parameter FastConnector:ContentDistributor, hr=0x80070002  [pluginconfig.cpp:81]  search\native\gather\plugins\contentpi\pluginconfig.cpp

Unable to get systemmanagerlocation from db

Exception stack trace:   
 at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.ExecuteTimerJob()    
 at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)

I am not the first one who got this error.

To confirm the problem is caused by AutoSPInstaller, I reinstalled the farm, but initialized the farm through "SharePoint Products Configuration Wizard". Same error happened again. So, AutoSPInstaller may not be the one we should blame.

After quite a while research, I realized this is not something easy to fix. It may caused by some file or folder created by previous installation, which don't allow current installed components to access it; or some settings pointed to an invalid server.

I don't want to dig out the root cause, but to fix it. So I uninstalled SharePoint, then deleted the folder "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server", then reinstalled SharePoint.

Finally, everything works now.

New tools to help to manage site collections

I always use site collection as the basic unit to build SharePoint platform. So, if possible, I would create separate site collection instead of sub site when new site request pop up.

Inevitable, there will be a lot of site collections in the farm.

That's all right. We keep millions of records in SQL table, and it's much better than keeping them in thousands of files.

But sometimes it's annoying when we need to create, backup, restore or move site collections through PowerShell scripts. That's why I built this tool.

It's easy to write the PowerShell scripts to do the work. However, as a SharePoint administrator, I need to do that many times every week. In that case, this tool is handy.

Give it a try! It doesn't change anything of the SharePoint server, and just generate some PowerShell scripts, which you can run from PowerShell console.

Please let me know if you find anything can be improved, or want to add more functionalities to it.

=============================

SharePoint 2013. Winform tool to generate PowerShell scripts, to move, create, backup and restore site collections.
https://spsiteadmin2013.codeplex.com/

SharePoint 2010. Winform tool to generate PowerShell scripts, to move, create, backup and restore site collections.
https://spsiteadmin2010.codeplex.com/

[update, 2017-02-04]

https://github.com/Eric-Fang/SPSiteAdmin2016

https://github.com/Eric-Fang/SPSiteAdmin2013

==============================

Screenshots:

Crawling component stopped working: Access is denied

A user reported that he could not do any search from a SharePoint site. I checked the log, and found the error message below.

Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled

It worked before, and other web applications all work well. Anyway, I doubled checked:

1. The rights of Default Content Access Account over the web application ("Full read");
2. DisableLoopbackCheck problem;
3. "Basic authentication" issue;
4. Change content access account;

When "start full crawl", the same error is always there.

I think something crashed. So I added one more "Crawl Component", and then deleted the old one.

[update 2013-09-24]
Yeah, the problem disappeared.

Unfortunately, although the crawling went well, the query didn't.

I checked the whole web application features through FeatureAdmin, nothing wrong there.

There is only one content database, so I think maybe something wrong with the content database. Test-SPContentDatabase gave me the message below:

Test-SPContentDatabase : Object reference not set to an instance of an object. At line:1 char:23
+ Test-SPContentDatabase <<<<  -Identity 4b146bda-9a6e-4ef7-8ed1-eb073ca0bccb
    + CategoryInfo          : InvalidData: (Microsoft.Share...ContentDatabase:    SPCmdletCheckContentDatabase) [Test-SPContentDatabase], NullReferenceException
    + FullyQualifiedErrorId : Microsoft.SharePoint.PowerShell.SPCmdletCheckContentDatabase

I spent a while, and didn't know how to fix it. It seems something wrong with the schema, but don't know what is wrong. So I backed up all site collections of that web application, then rebuilt the web application and did another full crawl.

Guess what? The problem was still there!

So I did an index reset, then, ok, the problem was still there!!

After a few days struggling, almost by accident, I rebuilt the web application, then did an index reset immediately.

That fixed the problem.

I know it doesn't make sense, but anyway, it works now.

( The SharePoint 2010 farm has got SP2 and CU201308 )

Reference link: http://wingleungchan.blogspot.com.au/2011/11/access-is-denied-when-crawling-despite.html

Error: New-SPEnterpriseSearchServiceApplication : A SharePoint database named xxxx already exists

The SharePoint 2010 "Search Service Application" stopped working.

I could not  crawl the data, and I could not modify the topology. The data crawling procedure was suspended there forever.

So I decided to rebuild it. Obviously this service application is corrupted, that's why I could not remove it from Central Admin site. The PowerShell seems all right.

1. Use "Get-SPEnterpriseSearchServiceApplication" to get the GUID of the search service application.
2. Use "stsadm -o deleteconfigurationobject -id [GUID]" to delete it.
3. Use "Get-SPEnterpriseSearchServiceInstance" to get the GUID of the search service instances.
4. Use "Stop-SPEnterpriseSearchServiceInstance -id [GUID]" to stop the service instances.

Now we need to rebuild it. To avoid GUID in database name, we have to use PowerShell (again). It's not easy. Lucky that Jeremy Jameson shared an excellent post about how to build it through PowerShell script

All seems OK, except the error below when running "New-SPEnterpriseSearchServiceApplication"

New-SPEnterpriseSearchServiceApplication : A SharePoint database named SP_EnterpriseSearch already exists.  You must supply another name for the new database.
At line:32 char:62
+         $searchApp = New-SPEnterpriseSearchServiceApplication <<<<  -Name $serviceAppName `
+ CategoryInfo          : InvalidData: (Microsoft.Offic...viceApplication: NewSearchServiceApplication) [New-SPEnterpriseSearchServiceApplication], SPDuplicateObjectException
+ FullyQualifiedErrorId : Microsoft.Office.Server.Search.Cmdlet.NewSearchServiceApplication

I double checked that the database "SP_EnterpriseSearch" was removed with "Search Service Application", so, what was wrong?

I had to assume this was caused by the "search service" corruption + SharePoint bug. There must be some orphaned objects left in the system.

So I removed the half-created "search service application", and then start to scan the database. With the help from Sorna Kumar Muthuraj, I searched the SharePoint Configuration database:

EXEC SP_SearchTables @Tablenames = '%', @SearchStr  = '%SP_EnterpriseSearch%'

Then I deleted all relevant objects (and their child objects) from table "SP_Config.dbo.Objects". In my case, it's totally 4 objects.

That's it.  The "Search Service Application" was created successfully after that.

CU installation error - "There are no products affected by this package installed on this system"

The installation of SharePoint 2013 CU201308 was fine. But when I ran the SharePoint 2013 Products Configuration Wizard on the first web front end server, I got error message complaining that Hotfix KB2817616 was missing on one of the servers.

I thought I missed that server, but when I tried to install SharePoint 2013 CU201308 on that server, it said "There are no products affected by this package installed on this system".

I tried to Clearing the Configuration Cache, restarted the timer service, but nothing worked.

In terms of Cumulative Patch installation rule, there is no way to uninstall or re-install a CU patch. So I stuck there.

So this server have to be sacrificed to this CU installation. I disconnected this server from the farm, and then upgraded the farm successfully.

It won't hurt to give it a last chance. After join the server back to the farm, it's upgraded without any problem!

No sure what's the problem, but anyway, it works now.

[update, 20140106]

We can disconnect the server from the farm through PowerShell command "Disconnect-SPConfigurationDatabase –Confirm:$false", reference: http://technet.microsoft.com/en-us/library/ff607702(v=office.14).aspx

SharePoint - How to shrink "UsageAndHealthDB" database

Two months ago, our SharePoint farm got some "Usage Logging" issue (thanks for a windows update pack). After resolving the issue, a huge database "SP_UsageAndHealthDB" was left there.

 

In the database, we can see the space is consumed by "ULSTraceLog_Partition" tables.

So I went to Central Admin -> Monitoring -> Configure usage and health data collection, changed the "Maximum log file size" to 1GB.  I thought it would affect both file system and logging database, but I was wrong. The logging database was not affected at all. Two months later, all historical log data is still there.

I just cannot find any solution online. So, I have to remove the data manually.

select MAX([LogTime]), MIN([LogTime]),COUNT(1)
 FROM [SP_UsageAndHealthDB].[dbo].[ULSTraceLog_Partition3]

truncate table [SP_UsageAndHealthDB].[dbo].[ULSTraceLog_Partition3]

So far so good.

I totally agree that we should not modify any data directly from SQL Server. But, it seems that we don't have any other choice.

Please let me know if you can remove the log data from SharePoint platform, that will be strongly appreciated.

Error Creating Control - Cannot find Web Project Item

I created a new SharePoint 2010 Visual Web Part project in Visual Studio 2012, and got the error message below when open a .ascx file in design mode:

Error Creating Control - LinkButtonSubmitCannot find web project item '~/sites/Sandpit/vwpUserProfileUpdate/vwpUserProfileUpdate.ascx'.

Warning message is:

Warning    1    E:\EricFang\VisualStudio\somepath\usercontrol.ascx: ASP.NET runtime error: Path cannot be null.
Parameter name: path    E:\EricFang\VisualStudio\somepath\usercontrol.ascx    1    1    usercontrol

I even could not drag and drop any control from the tool box to the web form.

I built some similar projects before, and this was the first time that got this error. So I compared the .csproj file with other projects, but didn't notice any difference which cause the problem.


Then I spent a while searching on internet. Many developers got similar issues, such as here and here, but they don't suit my case.

In the end, quite lucky, I noticed that in the "Site URL" of the project, I used the server alias name instead of the computer name. Since it wouldn't hurt, I changed it, and BING! The problem disappeared!

(This value is actually stored in ".csproj.user" file)

SharePoint error: "Failed to find ACL for ScopeId: b19c18cd-34a7-4c2d-aca7-557039b3a156"

I got this error message from ULS log when trying to download an image file from a document library. Don't see any other problem and just cannot download the image file.

It turns out quite simple: SharePoint generated a lot of "Policy Usage Report" files (15k+) in "UsageReports" document library. After deleting these files, the problem disappeared.

Below are the error message I got from the web page.


Server Error in '/' Application.
--------------------------------------------------------------------------------

Cannot complete this action.

Please try again.0x80004005 
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: System.Runtime.InteropServices.COMException: Cannot complete this action.

Please try again.0x80004005

Source Error: 

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.  

Stack Trace: 

[COMException (0x80004005): Cannot complete this action.

Please try again.0x80004005]
   Microsoft.SharePoint.Library.SPRequestInternalClass.GetAclForScope(String bstrWebUrl, Guid guidScopeId, Boolean fRequirePermissionCheck, Object& pvarAcl, UInt64& lAnonymousMask) +0
   Microsoft.SharePoint.Library.SPRequest.GetAclForScope(String bstrWebUrl, Guid guidScopeId, Boolean fRequirePermissionCheck, Object& pvarAcl, UInt64& lAnonymousMask) +183

[SPException: Cannot complete this action.

Please try again.]
   Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx) +27861714
   Microsoft.SharePoint.Library.SPRequest.GetAclForScope(String bstrWebUrl, Guid guidScopeId, Boolean fRequirePermissionCheck, Object& pvarAcl, UInt64& lAnonymousMask) +28084291
   Microsoft.SharePoint.SPReusableAcl..ctor(SPRequest request, String webUrl, Guid scopeId, Boolean requirePermissionCheck) +74
   Microsoft.SharePoint.SPSite.GetReusableAclForScope(Guid scopeId) +98
   Microsoft.SharePoint.Publishing.<>c__DisplayClass3f.b__39() +203
   Microsoft.SharePoint.<>c__DisplayClass4.b__2() +729
   Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode) +26712342
   Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param) +27759977
   Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode) +93
   Microsoft.SharePoint.Publishing.BlobCache.HandleCachedFile(HttpContext context, BlobCacheEntry target, Boolean anonymousUser, SiteEntry currentSiteEntry) +2184
   Microsoft.SharePoint.Publishing.BlobCache.RewriteUrl(Object sender, EventArgs e, Boolean preAuthenticate) +3917
   Microsoft.SharePoint.Publishing.PublishingHttpModule.AuthorizeRequestHandler(Object sender, EventArgs ea) +56
   System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +171

--------------------------------------------------------------------------------
Version Information: Microsoft .NET Framework Version:2.0.50727.5466; ASP.NET Version:2.0.50727.5456 

[update 20130701]

1. Another possible The solution is to reset all cached blob files. We can find the file location by this post, and then delete all sub-folders there.
2. In terms of the post here, you can try to flush the object cache if the fix above doesn't work.

SharePoint 2013 - PowerShell - How to update image url of web part pages

In SharePoint 2010, it's easy to update all "sub site logo url" and "web part page image url", below is the PowerShell script to update one Web Application.

$ImageUrl = "/_layouts/images/CompanyName/logo.png";

function UpdateSubsiteLogoUrl([string]$webApplicationURL) 
{ 
Write-Host "webApplication URL: " $webApplicationURL

$webApp = Get-SPWebApplication $webApplicationURL 
foreach($site in $webApp.Sites) 
{ 
Write-Host "site URL: " $site.Url
foreach($CurrentWeb in $site.AllWebs) 
{ 
[Boolean]$isContextNull = $false;
[System.Web.HttpContext]$htc = $null;
[System.IO.StringWriter]$sw = $null;
[System.Web.HttpResponse]$resp = $null;
[System.Web.HttpRequest]$req = $null;

Write-Host "CurrentWeb URL: " $CurrentWeb.Url
$CurrentWeb.SiteLogoUrl = $ImageUrl;
$CurrentWeb.Update();

# http://social.msdn.microsoft.com/Forums/en-US/sharepointdevelopment/thread/607f3980-53c0-4007-bb34-9969ed6ea4fc
$web = $CurrentWeb

            if ($null -eq [System.Web.HttpContext]::Current){
                Write-Debug "Setting HttpContext";
                $isContextNull = $true;
                $sw = New-Object System.IO.StringWriter;
                $resp = New-Object System.Web.HttpResponse $sw;
                $req = New-Object System.Web.HttpRequest "", $web.Url, "";
                $htc = New-Object System.Web.HttpContext $req, $resp;
                $htc.Items["HttpHandlerSPWeb"] = $web -as [Microsoft.SharePoint.SPweb];
                [System.Web.HttpContext]::Current = $htc;
                if ($sw -ne $null){
                    $sw.Dispose()
                }
            } else {
                Write-Debug "HttpContext already set" -ForegroundColor Red ;
            }

foreach($list in $web.Lists) 
{
if($list.Title -match "Pages" -and -not $list.Hidden)
{
#Write-Host "list.Title = " $list.Title
foreach ($currentPage in $list.RootFolder.Files)
{
if ($currentPage.Name -match ".aspx")
{
Write-Host "currentPage.ServerRelativeUrl = " $currentPage.ServerRelativeUrl
$CurrentWebPartManager = $currentPage.GetLimitedWebPartManager([System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared);

if ($CurrentWebPartManager -ne $null)
{
$CurrentWebPartManager.WebParts | ForEach-Object {
if ($_.Title -eq "Web Part Page Title Bar")
{
$_.Image = $ImageUrl;
$CurrentWebPartManager.SaveChanges($_);
}
}
}
}
}
}
}

if($isContextNull){
                Write-Debug "Resetting HttpContext";
                [System.Web.HttpContext]::Current = $null;
$isContextNull = $false;
            }
           
            $web.Dispose()

}
$site.Dispose();
}
}

However, it doesn't work for SharePoint 2013.

The "fake" httpcontext trick stopped working in SharePoint 2013. All webparts appear as ErrorWebParts now, so we can't modify the the property of "Web Part Page Title Bar" web parts of web part pages.

We know that for SharePoint 2013 upgrade, the farm system folder "/_layouts/images" is changed to "/_layouts/15/images", so how can we do the changes for all web part pages?

The only solution I found is to abandon all references of "SharePoint farm system folder".  It make sense.  In the future, most of web sites will be hosted in cloud. There is no "system folder" of cloud (for developers and SharePoint administrators).

What I did is quite simple.

1. Create a document library for each web application, say "SharedAssets", in the root site of the root site collection;

2. Grant "read" rights of that document library to all users;

3. Change the image url in PowerShell script.

$ImageUrl = "/SharedImages/logo.png";

4. Update all web applications before upgrading the sites.

5. Done.

This is not just for image files. All css, javascript files, etc. should be stored there if they are previously stored in farm system folder.

Please let me know if you know how to change web part property through PowerShell script in SharePoint 2013, I really appreciate it.

SharePoint 2010 patch could not be installed. Error code 1603

Recently I rebuilt a WFE server. SharePoint 2010 SP1 was installed successfully.  However, when trying to install CU 201304, I got the error message below:

MSI (s) (7C:CC) [13:46:54:523]: Product: Microsoft Search Server 2010 Core - Update 'Hotfix for Microsoft Project Server 2010 (KB2775426) 64-Bit Edition' could not be installed. Error code 1603. Additional information is available in the log file C:\Users\#SPSET~1\AppData\Local\Temp\2\coreserver-x-none_MSPLOG.LOG.

MSI (s) (7C:CC) [13:46:54:523]: Windows Installer installed an update. Product Name: Microsoft Search Server 2010 Core. Product Version: 14.0.6029.1000. Product Language: 0. Manufacturer: Microsoft Corporation. Update Name: Hotfix for Microsoft Project Server 2010 (KB2775426) 64-Bit Edition. Installation success or error status: 1603.

MSI (s) (7C:CC) [13:46:54:523]: Note: 1: 1729 

MSI (s) (7C:CC) [13:46:54:523]: Note: 1: 2205 2:  3: Error 

MSI (s) (7C:CC) [13:46:54:523]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1729 

MSI (s) (7C:CC) [13:46:54:523]: Note: 1: 2205 2:  3: Error 

MSI (s) (7C:CC) [13:46:54:523]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709 

MSI (s) (7C:CC) [13:46:54:523]: Product: Microsoft Search Server 2010 Core -- Configuration failed.

MSI (s) (7C:CC) [13:46:54:523]: Windows Installer reconfigured the product. Product Name: Microsoft Search Server 2010 Core. Product Version: 14.0.6029.1000. Product Language: 0. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 1603.

There was not much useful information in the error log. So I enabled windows installer logging, and got a lot of message like below.

Product: Microsoft Search Server 2010 Core. The application tried to modify a protected Windows registry key \Software\Classes\CLSID\{3050F667-98B5-11CF-BB82-00AA00BDCE0B}\InprocServer32.

Quite clear that the SharePoint patch installer doesn't have the rights to modify registry table. But, I had logged on with local administrator rights.   So I tried to change the registry table manually, and got the error message below.

The permission settings of those registry entries are here:

One possible cause is Windows UAC.  However, it's not this case. The UAC is disabled on local computer.

Not sure how this problem was raised.  I had installed SharePoint server patches many times cross several farms. This was the first time got that issue.

I contacted network administrators, and confirmed there was no group policy restriction.

To fix the problem is easy.  I downloaded a free tool DevxExec, then ran the command "devxexec.exe /user:TrustedInstaller E:\Setup\Microsoft\SharePoint\2010\ServicePack\CU201304\ubersrvprj2010-kb2775426-fullfile-x64-glb.exe".

As a quick fix, I am happy about the result.  :-)

How to repair "Distributed Cache" service in SharePoint 2013

I got the error below when trying to start up service "Distributed Cache".  After quite a while investigation, I believe it's corrupted.

Call to OpenService(...,SERVICE_START | SERVICE_QUERY_STATUS | SERVICE_STOP | SERVICE_PAUSE_CONTINUE) function to get handle to the service failed (0X424=1060)

What I need is to reinstall it in the farm.

Below is how I did it.

1. Uninstall the components "AppFabric 1.1 for Windows Server" and "Windows Fabric"

2. Reinstall SharePoint 2013 prerequisites.

3. Run SharePoint 2013 configuration wizard on all SharePoint servers

4. Remove and then add "Distributed Cache" service through PowerShell script on all Servers

Remove-SPDistributedCacheServiceInstance

Add-SPDistributedCacheServiceInstance

( In some cases, we might have to delete the service by GUID first )

5. Done.

People search returns 404 page not found - Sharepoint 2010

I got this error in a new web application.  Google leads me to here and here, but none of them fit my case. I had configured the "My Site Settings" in "Central Administration, User Profile Service Application", and the "Search settings" in "Site Collection Administration".

Then I noticed that the search centre site was a sub site of the root site collection, in the original web application. It looked strange. So I created a dedicated site collection based on "Enterprise Search Centre" template, and then reconfigured the relevant search settings.

BING! The problem was fixed.  :-)

Error: The specified module 'WorkflowManager' was not loaded because no valid module file was found in any module directory

All of sudden, on one of the development server, the "Workflow Manager PowerShell" stopped working.  When trying to start it, I got the error message below.

Import-Module : The specified module 'WorkflowManager' was not loaded because no valid module file was found in any module directory.
At C:\Program Files\Workflow Manager\1.0\Scripts\ImportWorkflowModule.ps1:20
char:1
+ Import-Module WorkflowManager
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (WorkflowManager:String) [Import-Module], FileNotFoundException
    + FullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Comm
   ands.ImportModuleCommand

Lucky I have another workflow server which works well. It turns out the "AppFabric server" or "Workflow Manager 1.0" installation & uninstallation caused this issue. The window environment variable "PSModulePath" was changed from

C:\Windows\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\PowerShell\Modules\;C:\Program Files\Service Bus\1.0\;C:\Program Files\Workflow Manager\1.0

to

C:\Windows\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\PowerShell\Modules\;C:\Program Files\Service Bus\1.0\;C:\Program Files\Workflow Manager\1.0"

After removing the double quote at the end of the environment variable value, "Workflow Manager PowerShell" is recovered!



PS: if got error message below during the installation of AppFabric server, possibly we just need to do the same change to window environment variable "PSModulePath".

Call to OpenService(...,SERVICE_START | SERVICE_QUERY_STATUS | SERVICE_STOP | SERVICE_PAUSE_CONTINUE) function to get handle to the service failed (0X424=1060)

The best way to build InfoPath forms

For most of the InfoPath form system, the development of forms is always easy, even if there is some coding got involved.  The real problem is about form maintenance.

Let's imagine that you, as a SharePoint administrator or a InfoPath form developer, is in charge of 500 forms. Each form, on average, needs to be changed twice a year. So you need to modify 500 * 2 / 220 (work days) = 4.5 (forms per day).

What would happen if there are 5000 forms?

Based on my experience, most of the changes don't require to change the C# code. From technical point of view, they are just cosmetic changes. However, since C# code is part of the forms, developer have to do the changes! Or, do they?

Below is how I handle this situation. It minimizes the work of SharePoint administrator (or InfoPath developer).

1. Install InfoPath program on business users' computer.
.Net framework support is needed.

2. All forms and C# code are stored on SharePoint development server

3. Move form relevant C# code into a separate project.
Normally we need a dedicated project for each InfoPath system (a SharePoint site collection).
This project is a standard SharePoint feature (solution), which shared by all relevant forms of one project.

4. Share the form folder and C# assembly folder to relevant business users.
Business users can modify forms, but can only view C# assembly folder.

5. In form project, call the methods of that shared assembly to implement functionality.

6. Business users can modify the form whenever they want, and then deploy to another shared folder.
They can change all stuff except coding, which include rules, validation, data connection, text, views, etc.

7. Once they completed the changes, they can send an request to SharePoint administrator to deploy the changed forms to development environment or test environment.

8. SharePoint administrator deploy the new forms, then ask business users to test them.

Normally through PowerShell script.

9. Once the test is passed, these forms are ready to be deployed to Production.

What do you think of this procedure? Any comments are appreciated.

SharePoint 2010 "Out of the Box" Workflow intermittent error: "Failed to Start"

When copy data from MS Excel sheet to SharePoint list datasheet view, some items triggered workflow successfully, but some got "Failed to Start" error.  For the failed ones, if I delete them and try again, normally the errors disappeared.


There are one application server and two web front servers in the SharePoint farm. NLB (network load balancer) is created for the two web front end servers. The "Microsoft SharePoint Foundation Workflow Timer Service" service is running on the application server.




Let's keep this post short. I started the workflow timer service on all servers, then the problem is resolved.


I think there is a bug with SharePoint timer service (SharePoint 2010 + sp1 + CU201206), and it cannot handle NLB requests properly.

SharePoint 2013 Workflow trouble shooting

Error: "Something went wrong.  To try again, reload the page and then start the workflow again"

Environment: Windows Server 2012, SharePoint 2013 RTM + PU 201303, Workflow manager 1.0 + CU1

I configured workflow manager 1.0 for one SharePoint 2013 farm. It worked well.  However, when I tried to do similar configuration for another SP2013 farm (belongs to another domain), I got the error message above. Blow is the details in SharePoint log.

Exception returned from back end service. System.ServiceModel.Security.MessageSecurityException: The HTTP request was forbidden with client authentication scheme 'Anonymous'. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.    
 at System.Net.HttpWebRequest.GetResponse()    
 at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     -
 -- End of inner exception stack trace ---    Server stack trace:     
 at System.ServiceModel.Channels.HttpChannelUtilities.ValidateAuthentication(HttpWebRequest request, HttpWebResponse response, WebException responseException, HttpChannelFactory`1 factory)    
 at System.ServiceModel.Channels.HttpChannelUtilities.ValidateRequestReplyResponse(HttpWebRequest request, HttpWebResponse response, HttpChannelFactory`1 factory, WebException responseException, ChannelBinding channelBinding)    
 at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)    
 at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)    
 at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)    
 at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)    
 at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)    
 at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown
 at [0]:     
 at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)    
 at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)    
 at Microsoft.SharePoint.Taxonomy.IMetadataWebServiceApplication.GetServiceSettings(Guid rawPartitionId)    
 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.<>c__DisplayClass32.b__31(IMetadataWebServiceApplication serviceApplication)    
 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.<>c__DisplayClass2f.b__2d()

Search engine told me that it might caused by incorrect web application authentication mode. The PowerShell command "convert-spwebapplication -identity http://mywebapp -To Claims -RetainPermission" doesn't work well, so I rebuilt the web application with "Claims" authentication.

Now, I can create SharePoint 2013 workflow in SharePoint Designer 2013. But, when trying to trigger that workflow, the workflow status was kept as "Started" for a few minutes, then been turned into "Suspended".

Below is the error message from SharePoint log file.

Original error: System.UnauthorizedAccessException: Attempted to perform an unauthorized operation.   
 at Microsoft.SharePoint.SPGlobal.HandleUnauthorizedAccessException(UnauthorizedAccessException ex)   
 at Microsoft.SharePoint.SPSecurableObject.CheckPermissions(SPBasePermissions permissionMask)   
 at Microsoft.SharePoint.Client.SPClientServiceHost.OnBeginRequest()   
 at Microsoft.SharePoint.Client.Rest.RestService.ProcessQuery(Stream inputStream, IList`1 pendingDisposableContainer)

SocialRESTExceptionProcessingHandler.DoServerExceptionProcessing - SharePoint Server Exception [System.UnauthorizedAccessException: Attempted to perform an unauthorized operation.   
 at Microsoft.SharePoint.SPGlobal.HandleUnauthorizedAccessException(UnauthorizedAccessException ex)   
 at Microsoft.SharePoint.SPSecurableObject.CheckPermissions(SPBasePermissions permissionMask)   
 at Microsoft.SharePoint.Client.SPClientServiceHost.OnBeginRequest()   
 at Microsoft.SharePoint.Client.Rest.RestService.ProcessQuery(Stream inputStream, IList`1 pendingDisposableContainer)]

Throw UnauthorizedAccessException instead of SPUtilityInternal.Send401 for client.svc request.

Application error when access /_vti_bin/client.svc, Error=Cannot redirect after HTTP headers have been sent. 
 at System.Web.HttpResponse.Redirect(String url, Boolean endResponse, Boolean permanent)   
 at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()   
 at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

On workflow server, I got these error information from windows events log.

Faulting application name: Microsoft.Workflow.ServiceHost.exe, version: 1.0.20922.0, time stamp: 0x505e1b24
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18015, time stamp: 0x50b8479b
Exception code: 0xe0434352
Fault offset: 0x0000000000009e5d
Faulting process id: 0x584
Faulting application start time: 0x01ce34b1db568cd3
Faulting application path: C:\Program Files\Workflow Manager\1.0\Workflow\Artifacts\Microsoft.Workflow.ServiceHost.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 1c0e9293-a0a5-11e2-9b79-005056b4129c

Application: Microsoft.Workflow.ServiceHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.Workflow.Common.FatalException
Stack:
   at Microsoft.Workflow.Common.Fx+<>c__DisplayClass2.b__0()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

Don't know what caused it. After quite a lot of struggling, I simply gave it up.

Last week, SharePoint 2013 CU 201304 was released. After installing it, BANG! The problem disappeared!