Tuesday, November 11, 2014

How to dispose SPWeb and SPSite objects in PowerShell

Whenever we get the message below from ULS log, we need to monitor it carefully. If the highlighted number("16") keeps growing and rarely goes down, then it's pretty clear that there is some SPWeb or SPSite objects not been disposed properly.

Potentially excessive number of SPRequest objects (16) currently unreleased on thread 10. Ensure that this object or its parent (such as an SPWeb or SPSite) is being properly disposed. This object is holding on to a separate native heap.This object will not be automatically disposed.

Some objects, such as SPWeb.ParentWeb and SPSite.RootWeb don't need to be disposed manually. But in loop statement such as "foreach($site in $allSites)" or "foreach($web in $allWebs)", we have to call "Dispose()".

Another case is the objects created by "Get-SPSite" or "Get-SPWeb". I have to say it's quite easy (for me) to forget releasing them. Thanks for the post from Dave Wyatt, we get the "Using" feature just like what we have in C#.

Below is how I implemented it, and happy to share it here.

function UsingObject
{
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true)]
        [AllowEmptyString()]
        [AllowEmptyCollection()]
        [AllowNull()]
        [Object]
        $InputObject,

        [Parameter(Mandatory = $true)]
        [scriptblock]
        $ScriptBlock
    )

    try
    {
        . $ScriptBlock
    }
    finally
    {
        if ($null -ne $InputObject -and $InputObject -is [System.IDisposable])
        {
            $InputObject.Dispose()
        }
    }
}

function DisposeSPSite([REF]$site)
{
    if ($site.Value -ne $null)
    {
        $site.Value.Dispose()
    }
}

function DisposeSPWeb([REF]$web)
{
    if ($web.Value -ne $null)
    {
        $web.Value.Dispose()
    }
}

UsingObject ($site = Get-SPSite -Identity $siteURL) {
$objSPWebApplication = $site.WebApplication
$siteCollections = $objSPWebApplication.Sites

foreach($itemSPSite in $siteCollections)
{
if ($itemSPSite.Url.IndexOf("Office_Viewing_Service_Cache") -ge 0)
{
DisposeSPSite ([REF]$itemSPSite)
continue
}

Blah, blah....

DisposeSPSite ([REF]$itemSPSite)
}
}

We still get the message below in ULS;

An SPRequest object was reclaimed by the garbage collector instead of being explicitly freed.  To avoid wasting system resources, dispose of this object or its parent (such as an SPSite or SPWeb) as soon as you are done using it.

It seems that after calling "Dispose()", the SPRequest resource is not released immediately. They are done by .Net garbage collection soon after.

One more question: Can we only use "Start-SPAssignment" and "Stop-SPAssignment" to release the resources? For small farm, maybe all right. But, it's definitely not recommended. With more and more resources occupied by one PowerShell session, the farm will slow down eventually, and it also generates huge ULS log files.


Any comments welcome!


References:

http://msdn.microsoft.com/en-us/library/aa973248(v=office.12).aspx#sharepointobjmodel__findingundisposedobjects

http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.administration.eventseverity(v=office.15).aspx

http://davewyatt.wordpress.com/2014/04/11/using-object-powershell-version-of-cs-using-statement/

Wednesday, October 1, 2014

How to stop duplicate entries based on two or more columns in a SharePoint list?

Well, there is no OOTB support of that.

Quick search leads to two solutions: coding (event receiver) or InfoPath rule.

The cost of "coding" is high. We need developer to build it, need someone to manage the source code, and need to upgrade the assembly when upgrading SharePoint. And, the deployment will cause farm outage!

InfoPath solution also has its own problem: it is always with expensive "license" fee.

So, is there simple approach to resolve the problem?  Below is how I did it, with a simple declarative workflow.

The workflow doesn't prevent "duplicate entries", but can generate warning message, so users can fix the invalid data after the remind. That's good enough in most of the cases.


  • The structure of the test list (CombinedKey = Title & ProjectCode)



  •  Declarative workflow triggered by "new item" and "edit item" event


  •  Test result

Any comments are welcome!

Friday, August 29, 2014

The substitute of farm solutions

We all know SharePoint 2013 provides "Apps mode". Microsoft encourages all developers to build "Apps" instead of classic solutions. However, there are many jobs Apps can't do. (Andrew Connell has some brilliant thoughts about that.)

In my opinion, there is one tool been seriously underestimated: PowerShell.

PowerShell helps administrators to automate jobs. It helps developers to manage SharePoint environment. Besides that, it can replace all routinely triggered customized workflows and timer jobs!

I didn't build any SharePoint Timer Job for production (but have two open source projects for fun: Simple Reminder and Workflow Timer). Comparing to a scheduled windows task (see the screenshot below), it's not easy. Really.


How about workflow or workflow activity/action? They also need SharePoint developer to build it. I can build them, but unless the business requirement is extremely complex, I prefer to build something even normal administrators can maintain and do simple modification.

In my experience, most of the business requirements are small, simple but urgent. Besides, non-software company don't want to manage the source code of hundreds of visual studio projects!

PowerShell script doesn't include any unnecessary xml or definition file. It simply focuses on business logic, which make it short and easy understand.

One of the complain about PowerShell is about the supporting tools, such as Visual Studio and ISE. There is no full intellisense during editing at the moment(Auguest 2014). Comparing to C#, the debugging is hard.

However, in my opinion, the critical issue is the lack of client side API. Like "Apps", we don't want to run PS scripts on SharePoint server. Is it possible to run the script on an application server with no SharePoint installed? Currently, CSOM and Restful API are supported, but, comparing to server object model, there are too much restriction. Actually we can only do basic read/write (CRUD) operations at client side.

That's far from enough. Any thing we can do through "server object model", we need to be able to do through "client object model".

This sounds like a balance between "security" and "functionality", and Microsoft needs to find a way to meet the ends.

[update, 2014-10-09]

To configure Task Scheduler in Failover Cluster, we can follow this one for Windows Server 2012. Only Windows Server 2012+ support "Clustered Tasks".

Tuesday, July 29, 2014

What is SPListItem/SPFile anyway?

When I started to touch SharePoint years ago, it's natural to treat "List" as "SQL table". They have so many similar features:

Features in commonSQL ServerSharePoint
ContainerDatabaseSite
OperationsInsert,Update,Delete,QueryInsert,Update,Delete,Query
MetadataColumnsColumns
Data volumeBillions of recordsMillions of records
Event triggerYesYes
Query languageSQLCAML

Apart from the exciting SharePoint features, such as versioning, soon I noticed two differences.
  1. All SPListItems are stored in one SQL table;
  2. Compare to SQL Server, "Insert,Update,Delete" operations of SPListItem are extremely slow;
Now, can we treat a SPListItem as a record in SQL table, with advanced features and poor performance on modification? And, further more, can we treat SPList as a SQL table which allow site owners to customize?

Definitely NOT!

This is the conclusion I got after seven years of development and administration on SharePoint platform. Believe it or not, I feel that SPListItem/SPFile is actually Message, and SPList/SPContentType is Queue.

Let me show an example here.

If there are millions of items in a SPList, and we need to update the "Title" field of all data based on some rules, how should we do it? We cannot update data in batch, instead, we have to do the update one by one. This is how we process messages, not records.

Once we understand this point, then it's easy to understand why all SPListItems are stored in one SQL table, why the modification is so slow, and how we can utilize SharePoint in better way.

What do you think? Any comments are welcome!

Friday, July 18, 2014

How to start workflow in PowerShell after sp2010 sp2

To start workflow in PowerShell script should be easy. Below is the one I used for long time.

$itemCurrent = $list.GetItemById($itemId) 
$wfRunOption = [Microsoft.SharePoint.Workflow.SPWorkflowRunOptions]::Synchronous 
$wf = $manager.StartWorkFlow($itemCurrent,$assoc,$data,$wfRunOption) 

However, after installing SP2010 SP2, it stopped working with "Failed on Start" workflow status.

I know "system account" cannot trigger workflow. So, it should be easy to fix: we can get the SPSite object under another user's token. Below is the script

$web = Get-SPWeb -Identity $webURL 
$user=$web.EnsureUser("domainName\anotherUser")
$token = $user.UserToken;
$impSite= New-Object Microsoft.SharePoint.SPSite($webURL, $token);
$web.Dispose()
$web = $impSite.OpenWeb()
$manager = $web.Site.WorkFlowManager 
$list = $web.Lists[$listName] 

However, nothing is changed. Still the same error: "Failed on Start".

I did some searching on Internet, which leads to pages like here and here. None of them resolve the problem.

Then I RDP to SharePoint server desktop as another user which has enough rights of database and site collection, and run PowerShell script manually. Still same. This is weird.

As last resort, I RDP to SharePoint server as a farm admin and then run the "SharePoint 2010 Management Shell" as a different user (which has enough rights, such as "domainName\SomeUser"). It worked!

I don't understand it although feel quite happy.

(I would really appreciate it if you can explain that in comments)

[update 20140718]

If still got error "Failed on Start", please check ULS log.  If the error message is similar to the one below, you need to grant windows local admin rights to "domainName\SomeUser", which is used to run "SharePoint 2010 Management Shell".

Errors in ULS:

Workflow Compile Failed while loading AuthorizedTypes from config: An error occurred creating the configuration section handler for System.Workflow.ComponentModel.WorkflowCompiler/authorizedTypes: Could not find file 'C:\Users\0efang\AppData\Local\Temp\qklxfvr3.dll'. (C:\inetpub\wwwroot\wss\VirtualDirectories\appsdev.unitingcare.local80\web.config line 654)

RunWorkflow: Microsoft.SharePoint.SPException: An error occurred creating the configuration section handler for System.Workflow.ComponentModel.WorkflowCompiler/authorizedTypes: Could not find file 'C:\Users\0efang\AppData\Local\Temp\qklxfvr3.dll'. (C:\inetpub\wwwroot\wss\VirtualDirectories\appsdev.unitingcare.local80\web.config line 654)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.<>c__DisplayClasse.b__c()    
 at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.DoCompileNewAppDomain(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, SPWeb web, CompilationPacket& packet, DirectoryInfo& tempDir)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.CompileBytes(Byte[] xomlBytes, Byte[] rulesBytes, Boolean doTestCompilation, String assemblyName, SPWeb web, Boolean forceNewAppDomain)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.LoadXomlAssembly(SPWorkflowAssociation association, SPWeb web)    
 at Microsoft.SharePoint.Workflow.SPWinOeHostServices.LoadDeclarativeAssembly(SPWorkflowAssociation association, Boolean fallback)    
 at Microsoft.SharePoint.Workflow.SPWinOeHostServices.CreateInstance(SPWorkflow workflow)    
 at Microsoft.SharePoint.Workflow.SPWinOeEngine.RunWorkflow(SPWorkflowHostService host, SPWorkflow workflow, Collection`1 events, TimeSpan timeOut)    
 at Microsoft.SharePoint.Workflow.SPWorkflowManager.RunWorkflowElev(SPWorkflow workflow, Collection`1 events, SPWorkflowRunOptionsInternal runOptions)

Microsoft.SharePoint.SPException: An error occurred creating the configuration section handler for System.Workflow.ComponentModel.WorkflowCompiler/authorizedTypes: Could not find file 'C:\Users\0efang\AppData\Local\Temp\qklxfvr3.dll'. (C:\inetpub\wwwroot\wss\VirtualDirectories\appsdev.unitingcare.local80\web.config line 654)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.<>c__DisplayClasse.b__c()    
 at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.DoCompileNewAppDomain(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, SPWeb web, CompilationPacket& packet, DirectoryInfo& tempDir)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.CompileBytes(Byte[] xomlBytes, Byte[] rulesBytes, Boolean doTestCompilation, String assemblyName, SPWeb web, Boolean forceNewAppDomain)    
 at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.LoadXomlAssembly(SPWorkflowAssociation association, SPWeb web)    
 at Microsoft.SharePoint.Workflow.SPWinOeHostServices.LoadDeclarativeAssembly(SPWorkflowAssociation association, Boolean fallback)    
 at Microsoft.SharePoint.Workflow.SPWinOeHostServices.CreateInstance(SPWorkflow workflow)    
 at Microsoft.SharePoint.Workflow.SPWinOeEngine.RunWorkflow(SPWorkflowHostService host, SPWorkflow workflow, Collection`1 events, TimeSpan timeOut)    
 at Microsoft.SharePoint.Workflow.SPWorkflowManager.RunWorkflowElev(SPWorkflow workflow, Collection`1 events, SPWorkflowRunOptionsInternal runOptions)

Friday, June 20, 2014

Error - Unable to retrieve topology component health states

I think many issues may cause this error message. Recently I also got this issue all of sudden.

Some people suggest to Repair SharePoint under Control Panel, restarting search servers, reinstall Search Service Application, change the service account, or recreate entirely the search service application and components.

My question is, why it worked before?

Then I noticed the post from Joel Plaut. Joel mentioned that we could change "memoryLimitMegabytes" settings in "noderunner.exe.config" to reduce the memory consumption, that rang the bell in my mind.

I got this problem in Development environment (SP1 + CU201406). Apart from a separate SQL Server instance, the farm contains 2 virtual machines, and each have 8GB RAM. Since I started "Microsoft SharePoint Foundation Web Application" on both servers, the hardware resource is far below the minimum requirement from MSDN.

So, I kept all web applications running on one machine, and moved most of the other services, such as "Search Services" and "Distributed Cache service" to the other VM.

After restarting "Search Host Controller Service", Bing! The problem is fixed!

Tuesday, June 17, 2014

"v15.master" "belltown.master" "overlay.master" are gone!

When reading "Professional SharePoint 2013 Administration", I noticed that "v15.master" "belltown.master" "overlay.master" were not there under "Master Pages" gallery folder.

I am pretty sure they were there a while ago, and, when googling it, there are so many posts mentioned these three master pages. Now, what we have in "Publishing Portal" site is below:



Adrian noticed this issue in the post , but didn't provide the reason. Considering the publish time of that post, I assume the change was made by SharePoint 2013 Public Update March 2013. However, Microsoft didn't mention this change in Description of the SharePoint Foundation 2013 update: March 12, 2013 and Description of the SharePoint Server 2013 update: March 12, 2013

In my opinion, use "seattle.master" and "oslo.master" to replace "v15.master" "belltown.master" "overlay.master" is good. It's more reasonable.

Anyway, this is the reality and we have to accept it. :-)

Below are two screenshots to compare the major differences.

seattle.master:


oslo.master: