Wednesday, September 21, 2016

Office 365 portal is awkward, confusing and daunting

It's really designed by and for developers. :-)

How many users are there happy to spend time to get familiar with these 18 icons, to figure out which one they should click, and then start to work there?

What users want to do here?

1. Create / modify a document (OneDrive, SharePoint, Video, Word, Excel, PowerPoint, OneNote, Sway)
2. Talk / contact some one. (Mail, People, Groups. Where's Lync?)
3. Figure out what need to do. (Mail, Calendar, Planner, Tasks, Delve)
4. Web surfing. (Yammer, Newsfeed, Delve)

If users could only see 4 entry icons here, I believe that they would feel much more comfortable.

But, it's still a bit messy.

SharePoint is the place to store information. Are emails information? How about People contact details? Calendar events? Tasks? Why so much information is stored in other separate places?

If we could store all information in SharePoint, then many of those services could be moved into SharePoint User Profile module, such as Mail, People, Groups, Calendar, Planner, Tasks, Yammer, Newsfeed. (OneDrive and Delve are already there)

We even don't need Microsoft Graph anymore! That's a big relief to developers, as they only need to learn how to access SharePoint data.

Do you like this idea?  Any comments are welcome!

[update, 2016-09-29]

If a document needs to be shared in SPO, Yammer, Planner and Groups, does that mean that the same document has 4 separate copies?

To manage the versions and permissions of this document must be hard.  :-)

Friday, June 17, 2016

SharePoint 2016 CU installation failure: "Exception: The upgraded database schema doesn't match the TargetSchema"

After installing CU 201606 for SharePoint 2016, "SharePoint 2016 Products Configuration Wizard" threw out error:

Database is in compatibility range and upgrade is recommended

In ULS log doesn't help much, and Windows Event Viewer tell us:

Failed to upgrade SharePoint Products.
An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown.  Additional exception information: The upgrade command is invalid or a failure has been encountered.
Failed to upgrade SharePoint Products.
Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException: Exception of type 'Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException' was thrown.
   at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

So I went to Central Admin, and then clicked "Upgrade and Migration" page:

It brings us here:


We can see that the content database "SP_Content_team80_tmp" caused the problem.

It's easy to solve the problem. Just run the PowerShell script below:

Upgrade-SPContentDatabase SP_Content_team80_tmp -NoB2BSiteUpgrade -Confirm:$false

Then, re-run "SharePoint 2016 Products Configuration Wizard".


Hope this trick save you some time   :-)

[update, 2016-08-17]

One line of PS script to upgrade all content databases.

Get-SPContentDatabase | ?{$_.NeedsUpgrade –eq $true} | Upgrade-SPContentDatabase -Confirm:$false

Friday, May 27, 2016

SharePoint Online - replace Server-side programming with Client-side programming

Finally, with the help of WebHooks, we can use Client-side programming to replace Server-side programming in SharePoint. ( )

But, does it really resolve the problem of customization on SharePoint Online sites?

In many cases, YES, but we need to be very careful. Because "data communication" is moved from RAM-RAM to Computer-Computer.

1. Hardware Latency

Latency of communication between different processes on the same machine, is totally different from the one between different machines. Let's check it here ( ). "Main memory reference" consumes around 100 ns, and "Round trip within same data center" takes around 500 us, which means the latter is 5000 times slower.

For external servers (not in the same data center), "Round trip" may take more than 30 ms. That's 300,000 times slower.

Caching doesn't help much in many cases.

2. Stability

Let's assume that all servers are in the same data center. Is the network in a data center as robust as the RAM on one computer?

3. Extra hardware overhead

How much work it needs to handle a web service request? Please check here for "IIS Architectures"

How much extra CPU, Memory Access, DISK IO, Network IO will be consumed for each request? Do we need to pay for that?

A data center may handle one million concurrent users easily, but, when one user open a customized page, it may cause many HTTP(s) requests by the JavaScript on that page. And, each triggered workflow instance may also submit many HTTP(s) requests!

4. Development and trouble shooting

To move a workflow activity from "Server Object model" to "Client object model", for me, it's painful.

SharePoint 2013 CSOM APIs are powerful, but, because there is one more layer, it's more complex. However, this post suggests to utilize mature third-party APIs instead of "reinventing wheels". I totally don't agree about that, because of "Reliability".

5. Reliability

If everything is on-premise, for a normal middle size enterprise, they may utilize 10,000 APIs(through assemblies) from 20 different software vendors. That's fine. Everything is fully tested before deploying to production servers. Any update/patch will also be fully tested on non-production servers.

But, if there are 10,000 APIs(through Web Services) from 20 different vendors, how can we keep the whole systems stable? If, on average, each API is upgraded/changed every 10 years, then there will be 3 APIs(Web Services) being changed every day. And not likely the changes can get fully tested on non-production environment first.

In general, the quality of Microsoft products is pretty good, but, how many times Microsoft recalled updates of their products? Can we expect the software/service/APIs from those 20 vendors are all as good as the one from Microsoft? How much we need to pay for these APIs every year?

In summary, we can move everything into cloud, just need to be cautious.

Wednesday, May 25, 2016

How to avoid top navigation menu item wraps

To support mobile view, SharePoint 2013 wraps menu item automatically. That's not user friendly to PC users.  As the screenshot below show, 7 items are displayed in 13 rows!

 Most of solutions recommend to overwrite CSS for element "ul.dynamic", such as here and here. But that requires uploading a css file and then add the CSS reference in master page. Is there any other place uses "ul.dynamic"? If yes, then all of them get affected.

Below is a tip so we can avoid modifying master page.

Just a dummy menu item.

Friday, May 20, 2016

"Invalid feature definition" error during SharePoint 2016 configuration

The error message is confusing, but easy to fix the problem: give the SQL instance more RAM.

It needs at least 3.5GB RAM.

Below is what I got when only gave SQL instance 2GB RAM.

05/20/2016 11:45:26  9  ERR                                    An exception of type System.Xml.Schema.XmlSchemaException was thrown.  Additional exception information: Feature definition with Id ca7bd552-10b1-4563-85b9-5ed1d39c962a failed validation, file 'fieldswss4.xml', line 68, character 9:
The 'ListInternal' attribute is not allowed.
System.Xml.Schema.XmlSchemaException: Feature definition with Id ca7bd552-10b1-4563-85b9-5ed1d39c962a failed validation, file 'fieldswss4.xml', line 68, character 9:
The 'ListInternal' attribute is not allowed. ---> System.Xml.Schema.XmlSchemaValidationException: The 'ListInternal' attribute is not allowed.
   --- End of inner exception stack trace ---
   at Microsoft.SharePoint.Administration.SPElementManifest.ElementXmlValidationCallBack(Object sender, ValidationEventArgs evtargs)
   at System.Xml.Schema.XmlSchemaValidator.SendValidationEvent(String code, String arg)
   at System.Xml.Schema.XmlSchemaValidator.ValidateAttribute(String lName, String ns, XmlValueGetter attributeValueGetter, String attributeStringValue, XmlSchemaInfo schemaInfo)
   at System.Xml.Schema.XmlSchemaValidator.ValidateAttribute(String localName, String namespaceUri, XmlValueGetter attributeValue, XmlSchemaInfo schemaInfo)
   at System.Xml.XsdValidatingReader.ValidateAttributes()
   at System.Xml.XsdValidatingReader.ProcessElementEvent()
   at System.Xml.XsdValidatingReader.Read()
   at System.Xml.XmlReader.MoveToContent()
   at System.Xml.XmlReader.IsStartElement()
   at Microsoft.SharePoint.Utilities.SPUtility.XsdValidateXml(XmlTextReader xmlStreamReader, String friendlyName, String pathXsdFile, String tagExpectedRootNode, Int32 desiredPathVersion, ValidationEventHandler xsdValEventHandler)
   at Microsoft.SharePoint.Administration.SPElementManifest.ValidateDefinition(String pathToFeatureAndElementManifestXsdFile)
   at Microsoft.SharePoint.Administration.SPFeatureDefinition.ValidateElementManifestXml(String pathToFeatureAndElementManifestXsdFile)
   at Microsoft.SharePoint.Administration.SPFeatureDefinition.ValidateDefinition(String pathToFeatureAndElementManifestXsdFile)
   at Microsoft.SharePoint.Administration.SPFeatureDefinitionCollection.AddCore(SPFeatureDefinition featdef, SPSite site, String solutionHash, Boolean fForce, Boolean fDoValidation, String pathToFeatureAndElementManifestXsdFile)
   at Microsoft.SharePoint.Administration.SPFarmFactory.EnsureOutOfBoxFeaturesInstalled(SPFarm farm, String[] rgsOutOfBoxFeatures, Int32 compatibilityLevel)
   at Microsoft.SharePoint.Administration.SPFarmFactory.CreateBasicServices(SPFarm farm)
   at Microsoft.SharePoint.Administration.SPFarmFactory.Create()
   at Microsoft.SharePoint.Administration.SPFarm.Create(SqlConnectionStringBuilder configurationDatabase, SqlConnectionStringBuilder administrationContentDatabase, SqlConnectionStringBuilder siteMapDatabase, IdentityType identityType, String farmUser, SecureString farmPassword, SecureString masterPassphrase)
   at Microsoft.SharePoint.PostSetupConfiguration.ConfigurationDatabaseTask.CreateOrConnectConfigDb()
   at Microsoft.SharePoint.PostSetupConfiguration.ConfigurationDatabaseTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

Tuesday, May 17, 2016

"DirectoryNotFoundException" from "Restore-SPSite"

There are a few possible causes of this exception, but the error message is confusing.  In my case, it's either about "cannot find directory" or "invalid data".  Below are the ones.

  • Error 1

This one is easy. As the error message shows, the SQL Server Instance needs more RAM.
During my test, it needs at least 3.5GB memory.

  • Error 2

The source and destination farm are with slightly different version.
We can get the version number through PowerShell script:


In the end, I use free tool "HxD Hex Editor" modified the version number in the site collection backup file, to make it identical to the target farm. Of course this should only be applied to non-production environment, and may cause unexpected issues.

Error 1 =========================================

Restore-SPSite : 0x80070003
At E:\EricFang\Scripts\Restore\Restore.intranet2013SIT.80.Main.ps1:30 char:1
+ Restore-SPSite -Identity $SiteUrl -Path $FilePath -DatabaseServer
$DatabaseServe ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (Microsoft.Share...dletRestoreSite:
   SPCmdletRestoreSite) [Restore-SPSite], DirectoryNotFoundException
    + FullyQualifiedErrorId : Microsoft.SharePoint.PowerShell.SPCmdletRestoreS

.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: There is insufficient system memory in resource pool 'default' to run this query.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.HasMoreRows()
   at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
   at System.Data.SqlClient.SqlDataReader.NextResult()
   at Microsoft.ResourceManagement.Data.DataAccess.RetrieveWorkflowDataForHostActivator(Int16 hostId, Int16 pingIntervalSecs, Int32 activeHostedWorkflowDefinitionsSequenceNumber, Int16 workflowControlMessagesMaxPerMinute, Int16 requestRecoveryMaxPerMinute, Int16 requestCleanupMaxPerMinute, Boolean runRequestRecoveryScan, Boolean& doPolicyApplicationDispatch, ReadOnlyCollection`1& activeHostedWorkflowDefinitions, ReadOnlyCollection`1& workflowControlMessages, List`1& requestsToRedispatch)
The Execute method of job definition Microsoft.SharePoint.Administration.SPConfigurationRefreshJobDefinition (ID 20715ec8-eff1-459b-a5a6-f1facd58609e) threw an exception. More information is included below.

There is insufficient system memory in resource pool 'internal' to run this query.
A severe error occurred on the current command.  The results, if any, should be discarded.

Error 2 =========================================

Restore-SPSite : 0x80070003
At E:\EricFang\Scripts\Restore\Restore.intranet2013DEV.80.Main.ps1:30 char:1
+ Restore-SPSite -Identity $SiteUrl -Path $FilePath -DatabaseServer
$DatabaseServe ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (Microsoft.Share...dletRestoreSite:
   SPCmdletRestoreSite) [Restore-SPSite], DirectoryNotFoundException
    + FullyQualifiedErrorId : Microsoft.SharePoint.PowerShell.SPCmdletRestoreS


Thursday, May 12, 2016

Workflow Manager 1.0 and SharePoint 2013 server object model

(This post is for SharePoint On Premise.)

With the release of Workflow Manager 1.0, developers are encouraged to choose "Client side object model" (CSOM or JSON).  The reason is simple: make SharePoint farm more stable, and easy to migrate to SharePoint Online.

But, what's the drawbacks?

1. Not easy to convert existing SharePoint 2010 workflow assemblies to the new version.

2. Client side object model is not as powerful as Server object model.

3. Learning curve for developers.

4. To improve performance & decrease network latency, may need to consider extra caching module.

All right, if, for some reason, we want to utilize the powerful Workflow Manager 1.0 and SharePoint server object model, what should we do?

One obvious options is to build web services which are hosted on SharePoint server. Then Workflow Manager can call them to get the functionalities.

That's awkard. We have to build two parts for a workflow.

Can we use Server object model in workflow CodeActivity directly, under CodeActivityContext?

The answer is YES. And, it's not too hard.

1. Install and configure Workflow Manager 1.0 on a SharePoint Server.

2. Build the activity.

Need to think about the parameters. I prefer to add 4 parameters to each activity. As the name suggested, the last one will pass in the SPListItem URL and ItemID, so we can get SPWeb and SPListItem easily.

I didn't find a way to put log into workflow history from activity, so all information need to be recorded in parameter "ReturnDetailedInfo".

"ReturnCode" tells the workflow instance whether the activity succeed or fail.
"ReturnValue" return the result.

<Parameter Name="ReturnCode"  Type="System.String, mscorlib" Direction="Out" DesignerType="ParameterNames" Description="Workflow variable output" />
<Parameter Name="ReturnValue"  Type="System.String, mscorlib" Direction="Out" DesignerType="ParameterNames" Description="Workflow variable output" />
<Parameter Name="ReturnDetailedInfo"  Type="System.String, mscorlib" Direction="Out" DesignerType="ParameterNames" Description="Workflow variable output" />
<Parameter Name="CurrentItemUrl" Type="System.String, mscorlib" Direction="In" DesignerType="StringBuilder" />

3. Deploy the workflow (activity) assemblies and solution files to the SharePoint farm and the  workflow server.

Don't forget to create/modify "AllowedTypes.xml" in "C:\Program Files\Workflow Manager\1.0\Workflow\WFWebRoot\bin" and "C:\Program Files\Workflow Manager\1.0\Workflow\Artifacts"

Need to reboot windows service "Workflow Manager Backend" during the deployment.

Reference link: 

4. Make sure workflow service account has "SPDataAccess" rights of SharePoint Configuration database and the relevant content databases

$PrincipleName = "domainname\_WFServices"
Get-SPWebApplication | %{$_.GrantAccessToProcessIdentity($PrincipleName)}

5. Make sure workflow service account is in "WSS_WPG" local windows user group
Reboot the server after adding the service account to "WSS_WPG".

6. Confirm workflow service account has rights on c2WTS

RDP to the workflow server as the service account. The test can be done through c2WTS Tester

7. Restore the site collection to the target SharePoint farm


8. Make sure workflow service account has rights to access target site collection

We can build a X64 windows console program to test it.
RDP to the workflow server as the service account to test it.

9. Clean up Distributed Cache

Reboot all caching servers, or run the script below on all caching servers.
The script below come from here.

Get-SPServiceInstance | Where-Object { $_.TypeName -eq "Distributed Cache" } | Stop-SPServiceInstance -Confirm:$false | Out-Null
While (Get-SPServiceInstance | Where-Object { $_.TypeName -eq "Distributed Cache" -and $_.Status -ne "Disabled" }) {
    Start-Sleep -Seconds 15

Get-SPServiceInstance | Where-Object { $_.TypeName -eq "Distributed Cache" } | Start-SPServiceInstance | Out-Null
While (Get-SPServiceInstance | Where-Object { $_.TypeName -eq "Distributed Cache" -and $_.Status -ne "Online" }) {
    Start-Sleep -Seconds 15

10. Build and publish the workflows through SharePoint Designer;

11. IISReset on Web Front End servers

Below is the sample code.

        protected override void Execute(CodeActivityContext context)
            string strTmp = string.Empty;

            SPList objSPList = null;
            SPListItem objSPListItem = null;

                this._ArgCurrentItemUrl = this.CurrentItemUrl.Get(context);
                Uri UriCurrentItemUrl = new Uri(_ArgCurrentItemUrl);
                NameValueCollection queryStrings = HttpUtility.ParseQueryString(UriCurrentItemUrl.Query);
                if (queryStrings["ID"] == null)
                    strTmp = string.Format("list '{0}' doesn't exist!", _CurrentListName);
                this._ItemID = int.Parse(queryStrings.Get("ID"));

                for (int i = 0; i < UriCurrentItemUrl.Segments.Length; i++)
                    if (UriCurrentItemUrl.Segments[i].Equals(@"Lists/", StringComparison.InvariantCultureIgnoreCase))
                        this._CurrentListName = UriCurrentItemUrl.Segments[i + 1];
                        this._CurrentListName = this._CurrentListName.Substring(0, this._CurrentListName.Length - 1);
                using (SPSite objSPSite = new SPSite(this._ArgCurrentItemUrl))
                    using (SPWeb objSPWeb = objSPSite.OpenWeb())
                        this._CurrentWebUrl = objSPWeb.Url;
                        objSPList = objSPWeb.Lists.TryGetList(this._CurrentListName);
                        if (objSPList == null)
                            objSPList = objSPWeb.Lists[this._CurrentListName];
                            if (objSPList == null)
                                strTmp = string.Format("list '{0}' doesn't exist in web '{1}'!", this._CurrentListName, this._CurrentWebUrl);
                                throw new Exception(strTmp);
                            this._CurrentListName = objSPList.Title;
                        objSPListItem = objSPList.GetItemById(_ItemID);
                        if (objSPListItem == null)
                            strTmp = string.Format("list item '{0}' doesn't exist in list '{1}', in web '{2}'!",
                                this._ItemID, this._CurrentListName, this._CurrentWebUrl);

                this._ArgSQLConnectionString = this.SQLConnectionString.Get(context);
                this._ArgColumnMappingXml = this.ColumnMappingXml.Get(context);
                this._ArgSQLSelectCommand = this.SQLSelectCommand.Get(context);

            catch (Exception ex)
                strTmp = string.Format(@"Execute(), ex.Message={0}", ex.Message);
                strTmp = string.Format(@"Execute(), ex.StackTrace={0}", ex.StackTrace);
                this.ReturnCode.Set(context, this._ArgReturnCode);
                this.ReturnValue.Set(context, this._ArgReturnValue);
                this.ReturnDetailedInfo.Set(context, this._ArgReturnDetailedInfo);