SharePoint 2013 April 2013 CU installation trouble shooting

SharePoint 2013 April 2013 CU is released!  I couldn't wait to try it.

In one farm, it went very well. But in another one, the upgrade failed.

Below is what I found in windows events log.

Failed to upgrade SharePoint Products.
An exception of type Microsoft.SharePoint.Upgrade.SPUpgradeException was thrown.  Additional exception information: Upgrade completed with errors.  Review the upgrade log file located in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\Upgrade-20130430-111738-752.log.  The number of errors and warnings is listed at the end of the upgrade log file.
Microsoft.SharePoint.Upgrade.SPUpgradeException: Upgrade completed with errors.  Review the upgrade log file located in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\Upgrade-20130430-111738-752.log.  The number of errors and warnings is listed at the end of the upgrade log file.
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.CheckPoint()
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.LogEnd()
   at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

Configuration of SharePoint Products failed.  Configuration must be performed in order for this product to operate properly.  To diagnose the problem, review the extended error information located at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\PSCDiagnostics_4_30_2013_11_17_34_737_955156400.log, fix the problem, and run this configuration wizard again.

In the upgrade log file, I found the error messages:

04/30/2013 10:55:26.80    OWSTIMER (0x10F8)    0x16FC    SharePoint Foundation Upgrade    SPUpgradeSession    aj0ur    ERROR    Upgrade Timer job is exiting due to exception: Microsoft.SharePoint.Upgrade.SPUpgradeException: Upgrade completed with errors.  Review the upgrade log file located in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\Upgrade-20130430-104811-344.log.  The number of errors and warnings is listed at the end of the upgrade log file.     at Microsoft.SharePoint.Upgrade.SPUpgradeSession.CheckPoint()     at Microsoft.SharePoint.Upgrade.SPUpgradeSession.LogEnd()     at Microsoft.SharePoint.Administration.SPUpgradeJobDefinition.Execute(Guid targetInstanceId)    91a9169c-2c55-d039-643e-29605165d88b
04/30/2013 10:55:26.88    OWSTIMER (0x10F8)    0x16FC    SharePoint Foundation Upgrade    SPManager    ajxo1    DEBUG    Removing exclusive upgrade regkey by setting the mode to none    91a9169c-2c55-d039-643e-29605165d88b
04/30/2013 10:55:26.88    OWSTIMER (0x10F8)    0x16FC    SharePoint Foundation Upgrade    SPUpgradeSession    ajxl4    INFO    ======================================================================================    91a9169c-2c55-d039-643e-29605165d88b
04/30/2013 10:55:26.88    OWSTIMER (0x10F8)    0x16FC    SharePoint Foundation Upgrade    SPUpgradeSession    ajxl5    INFO    Upgrade session failed. 4 errors and 0 warnings encountered.    91a9169c-2c55-d039-643e-29605165d88b
04/30/2013 10:55:26.88    OWSTIMER (0x10F8)    0x16FC    SharePoint Foundation Upgrade    SPUpgradeSession    ajxl6    INFO    Look for possible causes for upgrade issues by searching [ERROR] and [WARNING] strings in the upgrade log file. Refer to "http://go.microsoft.com/fwlink/?LinkId=259653" for more information about how to recover from upgrade failures.    91a9169c-2c55-d039-643e-29605165d88b
04/30/2013 10:55:26.88    OWSTIMER (0x10F8)    0x16FC    SharePoint Foundation Upgrade    SPUpgradeSession    ajxl7    INFO    ======================================================================================    91a9169c-2c55-d039-643e-29605165d88b

In PSCDiagnostics log file:

04/30/2013 11:15:31  1  ERR          Failed to upgrade SharePoint Products.
An exception of type Microsoft.SharePoint.Upgrade.SPUpgradeException was thrown.  Additional exception information: Upgrade completed with errors.  Review the upgrade log file located in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\Upgrade-20130430-111314-494.log.  The number of errors and warnings is listed at the end of the upgrade log file.
Microsoft.SharePoint.Upgrade.SPUpgradeException: Upgrade completed with errors.  Review the upgrade log file located in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS\Upgrade-20130430-111314-494.log.  The number of errors and warnings is listed at the end of the upgrade log file.
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.CheckPoint()
   at Microsoft.SharePoint.Upgrade.SPUpgradeSession.LogEnd()
   at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

Not sure what happened. The error message is not really helpful.  So I tried to upgrade it one more time manually through command "psconfig.exe -cmd upgrade -inplace b2b -wait -force".

This time, in PSCDiagnostics log file, finally there was something useful.


04/30/2013 11:18:39.07    PSCONFIG (0x0A00)    0x0E50    SharePoint Foundation Upgrade    SPUpgradeSiteSession    ajxn2    DEBUG    UPGRADE PERCENTAGE: 100.00000000% done, Total Elapsed Time 00:00:00.8281472.    00000000-0000-0000-0000-000000000000
04/30/2013 11:18:39.07    PSCONFIG (0x0A00)    0x0E50    SharePoint Foundation Upgrade    SPUpgradeSiteSession    aloox    DEBUG    End processing children of [SPSite Url=http://hvsp2013:8080/personal/pw2__spsetup] for upgrade.    00000000-0000-0000-0000-000000000000
04/30/2013 11:18:39.07    PSCONFIG (0x0A00)    0x0E50    SharePoint Foundation Upgrade    SPUpgradeSiteSession    alooy    DEBUG    Exiting upgrade for [SPSite Url=http://hvsp2013:8080/personal/pw2__spsetup].  Elapsed Time=[00:00:00]    00000000-0000-0000-0000-000000000000
04/30/2013 11:18:39.07    PSCONFIG (0x0A00)    0x0E50    SharePoint Foundation Upgrade    SPManager    ajxhh    INFO    Finished upgrading [SPSite Url=http://hvsp2013:8080/personal/pw2__spsetup].    00000000-0000-0000-0000-000000000000
04/30/2013 11:18:39.07    PSCONFIG (0x0A00)    0x0E50    SharePoint Foundation Upgrade    SPUpgradeSiteSession    ajxl4    INFO   

Not sure which feature caused the upgrade failure. As a quick fix, I deleted the personal site collection of user "pw2\_spsetup".

Then, "psconfig.exe -cmd upgrade -inplace b2b -wait -force" upgraded the whole farm successfully.