Setup – RPC, please be nice to me

Recently, I had looked into a SQL Server 2008 cluster setup failure where the following information was noted in the Detail.txt file:

2011-02-21 11:58:37 Slp: Configuration action failed for feature SQL_Engine_Core_Inst during timing ConfigRC and scenario ConfigRC.
2011-02-21 11:58:37 Slp: The RPC server is unavailable
2011-02-21 11:58:37 Slp: Configuration action failed for feature SQL_Engine_Core_Inst during timing ConfigRC and scenario ConfigRC.
2011-02-21 11:58:37 Slp: System.ComponentModel.Win32Exception: The RPC server is unavailable
2011-02-21 11:58:37 Slp:    at Microsoft.SqlServer.Configuration.Cluster.ClusterResource.UpgradeResourceDLL(String nodeName, String dllPathName)
2011-02-21 11:58:37 Slp:    at Microsoft.SqlServer.Configuration.SqlEngine.SQLEngineClusterFeature.UpgradeResourceDLL(SQLServiceResource sqlResource)
2011-02-21 11:58:37 Slp:    at Microsoft.SqlServer.Configuration.SqlEngine.SQLEngineClusterFeature.ConfigureSQLEngineResourceType()
2011-02-21 11:58:37 Slp:    at Microsoft.SqlServer.Configuration.SqlEngine.SqlEngineSetupPrivate.Patch_ConfigRC(EffectiveProperties properties)
2011-02-21 11:58:37 Slp:    at Microsoft.SqlServer.Configuration.SqlEngine.SqlEngineSetupPrivate.Patch(ConfigActionTiming timing, Dictionary`2 actionData, PublicConfigurationBase spcb)
2011-02-21 11:58:37 Slp:    at Microsoft.SqlServer.Configuration.SqlConfigBase.SqlFeatureConfigBase.Execute(ConfigActionScenario scenario, ConfigActionTiming timing, Dictionary`2 actionData, PublicConfigurationBase spcbCurrent)
2011-02-21 11:58:37 Slp:    at Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.ExecuteAction(String actionId)
2011-02-21 11:58:37 Slp:    at Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.Execute(String actionId, TextWriter errorStream)
2011-02-21 11:58:37 Slp: Exception: System.ComponentModel.Win32Exception.
2011-02-21 11:58:37 Slp: Source: Microsoft.SqlServer.Configuration.Cluster.
2011-02-21 11:58:37 Slp: Message: The RPC server is unavailable.

The RPC Server is unavailable is a pretty generic error and can happen due to a lot of reasons. One of the common reasons that we have seen in CSS while troubleshooting setup issues is with incorrect DNS entries. This can happen if DNS entry for the cluster network name wasn’t configured properly and is pointing to an incorrect IP Address.

Another point to keep in mind is Windows Firewall. Make sure that the Firewall configuration allows this RPC call through.

I worked with Shahryar (Twitter) on a similar issue last week and it was identified that a PING request to the cluster name or IP returned no response.

If all else fails, then network tracing would help in figuring out what is failing and where.

References:

Error message when you connect to a cluster virtual server by using the named pipes protocol: "The machines selected for remote communication is not available at this time."
http://support.microsoft.com/kb/306985/

How to troubleshoot Windows Internal Database setup issues

Windows Internal Database is bundled along with Windows applications that need to use a backend database solely for their application purpose. Eg: WSUS, Sharepoint.

The application’s installation files will call the MSI package for installing the Windows Installer Database. In case the application setup fails due to a failure in installing the Windows Internal Database, then you would need the MSI log file for the WID MSI package. This can be found normally in the <System Drive:>\Windows folder and would have the following naming convention: *WSSEE*.log. If you are unable to locate the file, then find out all *.log files in the folder which were created recently.

Once you have located the file, you would need to search for the following string “Return Value 3” in the file.

Eg:

MSI (s) (E0:88) [16:02:14:252]: Product: Windows Internal Database — Error 1402. Could not open key: <Reg Key Name>.  System error 5.  Verify that you have sufficient access to that key, or contact your support personnel.

Error 1402. Could not open key: <Reg Key Name> System error 5.  Verify that you have sufficient access to that key, or contact your support personnel.
Action ended 16:02:14: InstallFinalize. Return value 3.

If you check what Operating System error 5 stands for, you will find out that it is a permissions issue.

C:\>net helpmsg 5

Access is denied.

Based on the error message, you will have to take corrective actions and re-run the setup.

SQL Server 2005 setup failing with 29538 error

There are numerous occasions when we find that the SQL Server 2005 setup has failed due to the following error:
 
Product : Database Services (MSSQLSERVER)
Product Version (Previous): 3042
Product Version (Final) :
Status : Failure
Log File : C:\Program Files\Microsoft SQL Server\90\Setup Bootstrap\LOG\Hotfix\SQL9_Hotfix_KB955706_sqlrun_sql.msp.log
Error Number : 29538
Error Description : MSP Error: 29538 SQL Server Setup did not have the administrator permissions required to rename a file: C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\mssqlsystemresource1.ldf. To continue, verify that the file exists, and either grant administrator permissions to the account currently running Setup or log in with an administrator account. Then run SQL Server Setup again.
 
This can happen when we are trying to install SQL Server Service Packs, Hotfixes, Cumulative Updates or GDRs. Now what you need to do is find if any of the following are true:
1. The SQL Server service account doesn’t have sufficient privileges on the DATA folder of SQL Server
2. It is a recommendation that the MSSQLSYSTEMRESOURCE files (MDF and LDF) be in the same location as the master database. This issue is documented under KB947989.
3. Another point that you would want to check is if you have any services to re-start the SQL service if it is stopped. SQL Server Setup performs setup in two phases: the first phase being replacement of binaries and the second phase being running the configuration scripts for updating the metadata. During the first phase, the SQL Server service is stopped. In case, you have a service which restarts the SQL service at that point, then you would run into this issue.
 
The easiest way to verify this would be to check the status of the SQL service from the Services snap-in right after the failure and it should show up as started. Or if you are command prompt savvy Smile, then you could use "sc query mssqlserver" (provided it’s a default instance) to check the state. If it’s a named instance, then the service name would change from mssqlserver to mssql$<instance name>
 
Sample output:
SERVICE_NAME: mssqlserver
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 4  RUNNING
                                (STOPPABLE, PAUSABLE, ACCEPTS_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0
 
Two of the known service restart rules applied by other non-Microsoft services are:
1. Cisco security agent has a Service Restart rule which causes the SQL server service to restart once it is stopped.
2. Altiris has a similar rule.
 
The job of these services is to start services that were stopped. So when SQL setup stopped SQL Server service these services started SQL service which ended up holding a handle on this mssqlsystemresource1.ldf.

Troubleshooting SQL Setup Failures

  1. The Setup Logs for SQL Server 2005 are created in the following location
    “(system drive):\Program Files\Microsoft SQL Server\90\Setup Bootstrap\LOG” 
    There are two folders underneath this location:
    a. Files
    b. Hotfix
  2. The files under the Files folder are created during the RTM installation of SQL Server 2005
  3. The files under the Hotfix folder are created during Service Pack/Hotfix installation
  4. For every failed setup of SQL Server 2005, there will be a .CAB (Cabinet file) created under the Files or Hotifx depending on what installation you are performing: RTM/Hotfix/Service Pack installation
  5. Also, the LOG folder would have a summary.txt file which would give a brief overview of which component(s) failed to install
  6. SQL Server 2005 uses MSI to install the components. For RTM setup, we use .MSI files and for patches, we use MSP files. The error # that you see in the setup summary.txt file would be present in the associated MSP or MSI log along with extended information of which function was being called. Eg. If setup fails during the configuration phase, then you might get the connectivity errors related to the attempts that setup made to connect to SQL Server. This would further provide some more hints to troubleshoot the issue
  7. Using the Error Number and the Component Name, find out if you come back with any helpful links from http://support.microsoft.com/
  8. If you are getting nowhere, then please post a question on the MSDN/TechNet Forums (Setup and Upgrade) or open up a support incident with Microsoft Product Support Services

Please refer the following for more details:
http://technet.microsoft.com/en-us/library/ms143702.aspx
http://msdn2.microsoft.com/en-us/library/ms144287.aspx

Another comprehensive blog post which talks about SQL Server 2005 Setup Logs:

http://blogs.msdn.com/raja_sivagaminathan/archive/2010/01/07/sql-server-2005-setup-log-files.aspx

SQL Server 2000 Setup Failures
The setup logs for RTM installations for SQL Server 2000 are installed with the following log sqlstp.log getting created in the %windows% folder.
The Service Pack setup logs are created having the following filename sqlsp.log under the same location specified above.
However, the hotfix logs are created with the following prefix “KB***.log

SQL Server 2000 setup doesn’t create multiple setup logs. You can use sqlstp.log (RTM setup) or sqlsp.log (Service Pack setups) or KB*****.log (Patch setup) and look for “return value 3“. If an action specified by a setup function errors out, then it prints the Return Value as 3 in the setup logs.
The drawback with the SQL setup logs for SQL Server 2000 is that the setup logs get overwritten every time setup is run. But in SQL Server 2005, this is not the case. All the logs are retained in the BOOTSTRAP folder and a new set of log files are created with incremental numbers in the same location.