CHECKDB with Repair Allow Data Loss: Should I shoot myself?

Over the past years, I have seen multiple DBAs execute DBCC CHECKDB with REPAIR ALLOW DATA LOSS to repair the SQL Server database corruption in their database without understanding repercussions. Even when you are working with CSS on such issues, CSS engineers actually explain what the command does and also send across a disclaimer stating why this command should NOT be executed on a database and what the repercussions are. Alas due to a lack of a sound database backup strategy, a lot of DBAs are forced to resort to losing data. In a nutshell, Repair Allow Data Loss tries to first repair the corruption in the page but if it cannot, then it checks if it can drop the page and repair the error and hence the name Allow Data Loss! I had a question for all the DBAs who advocate the use of this command:

How would you like your Payroll DBA to run the command? (Good enough food for thought!!)

The only way to compare how much data you lost would be to perform a CHECKDB before running the CHECKDB with REPAIR ALLOW DATA LOSS and after the CHECKDB has completed. The row counts returned by the two CHECKDB outputs would have to be compared. If CHECKDB is failing due to system catalog corruption, then you are plain out-of-luck.

What most people fail to understand that CHECKDB can verify integrity of the database based on the SQL database rules imposed on it like referential integrity, data length, record structures, page structures, header correctness etc. But these are database integrity rules that we are aware of. What about the business logic rules that you imposed while designing the database.

Eg: Let’s say you had a database table called Employee which depended on a Department ID valued to be present in the Department table. No foreign key relationship exists between the two tables. You experienced corruption in the database and used Repair Allow Data Loss command along with CHECKDB which dropped a few pages from the Department table. However, CHECKDB cannot know that a Department ID is missing in the Department table because you didn’t create a Foreign Key between the two tables. So when your application now tries to retrieve the Department ID of the Employee, it will BARF on you! This is the kind of scenario we specifically want to avoid which is why we advise against the use of data loss commands.

SQL Server wouldn’t know what data is supposed to present in the table rows. The only thing that SQL can verify while you are retrieving data or checking integrity of the database is that the column data and column datatype match. I have heard questions like “Why can’t SQL Server figure out if the value is correct?”. Well, would you know why a person’s name is Uncle Sam and not Uncle Bill? 🙂 So, how would SQL Server know that the FirstName column (an example) was supposed to store Sam and not Bam. As per SQL Server, both strings match the column datatype, hence from a structural and logical integrity, the data is correct.

Why is VSS complaining about SQL Data files

I recently came across a question on why a NTBackup of a Text File on a drive reported the following error in the application event logs:

Event Type:    Error
Event Source:    VSS
Event Category:    None
Event ID:    6005
Description:
Sqllib error: Database %1 is stored on multiple volumes, only some of which are being shadowed.

The interesting part is that the drive that stores the Text file that I am backing up also has a LDF file for a database. The MDF file of the same database resides on another drive. The intelligent SQL Writer was able to find out that only one file for the database in question is on this drive and the other file(s) is not present on the same drive. Hence, it reported the error causing confusion in our minds. A red cross in any log file is a BAD thing!! Not in this case. 🙂

I have already blogged about the behavior of NTBackup/VSS when a Snapshot is created in a previous post.

The conclusion is that if you have database files residing on multiple drives and you are using VSS to backup non-SQL database files on only one of these drives, then you can ignore the above error (if that is the only error). This is yet again another of the infamous by-design behaviors! 🙂

Initial Data Collection Script

Where would you be able to use it?

The cases where you can use this script effectively are:

1. Performance issues where a quick initial look at the general state of the server is very valuable.

2. Situations where issue is actually happening right now. However, that said, it is always good to collect post mortem data as well for further diagnostics. Performance Dashboard is a good option here but if you want a snapshot of the server and store it somewhere, then this script can help you.

3. Situations where the problem is happening and you need to quickly verify that you are not hitting some known issues.

How is this different from the other scripts that are already existing?

There are scripts already out there which do the same thing. The advantage here is the fact that the output is in HTML format with bookmarks which makes it easy to determine what information is available. Also, it is easy to get to the sections that are of interest in the current scenario.

Usage

  1. Download the script using the link given at the bottom of the page and save it to a file named INITIAL_DIAG.SQL. Open the script file in a SSMS Query Window.
  2. Press CTRL+SHIFT+F so that the output results are put into a file. Doing this will not produce a message or any other notification.
  3. Execute the script and specify INITIAL_DIAG.html as the output file name so that we can get the output in the require HTML format.

Sample Output

********************************************************************************************
INITIAL MICROSOFT PSS DATA COLLECTION
********************************************************************************************

INDEX
1. General Server Configuration
a. Server Info
b. Non-default sp_configure settings
c. Server Settings
d. Active Trace Flags
e. Profiler Trace Information
2. Memory Diagnostics
a.DBCC MEMORYSTATUS output
b.Memory Clerks rollup output
3.Procedure Cache Statistics
4.Database Diagnostics
a.Replicated Databases
b.Database File Usage Stats
c.Index Statistics
d.sp_helpdb output
e.DBCC SQLPERF (LOGSPACE)
f.Hypothetical Indexes
5.Missing Indexes report
6.Blocking Information
7.Batch/Query Statistics

########################################

****************** SERVER INFO *********
########################################

—————————————————————–
Microsoft SQL Server 2005 – 9.00.3257.00 (X64)
Jun 12 2008 16:47:07
Copyright (c) 1988-2005 Microsoft Corporation
Developer Edition (64-bit) on Windows NT 6.0 (Build 6001: Service Pack 1)

—————————————————————
SQL SERVER NAME: SERVER1

CLUSTERED: No

MACHINE NAME: SERVER1
NET BIOS NAME: SERVER1

SQL PID: 5076
FTS Installed: YES
Security Mode: MIXED
Current SPID: 51

Click here to download the script.

Modification: June 20, 2012: Added additional code to account for the following:

  • Added code to identify read-write workload ratios
  • Added code to find out non-Microsoft DLLs loaded in SQL address space using sys.dm_os_loaded_modules
  • Added code to get output for SQL Server 2012 instances

How to troubleshoot Windows Internal Database setup issues

Windows Internal Database is bundled along with Windows applications that need to use a backend database solely for their application purpose. Eg: WSUS, Sharepoint.

The application’s installation files will call the MSI package for installing the Windows Installer Database. In case the application setup fails due to a failure in installing the Windows Internal Database, then you would need the MSI log file for the WID MSI package. This can be found normally in the <System Drive:>\Windows folder and would have the following naming convention: *WSSEE*.log. If you are unable to locate the file, then find out all *.log files in the folder which were created recently.

Once you have located the file, you would need to search for the following string “Return Value 3” in the file.

Eg:

MSI (s) (E0:88) [16:02:14:252]: Product: Windows Internal Database — Error 1402. Could not open key: <Reg Key Name>.  System error 5.  Verify that you have sufficient access to that key, or contact your support personnel.

Error 1402. Could not open key: <Reg Key Name> System error 5.  Verify that you have sufficient access to that key, or contact your support personnel.
Action ended 16:02:14: InstallFinalize. Return value 3.

If you check what Operating System error 5 stands for, you will find out that it is a permissions issue.

C:\>net helpmsg 5

Access is denied.

Based on the error message, you will have to take corrective actions and re-run the setup.

Microsoft Virtualization Support Policy

We do work on a lot of environments which have VMWare Virtualization. The first thing that needs to be done is to find the version of the ESX in use in the environment.

VMware ESX Update

Build Number

ESX 3.5.0 Update 1

64607

ESX 3.5.0 Update 2

110268

ESX 3.5.0 Update 3

123630

ESX 3i (3.5.0) Update 3

123629

If you trying to seek Microsoft CSS support for a non-Microsoft virtualized environment, then you need to be aware of the Virtualization Support Policy:

http://support.microsoft.com/kb/897615

Also if you want to find out if your virtualized environment is supported, then you can use the Support Policy Wizard.

Here is link for a step-by-step guide to find out the ESX versions and builds:

http://www.techhead.co.uk/how-to-determine-the-vmware-esx-or-esxi-build-version

Reference articles:

Determining detailed build number information for VMware ESX 4.0.x hosts

Determining detailed build number information for VMware ESX 3.0.x and 3.5.x hosts