After seeing various issues with SQL Server startup problem, I felt that I know almost most of the errors, but I was wrong till someone contacted me with a new error. But I learned that Errorlog is always a good place to start with. SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location. Let us learn in this blog post we are going to learn how to fix unable to load user-specified certificate. Here is what I saw in my client’s machine.

2016-11-03 08:55:09.64 spid9s Server name is ‘SQLSAPPROD\BILLING’. This is an informational message only. No user action is required.
2016-11-03 08:55:09.64 spid9s The NETBIOS name of the local node that is running the server is ‘SQLSAPNODE1’. This is an informational message only. No user action is required.
2016-11-03 08:55:09.64 Server Error: 26014, Severity: 16, State: 1.
2016-11-03 08:55:09.64 Server Unable to load user-specified certificate [Cert Hash(sha1) “FD757A4A777966D5EEB2BD5445D151528E47A62E”]. The server will not accept a connection. You should verify that the certificate is correctly installed. See “Configuring Certificate for Use by SSL” in Books Online.
2016-11-03 08:55:09.64 Server Error: 17182, Severity: 16, State: 1.
2016-11-03 08:55:09.64 Server TDSSNIClient initialization failed with error 0x80092004, status code 0x80. Reason: Unable to initialize SSL support. Cannot find object or property.

Above snippet of ERRORLOG has interesting message “Unable to load user-specified certificate [Cert Hash(sha1) “FD757A4A777966D5EEB2BD5445D151528E47A62E”]. The server will not accept a connection. You should verify that the certificate is correctly installed. See “Configuring Certificate for Use by SSL” in Books Online”

The certificate hash value if picked via “Certificate” registry key and once value is picked, certificate store is checked for the certificate (type, subject, thumbprint etc. would be checked)

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL<Version>.<InstanceID>\MSSQLServer\SuperSocketNetLib

Below is the registry key on my client’s computer.

The value of <Version> would be dependent on SQL Server version:

When I asked my client, they said there is no certificate they are using.

WORKAROUND

If you are not using a certificate, then you can go ahead and clean up the value in the registry. If you are using the certificate, then make sure it is installed correctly and it is having right thumbprint, subject etc.

Reference: Pinal Dave (http://blog.sqlauthority.com)

First appeared on SQL SERVER – Error: 26014 – Unable to Load User-Specified Certificate

Why won't my SP work? I did this exact thing. You're never too old to do something like this.

The post Why won’t my SP work THIS time? appeared first on The MidnightDBA Star-Times.

One of my clients reported a usual behavior of the AlwaysOn availability group. He informed me that he is trying to add a login on the secondary replica in the AlwaysOn AG environment but it is failing with below error:

Msg 3906, Level 16, State 1, Line 3
Failed to update database “SCCMDB” because the database is read-only.

As a normal troubleshooting, I asked to do it from SQL Server Management Studio and got below error.

Create failed for Login ‘TestLogin’. (Microsoft.SqlServer.Smo)
Failed to update database “SCCMDB” because the database is read-only. (Microsoft SQL Server, Error: 3906)

One strange this which he informed me that it works well on primary replica.

I was curious to know why would creating a login would try to write information into a user database? As a part of troubleshooting, I asked to capture a profile trace while reproducing the error. I was able to see that there is a trigger getting fired while creating a login.

SOLUTION/WORKAROUND

We executed this query

SELECT t.name
,t.object_id
,t.is_disabled
,te.type_desc
FROM master.sys.server_triggers t
INNER JOIN master.sys.server_trigger_events te ON t.object_id = te.object_id

And found that there was a DDL trigger defined for the login creation which was logging an entry in SCCMDB.  Once we disabled the trigger, the issue was resolved.

Reference : Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Unable to create login AlwaysOn availability group secondary replica

While I was playing with SQL Cluster in my lab, I restarted the VMs and found that I was not able to bring SQL Server online. As always I was looking for error message, but there was nothing interesting. Let us see in this blog post how to fix Clustered Instance Online Error.

Here were the observations:

1. SQL ERRORLOG is getting created.
2. If I start SQL from the services it runs fine.
3. If I try to bring SQL resource online in the cluster, it stays for “Online Pending” and then it goes to “Failed” state

To get more about failure in the cluster, I generated cluster log using steps in my own article.

INFO [API] s_ApiGetQuorumResource final status 0.
INFO [RES] Network Name: Agent: Sending request Netname/RecheckConfig to NN:5447358a-a102-4fc9-95f4-c040e8716859:Netbios
ERR [RES] SQL Server : [sqsrvres] ODBC Error: [08001] [Microsoft][SQL Server Native Client 11.0]SQL Server Network Interfaces: Error Locating Server/Instance Specified [xFFFFFFFF]. (268435455)
ERR [RES] SQL Server : [sqsrvres] ODBC Error: [HYT00] [Microsoft][SQL Server Native Client 11.0]Login timeout expired (0)
ERR [RES] SQL Server : [sqsrvres] ODBC Error: [08001] [Microsoft][SQL Server Native Client 11.0]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online. (268435455)
INFO [RES] SQL Server : [sqsrvres] Could not connect to SQL Server (rc -1)
INFO [RES] SQL Server : [sqsrvres] SQLDisconnect returns following information
ERR [RES] SQL Server : [sqsrvres] ODBC Error: [08003] [Microsoft][ODBC Driver Manager] Connection not open (0)
INFO [RES] Network Name: Agent: Sending request Netname/RecheckConfig to NN:52cf277d-234b-4a81-a9a7-0f078fca2a17:Netbios

As per cluster logs, the cluster is not able to connect to SQL Service.

WORKAROUND / SOLUTION

Here are the normal causes of the above error:

1. Incorrect client alias created in the configuration manager
2. SQL Browser isn’t running when SQL is listening on a non-default port or a named instance.
3. TCP port connection issue.

I already have detailed checklist for common causes.

SQL SERVER – FIX : ERROR : (provider: Named Pipes Provider, error: 40 – Could not open a connection to SQL Server) (Microsoft SQL Server, Error: )

In my lab, I found that I had a TCP alias created and port of SQL Server was changed after reboot, causing the SQL cluster issue.

To fix that forever, I changed SQL Server to listen on a static port instead of dynamic port.

Have you ever encountered same situation where the cluster log has helped you?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Clustered Instance Online Error – SQL Server Network Interfaces: Error Locating Server/Instance Specified [xFFFFFFFF]

Speaking and presenting

One of my biggest fears is public speaking and judging by this article, I’m not alone.

However it’s something that I’ve always wanted to do and as such this year I’ve done a few lightening talk sessions at various events. For those that don’t know, a lightening talk is a short (5 to 15 min) session on any subject that you are free to pick and they’re bundled up together so you’ll get 4/5 speakers within an hour.

They’re a great way to get into public speaking, the time goes incredibly fast and there’s no time for Q&A afterwards (usually). I would advise anyone looking to get into presenting to do a couple of these sessions first, just to get a feel for it. I’d also say that, as the session is so short, you should memorise your entire presentation. Practice it over and over so that you can do it in your sleep!

I’d also say that even though I’ve just said you need to memorise your presentation, don’t have a script that you need to stick to. If you plan what you’re going to say word for word and lose your way, you’ll get lost and find it difficult to recover. Instead have a general guide in your head that allows for things to be moved around a bit.

Now demos, if you’re going to do a demo, it needs to be flawless. There’s no time for you to be fixing things. There’s also no time for typing so have everything prepared to be run immediately. Time is short but I’ve seen a few great demos done in lightening talks!

Hopefully that’s a bit of advice that will help you out if you’re thinking of doing one of these talks.

I’m going to move onto some longer talks in 2017, I’ll post up any that I do and let you know how they go! Thanks for reading.

During my last consulting engagement, I was pulled by my client to consider an issue which they were facing. They informed that they have applied service pack on one of their clustered environment and since than SQL Server is not coming online. I asked to share ERRORLOG from the SQL instance. SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location. Let us learn about how to fix error after cluster patching.

2016-11-20 21:09:49.44 spid9s Starting execution of PREINSTMSDB100.SQL
2016-11-20 21:09:49.44 spid9s —————————————-
2016-11-20 21:10:01.67 spid9s Error: 5184, Severity: 16, State: 2.
2016-11-20 21:10:01.67 spid9s Cannot use file ‘D:\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\temp_MS_AgentSigningCertificate_database.mdf’ for clustered server. Only formatted files on which the cluster resource of the server has a dependency can be used. Either the disk resource containing the file is not present in the cluster group or the cluster resource of the Sql Server does not have a dependency on it.
2016-11-20 21:10:01.67 spid9s Error: 1802, Severity: 16, State: 1.
2016-11-20 21:10:01.67 spid9s CREATE DATABASE failed. Some file names listed could not be created. Check related errors.
2016-11-20 21:10:01.67 spid9s Error: 912, Severity: 21, State: 2.
2016-11-20 21:10:01.67 spid9s Script level upgrade for database ‘master’ failed because upgrade step ‘sqlagent100_msdb_upgrade.sql’ encountered error 598, state 1, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
2016-11-20 21:10:01.67 spid9s Error: 3417, Severity: 21, State: 3.
2016-11-20 21:10:01.67 spid9s Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
2016-11-20 21:10:01.67 spid9s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

The start of the problem is Error: 5184, Severity: 16, State: 2.

If we look at error message is clear that the D drive is not having dependency with the SQL Server resource. We checked failover cluster manager and found below.

As we can see we have only cluster disk 4 which was E drive. We added by clicking on a highlighted area. Once we added the disk we found that issue was still not solved and SQL was not coming online. Checked ERRORLOG again and found a new problem.

2016-11-20 21:09:48.32 Logon Error: 18456, Severity: 14, State: 11.
2016-11-20 21:09:48.32 Logon Login failed for user ‘NT AUTHORITY\SYSTEM’. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors. [CLIENT: 100.168.11.171]

I asked them series of action and they informed that they have already attempted to rebuild the system databases – which was a news to me. So now the problem was that this login was not existing in SQL Server as System databases were rebuilt. Here were the steps to fix this issue.

1. Start SQL using command prompt

NET START MSSQLSERVER /m

1. Added ‘NT AUTHORITY\SYSTEM’ account
2. Stopped SQL Server

NET STOP MSSQLSERVER

After this we could bring SQL Server online and issue was resolved. Have you seen a similar issue where rebuild was done in the cluster and it didn’t work?

Reference: Pinal Dave (http://blog.sqlauthority.com)

First appeared on SQL SERVER – Error After Cluster Patching – Error: 5184, Severity: 16, State: 2

Spotlight Developer v6.5

The key features added in this release

• SQL Server Discovery.
  • Scanning SQL Server instances can very useful especially in large and dynamic environment. The scan engine can be filtered by constraining IP address range and/or selecting one or more network domains. Each scan result can be kept and used to compare previous scan results. Thus able to determine new instances and retired instances. I will go deeper into this feature in a follow up blog (stay tuned).
• Configurable Port Numbers.
  • As you are aware Spotlight Developer is multi-tiered. The “client” installed into SSMS, the Diagnostic Server on your network and Spotlight Essentials’ Cloud services. The communication between these use http(s) which means tcp/ip ports are required. The default installation uses ports 443, 60000, 59883. Sometimes customer have installed other vendor software using these one or both port(s) (i.e. 60000, 59883). Thus customers had to find another machine to install. In previous releases these ports where hard coded! Now we have configurable files for both the SSMS component and the Diagnostic Server. Please contact Spotlight Developer support (SpotlightDeveloper@quest.com) to apply non default ports.
• Fixed an issue when the installer cannot connect to a SoSSE 11.x DS.
  • Some Customer reported difficulty configuring to connect to a Spotlight on SQL Server Diagnostic Server during the install process. The workaround was to install a local Diagnostic Server, then launch Spotlight Developer to change to the desired SQL Server Diagnostic Server via the Settings.
• Change the support email to spotlightdeveloper@quest.com.
  • Spotlight Developer now has its own support email address (previously shared with Spotlight Cloud services). For you the customer this change has little impact.

It is second to human nature to compare when given options. When I wrote about the blog, SQL SERVER – Playing with Backups and Compression, I did compare some of the space savings one will achieve when working with backups and compression. Though the space savings were significant, one of the blog readers wanted to understand how this would be on time? Is it faster or slower to do the same. I went about taking the same script and I collected the time taken. Let us learn about Performance Analysis of Backup to Azure.

SQL Server parse and compile time:
CPU time = 0 ms, elapsed time = 0 ms.
10 percent processed…. <<<removed for blog>>
70 percent processed.
Processed 53423 pages for database ‘AdventureWorks2016’, file ‘AdventureWorks2016CTP3_Log’ on file 1.
100 percent processed.
BACKUP DATABASE successfully processed 93263 pages in 1.970 seconds (369.852 MB/sec).
SQL Server Execution Times: CPU time = 140 ms, elapsed time = 2180 ms.

SQL Server parse and compile time:
CPU time = 0 ms, elapsed time = 0 ms.
10 percent processed……<<<removed for blog>>
Processed 53425 pages for database ‘AdventureWorks2016’, file ‘AdventureWorks2016CTP3_Log’ on file 1.
100 percent processed.
BACKUP DATABASE successfully processed 93264 pages in 2.149 seconds (339.052 MB/sec).
SQL Server Execution Times: CPU time = 109 ms, elapsed time = 2373 ms.

As you can see, there is hardly any difference in time when it comes to working with backup compressions. Even for large backups, the difference is not significant that one can complain as a DBA.

On a completely different note, I wanted to play around and see what will be the effect when working with Azure based backup to the URL. I wanted to test the same database and check on performance. I used the following script:

On completion of the above script, the backup was taken to a blob as mentioned in the URL. I took the same performance metrics of time to check the difference. It is shown below:

SQL Server parse and compile time:
CPU time = 0 ms, elapsed time = 0 ms.
10 percent processed.
20 percent processed….
100 percent processed.
BACKUP DATABASE successfully processed 93266 pages in 3.809 seconds (191.293 MB/sec).
SQL Server Execution Times:
CPU time = 94 ms,  elapsed time = 4385 ms.

As you can see, in this case the time difference is significant for sure. This is fundamentally because as the backup is being taken, it must be streamed onto Azure in parallel. This increased network bandwidth usage is going to take a toll on the overall process. As you can see based on my small database, it is close to twice the time. In this case, I wanted to bring this to notice because as a DBA we need to make sure this has been considered when working with Backup to Azure.

I hope you enjoyed Performance Analysis. On a completely side note, I would love to learn if you are using the backup to Azure capability in your environments? How big are these databases? Do let me know via comments about your experience.

Reference: Pinal Dave (http://blog.sqlauthority.com)

First appeared on SQL SERVER – Performance Analysis of Backup to Azure

The contentious issue here is the use of a non-default port for SQL Server.

The post Using Non-default Ports for SQL Server appeared first on Thomas LaRock.

If you liked this post then consider subscribing to the IS [NOT] NULL newsletter: http://thomaslarock.com/is-not-null-newsletter/

Many times, we need to find information about the various features / version of SQL Server installed on a machine.  If you ask a DBA, he can provide you information by looking at services and tell what all are installed. He might run some WMI/T-SQL queries to list the services of SQL Server and their versions. Let us learn about the Discovery Report.

Well, we don’t need a DBA to do that and even Windows Admin (good news for Wintel team J) can do it.  The SQL Server setup provides an option called “Installed SQL Server features Discovery Report” under Tools in Installation Center which can provide this information in a single place.  Here is the installation center of the Start Menu.

Alternatively, we can also double click on setup.exe, which also launches installation center. Once launched, we can go to below option and click it. (Tools > Installed SQL Server features discovery report)

Once we click, it would fire setup.exe in the background and report various components installed and provide an HTML page as below.

As we can see it shows the version of all components installed not just SQL Engine.

Additional information:

1. The SQL Server Discovery Report is saved to %ProgramFiles%\Microsoft SQL Server\version\Setup Bootstrap\Log\<date_timestamp>\SqlDiscoveryReport.htm.

In above version would be

• 100 for SQL 2008 and SQL 2008 r2
• 110 for SQL 2012
• 120 for SQL 2014
• 130 for SQL 2016
• … and so on

1. We can also run this same report without coming to UI by running below command from a command prompt.

Setup.exe /q /Action=RunDiscovery

I have put /q for silent installation.

Have ever used this tool? I have used it for one of my clients today and they were surprised to see that they had an evaluation version of SQL installed.

Reference: Pinal Dave (http://blog.sqlauthority.com)

First appeared on SQL SERVER – Discovery Report – How to Find Information About Installed Features?

A few days ago I wrote a blog about a cluster related issue where the configuration was not same on both the nodes, which was causing issues. SQL SERVER – Unable to bring resource online. Error – Data source name not found and no default driver specified. Let us learn how to fix error related to Unable to bring resource online.

After reading above article, one of my clients contacted me and said that he is seeing the same behavior but there is no issue with drivers. What else can be done? I have asked for Cluster log again and found below.

00000d64.00001098::2016/10/18-09:07:13.839 ERR [RES] SQL Server : [sqsrvres] Worker Thread (11FE840): Failed to retrieve the ftdata root registry value (hr = 2147942402, last error = 0). Full-text upgrade will be skipped.
00000d64.00001098::2016/10/18-09:07:13.839 WARN [RES] SQL Server : [sqsrvres] Worker Thread (11FE840): ReAclDirectory : Failed to apply security to H:\MSSQL12.MSSQLSERVER\MSSQL\Data (1008).
00000d64.00001098::2016/10/18-09:07:13.995 WARN [RES] SQL Server : [sqsrvres] Worker Thread (11FE840): DoREPLSharedDataUpgrade : Failed to create working directory.
00000d64.00001384::2016/10/18-09:07:13.995 ERR [RES] SQL Server : [sqsrvres] SQL Cluster shared data upgrade failed with error 0 (worker retval = 3). Please contact customer support
00000d64.00001384::2016/10/18-09:07:13.995 ERR [RES] SQL Server : [sqsrvres] Failed to prepare environment for online. See previous message for detail. Please contact customer support

It looks like there is some problem with the cluster shared data upgrade. My client informed that this happened after a service pack was installed. So we searched in the registry to see if there were any incorrect locations being pointed and found that Under [HKLM\Software\Microsoft\Microsoft SQL Server\MSSQL12.MSSQLSERVER\Replication] we noticed that WorkingDirectory pointed to the invalid drive.

SOLUTION / WORKAROUND

We went ahead and corrected incorrect path in WorkingDirectory and then tried to bring the SQL Server resource online on node and it came online without any issues.

I was able to use Process Monitor to find the key which was needed. Have you ever used this tool?

Reference: Pinal Dave (http://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to bring resource online – Error DoREPLSharedDataUpgrade : Failed to create working directory

During my recent conversation about Oracle linked server with one of my client, I learned something new about “Microsoft OLE DB Provider for Oracle”. This blog has highlights of a lengthy conversation.

1. MSDAORA is short form for “Microsoft OLE DB Provider for Oracle”.
2. MSDAORA is driver provided by Microsoft to connect to Oracle database server.
3. There is no 64-bit version provided by Microsoft for MSDAORA. Its only available for 32 bit.
4. MSDAORA is no longer supported by current versions of Oracle. It was last updated for Oracle 9, and is no longer being updated.
5. For 64 bit, we need to download and use the Oracle client and the provider from their site for connecting with SQL. Oracle has a 64-bit version that is usable for us.
6. Oracle is a provider which is supported by Oracle.

Here is a snip from one of Microsoft connect item.

If you are using Oracle data sources, you should migrate to the Oracle-supplied provider and driver. Microsoft OLEDB Provider for Oracle (msdaora.dll) and Microsoft ODBC driver for Oracle (msorcl32.dll) are built by using Oracle Call Interface (OCI) version 7. Oracle no longer supports applications that use OCI version 7 calls, and these technologies are deprecated.

Now you may ask, how do I get 32-bit version? Well, it’s already available as a part of the operating system. You can do that by creating a UDL file as suggested in the link and opening it via 32-bit command prompt. Hopefully you know that there are two cmd.exe on 64-bit machine, one is under C:\Windows\System32 and another is under C:\Windows\SysWOW64.

I have created a UDL file and opened via both commands prompts.

As we can see above, we can see provider under 32 bit but not 64 bit.

Hopefully this blog would help you in getting the common question which I found is very difficult to search and get answer.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – How to Download Microsoft OLE DB Provider for Oracle (MSDAORA) for 64 bit?