David Barbarin » SQL Server 2012

SQL Server index rebuid online and blocking scenario

mikedavem — Sun, 30 Aug 2020 21:18:28 +0000

A couple of months ago, I experienced a problem about index rebuild online operation on SQL Server. In short, the operation was supposed to be online and to never block concurrent queries. But in fact, it was not the case (or to be more precise, it was partially the case) and to make the scenario more complex, we experienced different behaviors regarding the context. Let’s start the story with the initial context: in my company, we usually go through continuous deployment including SQL modification scripts and because we usually rely on daily pipeline, we must ensure related SQL operations are not too disruptive to avoid impacting the user experience.

Sometimes, we must introduce new indexes to deployment scripts and according to how disruptive the script can be, a discussion between Devs and Ops is initiated, and it results either to manage manually by the Ops team or to deploy it automatically through the automatic deployment pipeline by Devs.

Non-disruptive operations can be achieved in many ways and ONLINE capabilities of SQL Server may be part of the solution and this is what I suggested with one of our scripts. Let’s illustrate this context with the following example. I created a table named dbo.t1 with a bunch of rows:

USE [test];

SET NOCOUNT ON;

DROP TABLE IF EXISTS dbo.t1;
GO

CREATE TABLE dbo.t1 (
id INT IDENTITY(1,1) NOT NULL PRIMARY KEY,
col1 VARCHAR(50) NULL
);
GO

INSERT INTO dbo.t1 (col1) VALUES (REPLICATE('T', 50));
GO …
EXEC sp_spaceused 'dbo.t1'
--name rows reserved data index_size unused
--t1 5226496 1058000 KB 696872 KB 342888 KB 18240 KB

Go ahead and let’ set the context with a pattern of scripts deployment we went through during this specific deployment. Let’s precise this script is over simplified, but I keep the script voluntary simple to focus only on the most important part. You will notice the script includes two steps with operations on the same table including updating / fixing values in col2 first and then rebuilding index on col1.

/* Code before */

-- Update some values in the col1 colum
UPDATE [dbo].[t1]
SET col1 = REPLICATE('B', 50)

-- Then create an index on col1 column
CREATE INDEX [col1]
ON [dbo].[t1] (col1) WITH (ONLINE = ON);
GO

At the initial stage, the creation of index was by default (OFFLINE). Having discussed this point with the DEV team, we decided to create the index ONLINE in this context. The choice between OFFLINE / ONLINE operation is often not trivial and should be evaluated carefully but to keep simple, let’s say it was the right way to go in our context. Generally speaking, online operations are slower, but the tradeoff was acceptable in order to minimize blocking issues during this deployment. At least, this is what I thought …

In my demo, without any concurrent workload against the dbo.t1 table, creating the index offline took 6s compared to the online method with 12s. So, an expected result here …

Let’s run this another query in another session:

SELECT id, col1
FROM dbo.t1
WHERE id BETWEEN 1 AND 2

In a normal situation, this query should be blocked in a short time corresponding to the duration of the update operation. But once the update is done, blocking situation should disappear even during the index rebuild operation that is performed ONLINE.

But now let’s add Flyway to the context. Flyway is an open source tool we are using for automatic deployment of SQL objects. The deployment script was executed from it in ACC environment and we noticed longer blocked concurrent accesses this time. This goes against what we would ideally like. Digging through this issue with the DEV team, we also noticed the following message when running the deployment script:

Warning: Online index operation on table ‘dbo.t1 will proceed but concurrent access to the table may be limited due to residual lock on the table from a previous operation in the same transaction.

This is something I didn’t noticed from SQL Server Management Studio when I tested the same deployment script. So, what happened here?

Referring to the Flyway documentation, it is mentioned that Flyway always wraps the execution of an entire migration within a single transaction by default and it was exactly the root cause of the issue.

Let’s try with some experimentations:

Test 1: Update + rebuilding index online in implicit transaction mode (one transaction per query).

-- Update some values in the col1 colum
UPDATE [dbo].[t1]
SET col1 = REPLICATE('B', 50)

-- Then create an index on col1 column
CREATE INDEX [col1]
ON [dbo].[t1] (col1) WITH (ONLINE = ON);
GO
-- In another session
SELECT id, col1
FROM dbo.t1
WHERE id BETWEEN 1 AND 2

Test 2: Update + rebuilding index online within one single explicit transaction

BEGIN TRAN;

-- Update some values in the col1 colum
UPDATE [dbo].[t1]
SET col1 = REPLICATE('B', 50)

-- Then create an index on col1 column
CREATE INDEX [col1]
ON [dbo].[t1] (col1) WITH (ONLINE = ON);
GO
COMMIT TRAN;
-- In another session
SELECT id, col1
FROM dbo.t1
WHERE id BETWEEN 1 AND 2

After running these two scripts, we can notice the blocking duration of SELECT query is longer in test2 as shown in the picture below:

In the test 1, the duration of the blocking session corresponds to that for updating operation (first step of the script). However, in the test 2, we must include the time for creating the index but let’s precise the index is not the blocking operation at all, but it increases the residual locking put by the previous update operation. In short, this is exactly what the warning message is telling us. I think you can imagine easily which impact such situation may implies if the index creation takes a long time. You may get exactly the opposite of what you really expected.

Obviously, this is not a recommended situation and creating an index should be run in very narrow and constrained transaction.But from my experience, things are never always obvious and regarding your context, you should keep an eye of how transactions are managed especially when it comes automatic deployment stuff that could be quickly out of the scope of the DBA / Ops team. Strong collaboration with DEV team is recommended to anticipate this kind of issue.

See you !!

Monitoring Azure SQL Databases with Azure Monitor and Automation

mikedavem — Sun, 23 Aug 2020 15:32:07 +0000

Supervising Cloud Infrastructure is an important aspect of Cloud administration and Azure SQL Databases are no exception. This is something we are continuously improving at my company.

On-prem, DBAs often rely on well-established products but with Cloud-based architectures, often implemented through DevOps projects and developers, monitoring should be been redefined and include some new topics as:

1) Cloud service usage and fees observability
2) Metrics and events detection that could affect bottom line
3) Implementing a single platform to report all data that comes from different sources
4) Trigger rules with data if workload reaches over or drops below certain levels or when an event is enough relevant to not meet the configuration standard and implies unwanted extra billing or when it compromises the company security rules.
5) Monitoring of the user experience

A key benefit often discussed about Cloud computing, and mainly driven by DevOps, is how it enables agility. One of the meaning of term agility is tied to the rapid provisioning of computer resources (in seconds or minutes) and this shortening provisioning path enables work to start quickly. You may be tempted to grant some provisioning permissions to DEV teams and from my opinion this is not a bad thing, but it may come with some drawbacks if not under control by Ops team including database area. Indeed, for example I have in mind some real cases including architecture configuration drift, security breaches created by unwanted item changes, or idle orphan resources for which you keep being charged. All of these scenarios may lead either to security issues or extra billing and I believe it is important to get clear visibility of such events.

In my company, Azure built-in capabilities with Azure Monitor architecture are our first target (at least in a first stage) and seem to address the aforementioned topics. To set the context, we already relied on Azure Monitor infrastructure for different things including Query Performance Insight, SQL Audit analysis through Log Analytics and Azure alerts for some performance metrics. Therefore, it was the obvious way to go further by adding activity log events to the story.

In this blog post, let’s focus on the items 2) 4). I would like to share some experimentations and thoughts about them. As a reminder, items 2) 4) are about catching relevant events to help identifying configuration and security drifts and performing actions accordingly. In addition, as many event-based architectures, additional events may appear or evolve over the time and we started thinking about the concept with the following basic diagram …

… that led to the creation of the two following workflows:
– Workflow 1: To get notified immediately for critical events that may compromise security or lead immediately to important extra billing
– Workflow 2: To get a report of other misconfigured items (including critical ones) on schedule basis but don’t require quick responsiveness of Ops team.

Concerning the first workflow, using alerts on activity logs, action groups and webhooks as input of an Azure automation runbook appeared to be a good solution. On another side, the second one only requires running an Azure automation workbook on schedule basis. In fact, this is the same runbook but with different input parameters according to the targeted environment (e.g. PROD / ACC / INT). In addition, the runbook should be able to identity unmanaged events and notified Ops team who will decide either to skip it or to integrate it to runbook processing.

Azure alerts which can be divided in different categories including metric, log alerts and activity log alerts. The last one drew our attention because it allows getting notified for operation of specific resources by email or by generating JSON schema reusable from Azure Automation runbook. Focusing on the latter, we had come up (I believe) with what we thought was a reasonable solution.

Here the high-level picture of the architecture we have implemented:

1- During the creation of an Azure SQL Server or a database, corresponding alerts are added with Administrative category with a specific scope. Let’s precise that concerned operations must be registered with Azure Resource Manager in order to be used in Activity Log and fortunately they are all including in the Microsoft.Sql resource provider in this case.
2- When an event occurs on the targeted environment, an alert is triggered as well as the concerned runbook.
3- The execution of the same runbook but with different input parameters is scheduled on weekly basis to a general configuration report of our Azure SQL environments.
4- According the event, Ops team gets notified and acts (either to update misconfigured item, or to delete the unauthorized item, or to update runbook code on Git Repo to handle the new event and so on …)

The skeleton of the Azure automation runbook is pretty similar to the following one:

[OutputType("PSAzureOperationResponse")]
param
(
[Parameter (Mandatory=$false)]
[object] $WebhookData
,
[parameter(Mandatory=$False)]
[ValidateSet("PROD","ACC","INT")]
[String]$EnvTarget
,
[parameter(Mandatory=$False)]
[Boolean]$DebugMode = $False
)

If ($WebhookData)
{

# Logic to allow for testing in test pane
If (-Not $WebhookData.RequestBody){
$WebhookData = (ConvertFrom-Json -InputObject $WebhookData)
}

$WebhookBody = (ConvertFrom-Json -InputObject $WebhookData.RequestBody)

$schemaId = $WebhookBody.schemaId

If ($schemaId -eq "azureMonitorCommonAlertSchema") {
# This is the common Metric Alert schema (released March 2019)
$Essentials = [object] ($WebhookBody.data).essentials
# Get the first target only as this script doesn't handle multiple
$status = $Essentials.monitorCondition

# Focus only on succeeded or Fired Events
If ($status -eq "Succeeded" -Or $Status -eq "Fired")
{
# Extract info from webook
$alertTargetIdArray = (($Essentials.alertTargetIds)[0]).Split("/")
$SubId = ($alertTargetIdArray)[2]
$ResourceGroupName = ($alertTargetIdArray)[4]
$ResourceType = ($alertTargetIdArray)[6] + "/" + ($alertTargetIdArray)[7]

# Determine code path depending on the resourceType
if ($ResourceType -eq "microsoft.sql/servers")
{
# DEBUG
Write-Output "This is a SQL Server Resource."

$firedDate = $Essentials.firedDateTime
$AlertContext = [object] ($WebhookBody.data).alertContext
$channel = $AlertContext.channels
$EventSource = $AlertContext.eventSource
$Level = $AlertContext.level
$Operation = $AlertContext.operationName
$Properties = [object] ($WebhookBody.data).alertContext.properties
$EventName = $Properties.eventName
$EventStatus = $Properties.status
$Description = $Properties.description_scrubbed
$Caller = $Properties.caller
$IPAddress = $Properties.ipAddress
$ResourceName = ($alertTargetIdArray)[8]
$DatabaseName = ($alertTargetIdArray)[10]
$Operation_detail = $Operation.Split('/')

# Check firewall rules
If ($EventName -eq 'OverwriteFirewallRules'){
Write-Output "Firewall Overwrite is detected ..."
# Code to handle firewall update event
}
# Update DB => No need to be monitored in real time
Elseif ($EventName -eq 'UpdateDatabase') {
# Code to handle Database config update event or skip
}
# Create DB
Elseif ($EventName -eq 'CreateDatabase' -Or `
$Operation -eq 'Microsoft.Sql/servers/databases/write'){
Write-Output "Azure Database creation has been detected ..."
# Code to handle Database creation event or skip
}
# Delete DB
Elseif ($EventName -eq 'DeleteDatabase' -Or `
$Operation -eq 'Microsoft.Sql/servers/databases/delete') {
Write-Output "Azure Database has been deleted ..."
# Code to handle Database deletion event or skip
}
Elseif ($Operation -eq 'Microsoft.Sql/servers/databases/transparentDataEncryption/write') {
Write-Output "Azure Database Encryption update has been detected ..."
# Code to handle Database encryption update event or skip
}
Elseif ($Operation -eq 'Microsoft.Sql/servers/databases/auditingSettings/write') {
Write-Output "Azure Database Audit update has been detected ..."
# Code to handle Database audit update event or skip
}
Elseif ($Operation -eq 'Microsoft.Sql/servers/databases/securityAlertPolicies/write' -or $Operation -eq 'Microsoft.Sql/servers/databases/vulnerabilityAssessments/write') {
Write-Output "Azure ADS update has been detected ..."
# Code to handle ADS update event or skip
}
ElseIf ($Operation -eq 'Microsoft.Sql/servers/databases/backupShortTermRetentionPolicies/write'){
Write-Output "Azure Retention Backup has been modified ..."
# Code to handle Database retention backup update event or skip
}
# ... other ones
Else {
Write-Output "Event not managed yet "
}
else {
# ResourceType not supported
Write-Error "$ResourceType is not a supported resource type for this runbook."
}
}
}
Else {
# The alert status was not 'Activated' or 'Fired' so no action taken
Write-Verbose ("No action taken. Alert status: " + $status) -Verbose
}
}
Else{
# SchemaID doesn't correspond to azureMonitorCommonAlertSchema =>> Skip
Write-Host "Skip ..."
}
}
Else {
Write-Output "No Webhook detected ... switch to normal mode ..."

If ([String]::IsNullOrEmpty($EnvTarget)){
Write-Error '$EnvTarget is mandatory in normal mode'
}

#########################################################
# Code for a complete check of Azure SQL DB environment #
#########################################################
}

Some comments about the PowerShell script:

1) Input parameters should include either the Webhook data or specific parameter values for a complete Azure SQL DB check.
2) The first section should include your own functions to respond to different events. In our context, currently we drew on DBAChecks thinking to develop a derived model but why not using directly DBAChecks in a near future?
3) When an event is triggered, a JSON schema is generated and provides insight. The point here is you must navigate through different properties according to the operation type (cf. BOL).
4) The increase of events to manage could be a potential issue making the runbook fat especially if we keep both the core functions and event processing. To mitigate this topic, we are thinking to move functions into modules in Azure automation (next step).

Bottom line

Thanks to Azure built-in capabilities we improved our visibility of events that occur on the Azure SQL environment (both expected and unexcepted) and we’re now able to act accordingly. But I should tell you that going this way is not a free lunch and we achieved a reasonable solution after some programming and testing efforts. If you can invest time, it is probably the kind of solution you can add to your study.

See you

Dealing with SQL Server on Linux on WSL2

mikedavem — Mon, 27 Jul 2020 06:53:32 +0000

This is a blog post I intended to write sometime ago … about using SQL Server on Linux on WSL2. For whom already installed it on Windows 10, version 2004, you are already aware it doesn’t come with the support of systemd. Indeed, although it does exist in the file system, systemd is not running. If you intend to use directly SQL Server on WSL2, you need to read this carefully, because installation and management relies precisely on systemd!

First, let’s say if you want to run SQL Server on WSL2 in a supported way, docker containers remain probably the better fit. Podman may be also an alternative but it seems to remain an experimental stuff so far. The interesting point is that some people prefer to use Docker directly on WSL2 whereas other ones prefer to use Docker Desktop for Windows with the integration of WSL2 based engine. The option seems to be available for a few months as shown below:

But this is not the main topic of this blog post. Having interesting discussions with some of my colleagues, we wanted to know if we were able to install directly SQL Server on WSL2 and the good news is we finally managed to do it but at the cost of some (dirty) tricks required to make SQL Server starting / stopping correctly.

Before continuing, let’s precise we did it in a pure experimental and academic way and this is definitely not supported by Microsoft. But before starting my holidays in August, it was something very fun and a good reminder of some Linux concepts …

I did my test with the Ubuntu-18.04 distro in WSL v2.

First step consisted in installing SQL Server in an usual way as per the Microsoft BOL. The installation was done correctly but after running the setup, an error message raised quickly in the last step related to service initialization:

In fact, if you think about it, this message is expected because systemd is not running on WSL2 and the configuration process attempts to initialize the mssql-server service based on systemd. But let’s dig further on the installation process … Referring to how a deb package is made, there exist preinst, postinst, prerm, postrm scripts for mssqlserver deb package located in /var/lib/dpkg/info/:

$ sudo ls -l /var/lib/dpkg/info/mssql-server*
-rwxr-xr-x 1 root root 108 Jun 16 20:01 /var/lib/dpkg/info/mssql-server.config
-rw-r--r-- 1 root root 8579 Jul 26 00:31 /var/lib/dpkg/info/mssql-server.list
-rw-r--r-- 1 root root 11543 Jun 16 20:01 /var/lib/dpkg/info/mssql-server.md5sums
-rwxr-xr-x 1 root root 1436 Jun 16 20:01 /var/lib/dpkg/info/mssql-server.postinst
-rwxr-xr-x 1 root root 289 Jun 16 20:01 /var/lib/dpkg/info/mssql-server.postrm
-rwxr-xr-x 1 root root 1353 Jun 16 20:01 /var/lib/dpkg/info/mssql-server.preinst
-rwxr-xr-x 1 root root 365 Jun 16 20:01 /var/lib/dpkg/info/mssql-server.prerm
-rw-r--r-- 1 root root 72 Jun 16 20:01 /var/lib/dpkg/info/mssql-server.shlibs
-rw-r--r-- 1 root root 305 Jun 16 20:00 /var/lib/dpkg/info/mssql-server.templates
-rw-r--r-- 1 root root 74 Jun 16 20:01 /var/lib/dpkg/info/mssql-server.triggers

My suspicion was those files content references of systemctl commands and yes, there were in preinst/prerm/postins/prerm files:

Well, we found a reasonable explanation of the previous error message at least. At this stage, I would say you can use your new fresh installed SQL Server instance, but it requires to manually start / stop it because there are no systemctl commands to handle it. But thinking about it … we are still using the famous init daemon (PID1) on WSL2 and it works for regular services.

A good alternative could be to rely on init scripts and start-stop-daemon wrapped into LSB-compliant init scripts with:

– At least start, stop, restart, force-reload, and status
– Return Proper exit code
– Document run-time dependencies

LSB provides default set of functions which is in /lib/lsb/init-functions and we can make use of those functions in our Init scripts. The script file is located to /etc/init.d and named mssql-server.

$ sudo ls -l /etc/init.d/mssql*
-rwxr-xr-x 1 root root 1606 Jul 26 22:13 /etc/init.d/mssql-server

Here the content of my mssql-server script file:

#! /bin/sh -e
#
### BEGIN INIT INFO
# Provides: sqlserver
# Required-Start: $all
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Manages SQL Server instance on Linux
### END INIT INFO

DAEMON="/opt/mssql/bin/sqlservr"
daemon_OPT=""
DAEMONUSER="mssql"
daemon_NAME="sqlservr"

export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/var/opt/mssql

# Check sqlserver is present
if [ ! -x $DAEMON ]; then
log_failure_msg "$DAEMON not present or not executable"
exit 1
fi

# Load init functions
. /lib/lsb/init-functions

d_start () {
log_daemon_msg "Starting system $daemon_NAME Daemon"
start-stop-daemon --background --name $daemon_NAME --start --quiet --chuid $DAEMONUSER --exec $DAEMON --umask 007 -- $DAEMON_OPTS --oknodo
log_end_msg $?
}

d_stop () {
log_daemon_msg "Stopping system $daemon_NAME Daemon"
start-stop-daemon --name $daemon_NAME --stop --retry 5 --quiet --name $daemon_NAME --oknodo
log_end_msg $?
}

case "$1" in

start|stop)
d_${1}
;;

restart|reload|force-reload)
d_stop
d_start
;;

force-stop)
d_stop
;;

status)
status_of_proc "$daemon_NAME" "$DAEMON" "system-wide $daemon_NAME" && exit 0 || exit $?
;;
*)
echo "Usage: /etc/init.d/$daemon_NAME {start|stop|force-stop|restart|reload|force-reload|status}"
exit 1
;;
esac
exit 0

Thus, I was now able to manage with status / start / stop / restart operations:

$ sudo service mssql-server status
/opt/mssql/bin/sqlservr is not running
$ sudo service mssql-server start
Starting system sqlservr Daemon [ OK ]

The SQL Server engine was started successfully, and I can double check with the top command …

… and with a quick connect to my SQL Server instance:

As starting my SQL Server instance, stopping it can be performed with the service command and stop:

$ sudo service mssql-server stop
Stopping system sqlservr Daemon [ OK ]

$ sudo service mssql-server status
/opt/mssql/bin/sqlservr is not running

Generally speaking, there is no concept of runlevel with WSL, so starting automatically the mssql-service with the WSL2 startup can be achieved in different ways with entries in .bashrc file or tricks from github projects (not tested from my side).

Finally, let’s talk about removing SQL Server from WSL2. As a reminder, during the installation / setup we faced an error message related to systemd. We also identified some dependencies to systemd in some dkpg files as shown below:

Similar to the installation step, you will experience the same kind of issue with prerm / postrm files when removing SQL Server. In my case, I had to comment concerned lines in those files to uninstall successfully my SQL Server instance.

Hope this blog post helps!

dbachecks and AlwaysOn availability group checks

mikedavem — Mon, 20 Apr 2020 19:57:31 +0000

When I started my DBA position in my new company, I was looking for a tool that was able to check periodically the SQL Server database environments for several reasons. First, as DBA one of my main concern is about maintaining and keeping the different mssql environments well-configured against an initial standard. It is also worth noting I’m not the only person to interact with databases and anyone in my team, which is member of sysadmin server role as well, is able to change any server-level configuration settings at any moment. In this case, chances are that having environments shifting from our initial standard over the time and my team and I need to keep confident by checking periodically the current mssql environment configurations, be alerting if configuration drifts exist and obviously fix it as faster as possible.

A while ago, I relied on SQL Server Policy Based Management feature (PBM) to carry out this task at one of my former customers and I have to say it did the job but with some limitations. Indeed, PBM is the instance-scope feature and doesn’t allow to check configuration settings outside the SQL Server instance for example. During my investigation, dbachecks framework drew my attention for several reasons:

– It allows to check different settings at different scopes including Operating System and SQL Server instance items
– It is an open source project and keeps evolving with SQL / PowerShell community contributions.
– It is extensible, and we may include custom checks to the list of predefined checks shipped with the targeted version.
– It is based on PowerShell, Pester framework and fits well with existing automation and GitOps process in my company

The first dbacheck version we deployed in production a couple of month ago was 1.2.24 and unfortunately it didn’t include reliable tests for availability groups. It was the starting point of my first contributions to open source projects and I felt proud and got honored when I noticed my 2 PRs validated for the dbacheck tool including Test Disk Allocation Unit and Availability Group checks:

Obviously, this is just an humble contribution and to be clear, I didn’t write the existing tests for AGs but I spent some times to apply fixes for a better detection of all AG environments including their replicas in a simple and complex topologies (several replicas on the same server and non-default ports for example).

So, here the current list of AG checks in the version 1.2.29 at the moment of this write-up:

– Cluster node should be up
– AG resource + IP Address in the cluster should be online
– Cluster private and public network should be up
– HADR should be enabled on each AG replica
– AG Listener + AG replicas should be pingable and reachable from client connections
– AG replica should be in the correct domain name
– AG replica port number should be equal to the port specified in your standard
– AG availability mode should not be in unknown state and should be in synchronized or synchronizing state regarding the replication type
– Each high available database (member of an AG) should be in synchronized / synchronizing state, ready for failover, joined to the AG and not in suspended state
– Each AG replica should have an extended event session called AlwaysOn_health which is in running state and configured in auto start mode

Mandatory parameters are app.cluster and domain.name.

Get-DbcCheck -Tag HADR | ft Group, Type, AllTags, Config -AutoSize

Group Type AllTags Config
----- ---- ------- ------
HADR ClusterNode ClusterHealth, HADR app.sqlinstance app.cluster skip.hadr.listener.pingcheck domain.name policy...

The starting point of the HADR checks is the Windows Failover Cluster component and hierarchically other tests are performed on each sub component including availability group, AG replicas and AG databases.

Then you may change the behavior on the HADR check process according to your context by using the following parameters:

– skip.hadr.listener.pingcheck => Skip ping check of hadr listener
– skip.hadr.listener.tcpport => Skip check of standard tcp port about AG listerners
– skip.hadr.replica.tcpport => Skip check of standard tcp port about AG replicas

For instance, in my context, I configured the hadr.replica.tcpport parameter to skip checks on replica ports because we own different environments that including several replicas on the same server and which listen on a non-default port.

Get-DbcConfig skip.hadr.*
Name Value Description
---- ----- -----------
skip.hadr.listener.pingcheck False Skip the HADR listener ping test (especially useful for Azure and AWS)
skip.hadr.listener.tcpport False Skip the HADR AG Listener TCP port number (If port number is not standard acro...
skip.hadr.replica.tcpport True Skip the HADR Replica TCP port number (If port number is not standard across t...

Running the HADR check can be simply run by using HADR tag as follows:

Invoke-DbcCheck -Tag HADR
Pester v4.10.1 Executing all tests in 'C:\Program Files\WindowsPowerShell\Modules\dbachecks\1.2.29\checks\HADR.Tests.ps1' with Tags HADR
...

Well, this a good start but I think some almost of checks are state-oriented and some configuration checks are missing. I’m already willing to add some of them in a near the future or/and feel free to add your own contribution as well

Stay tuned!

Availability Group 2017 Direct seeding and updated default path policy

mikedavem — Fri, 13 Mar 2020 18:10:10 +0000

A couple of days ago, I ran into an issue when adding a new database in direct seeding mode that led me to reconsider refreshing my skills on this feature. Going through the AG database wizard for adding database, I faced the following error message …

… and I was surprised by the required directories value (L:\SQL\Data) because the correct topology should be:

– D:\SQL\Data (SQL data files)
– L:\SQL\Logs (SQL Log files)

SQL Server 2016 required to have symmetric storage layout for both AG replicas but SQL Server 2017 and above seems to tell another story as specified to the BOL. In my context, I got the check script executed by the wizard and it became obvious that the direct seeding feature is checking if folders based on default path values exist on each replica.

In my context, I got the check script executed by the wizard and it became obvious that the direct seeding feature is checking if folders based on default path values exist on each replica.

declare @SmoAuditLevel int
exec master.dbo.xp_instance_regread N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'AuditLevel', @SmoAuditLevel OUTPUT

declare @NumErrorLogs int
exec master.dbo.xp_instance_regread N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'NumErrorLogs', @NumErrorLogs OUTPUT

declare @SmoLoginMode int
exec master.dbo.xp_instance_regread N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'LoginMode', @SmoLoginMode OUTPUT

declare @ErrorLogSizeKb int
exec master.dbo.xp_instance_regread N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'ErrorLogSizeInKb', @ErrorLogSizeKb OUTPUT

declare @SmoMailProfile nvarchar(512)
exec master.dbo.xp_instance_regread N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'MailAccountName', @SmoMailProfile OUTPUT

declare @BackupDirectory nvarchar(512)
if 1=isnull(cast(SERVERPROPERTY('IsLocalDB') as bit), 0)
select @BackupDirectory=cast(SERVERPROPERTY('instancedefaultdatapath') as nvarchar(512))
else
exec master.dbo.xp_instance_regread N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'BackupDirectory', @BackupDirectory OUTPUT

declare @SmoPerfMonMode int
exec master.dbo.xp_instance_regread N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'Performance', @SmoPerfMonMode OUTPUT

if @SmoPerfMonMode is null
begin
set @SmoPerfMonMode = 1000
end

SELECT
@SmoAuditLevel AS [AuditLevel],
ISNULL(@NumErrorLogs, -1) AS [NumberOfLogFiles],
(case when @SmoLoginMode < 3 then @SmoLoginMode else 9 end) AS [LoginMode],
ISNULL(SERVERPROPERTY('instancedefaultdatapath'),'') AS [DefaultFile],
SERVERPROPERTY('instancedefaultlogpath') AS [DefaultLog],
ISNULL(@ErrorLogSizeKb, 0) AS [ErrorLogSizeKb],
-1 AS [TapeLoadWaitTime],
ISNULL(@SmoMailProfile,N'') AS [MailProfile],
@BackupDirectory AS [BackupDirectory],
@SmoPerfMonMode AS [PerfMonMode]

Primary replica

Secondary replica

Even if directing seeding allows asymmetric storage layout, a mistake was introduced in my context and both replicas should have been aligned. It is therefore all the more important that using direct seeding capabilities from PowerShell cmdlets like Add-DbaAgDatabase doesn’t generate any errors and fixing the default path value for data and log require restarting the SQL Server instance. See you!

Hope this tips helps!

See you!

SQL Server service broker and curious duplicate messages case

mikedavem — Sun, 22 Dec 2019 18:45:44 +0000

One of my last write-up to finish the year 2019. A couple of days ago, I ran into an interesting issue concerning a service broker architecture. Let’s introduce quickly the context: this is a start-based architecture with one target and more than 80 initiators as shown below:

All messages are sent from the initiators to a single central target. Windows authentication for endpoint’s connection and messages are encrypted before to be sent to the remote service by using certificates mapped to database users.

Installing such infrastructure is not trivial and during this year, we often had to deploy and reinitialize service broker configurations because we added new initiators, or some initiators were replaced by other ones. Therefore, on the DBA side we worked on an automatic way to interact with service broker configuration and it turns out that the natural way for us was to write a PowerShell module to address it.

A couple of days, for a sake of curiosity I implemented a service broker event based XE session and I noticed a bunch of broker_message_undeliverable messages with the reason: The message could not be delivered because it is a duplicate. See below:

What does it mean exactly? Well, according to my Google-fu and very-explained blog from Remus Rusanu, it seems that the messages are accepted by the target, but the acks don’t reach back to the sender and therefore the sender is retrying the message again and again. But in my case, the sender ended up managing sending back the acknowledgement.

First step was to identify a misconfigured route between the target and the corresponding initiator. From broker instance value corresponds to the service broker id of the initiator. But surprisingly I didn’t find any misconfigured item. But I remembered the campaign of server rollout that had been going on for a few months where some servers were renamed following a new named convention. Furthermore, in our deployment process, dealing with new or renamed servers is exactly the same and we just initialize a new initiator and install a new route on the target regardless the scenario. But we forget one important point: because all routes are based on server name renaming a server leads to potentially to duplicate routes in this case. I verified my assumption by counting the number of existing routes on the target site:

SELECT COUNT(*) FROM sys.routes

The result was 148 and we expect to get 85 routes in the current architecture confirming we have duplicate items here. Next step consisted in identifying and fixing duplicate routes:

Here a sample of duplicate routes I founded. The renaming server kept its service broker id and I easily identified duplicates by grouping by the broker instance.

After fixing the issue, messages related to duplicates disappear from the XE output.
Hope this helps!

David Barbarin

Recherche d’index et IO résiduels

mikedavem — Mon, 12 Feb 2018 17:57:10 +0000

Ce billet fait suite au précédent à propos des recherches d’index et les pièges associés. J’ai pu rencontrer un autre cas intéressant mais ce n’est pas tellement la nature du problème qui m’a décidé à écrire mais plutôt les différentes façons qu’il existe de l’appréhender.

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

Déplacer des tables en ligne vers des groupes de fichiers avec contraintes et LOB

mikedavem — Thu, 25 Jan 2018 07:01:00 +0000

Commençons cette semaine en revenant sur une discussion intéressante que j’ai eu avec un de mes clients pour déplacer plusieurs tables vers différents groupes de fichiers. Certaines d’entre elles contenaient des données LOB. Ajoutons à cela une contrainte supplémentaire du client: déplacer tout ce petit monde en ligne pour éviter d’impacter la disponibilité des données durant le processus de migration. Les tables concernées possèdaient des contraintes de schéma comme une clé primaire et des clés étrangères ainsi que des index non cluster. Finalement quelque chose de plutôt classique avec lequel nous pouvons avoir à faire tous les jours chez les clients.

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

SQL Saturday Toulouse 2017: SQL Server sur Linux – les slides

mikedavem — Tue, 02 Jan 2018 19:10:38 +0000

#SQLSatToulouse a été un bon cru. Premierèment, je voudrais remercier chaleureusement l’équipe organisatrice (@Guss et @Fredg_31) ainsi que les sponsors et participants (sans qui ce qui ce genre d’événement ne serait pas possible).

Comme promis, vous trouverez les slides de ma session « Introduction to SQL Server on Linux for DBAs« . Définitivement un bon moment de partage.

Merci à tous!

David Barbarin
MVP & MCM SQL Server

Première installation de SQL Server sur Linux Red Hat 7.2

mikedavem — Sun, 15 Jan 2017 15:56:05 +0000

Les premières annonces de Microsoft sur la sortie d’une version sous Linux sont prometteuses. Bien entendu nous sommes loin d’une version finale et la liste des fonctionnalités non supportées sera probablement un bon indicateur de la maturité du produit avec le temps. Comme beaucoup de gens je suppose, je n’ai pas attendu longtemps pour tenter une première installation sur Linux. Le processus d’installation semble bien documenté et ca sera l’occasion de partager mes premières impressions.

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server