David Barbarin » SQL Server 2005

Extending SQL Server monitoring with Raspberry PI and Lametric

mikedavem — Thu, 07 Jan 2021 21:59:25 +0000

First blog of this new year 2021 and I will start with a fancy and How-To Geek topic

In my last blog post, I discussed about monitoring and how it should help to address quickly a situation that is going degrading. Alerts are probably the first way to raise your attention and, in my case, they are often in the form of emails in a dedicated folder. That remains a good thing, at least if you’re not focusing too long in other daily tasks or projects. In work office, I know I would probably better focus on new alerts but as I said previously, telework changed definitely the game.

I wanted to find a way to address this concern at least for main SQL Server critical alerts and I thought about relying on my existing home lab infrastructure to address the point. Reasons are it is always a good opportunity to learn something and to improve my skills by referring to a real case scenario.

My home lab infrastructure includes a cluster of Raspberry PI 4 nodes. Initially, I use it to improve my skills on K8s or to study some IOT stuff for instance. It is a good candidate for developing and deploying a new app for detecting new incoming alerts in my mailbox and sending notifications to my Lametric accordingly.

Lametric is a basically a connected clock but works also as a highly-visible display showing notifications from devices or apps via REST APIs. First time I saw such device in action was in a DevOps meetup in 2018 around Docker and Jenkins deployment with Eric Dusquenoy and Tim Izzo (@5ika_). In addition, one of my previous customers had also one in his office and we had some discussions about cool customization through Lametric apps.

Connection through VPN to my company network is mandatory to work from home and unfortunately Lametric device doesn’t support this scenario because communication is limited to local network only. So, I need an app that run on my local (home) network and able to connect to my mailbox, get new incoming emails and finally sending notifications to my Lametric device.

Here my setup:

There are plenty of good blog posts to create a Raspberry cluster on the internet and I would suggest to read that of Andrew Pruski (@dbafromthecold).

As shown above, there are different paths for SQL alerts referring our infrastructure (On-prem and Azure SQL databases) but all of them are send to a dedicated distribution list for DBA.

The app is a simple PowerShell script that relies on Exchange Webservices APIs for connecting to the mailbox and to get new mails. Sending notifications to my Lametric device is achieved by a simple REST API call with well-formatted body. Details can be found the Lametric documentation. As prerequisite, you need to create a notification app from Lametric Developer site as follows:

As said previously, I used PowerShell for this app. It can help to find documentation and tutorials when it comes Microsoft product. But if you are more confident with Python, APIs are also available in a dedicated package. But let’s precise that using PowerShell doesn’t necessarily mean using Windows-based container and instead I relied on Linux-based image with PowerShell core for ARM architecture. Image is provided by Microsoft on Docker Hub. Finally, sensitive information like Lametric Token or mailbox credentials are stored in K8s secret for security reasons. My app project is available on my GitHub. Feel free to use it.

Here some results:

– After deploying my pod:

– The app is running and checking new incoming emails (kubectl logs command)

When email is detected, notification is sendig to Lametric device accordingly

Geek fun good (bad?) idea to start this new year 2021

AAD user creation on behalf AAD Service Principal with Azure SQL DB

mikedavem — Sun, 02 Aug 2020 22:28:06 +0000

An interesting improvement was announced by the SQL AAD team on Monday 27th July 2020 and concerns the support for Azure AD user creation on behalf of Azure AD Applications for Azure SQL as mentioned to this Microsoft blog post.

In my company, this is something we were looking for a while with our database refresh process in Azure. Before talking this new feature, let me share a brief history of different considerations we had for this DB refresh process over the time with different approaches we went through. First let’s precise DB Refresh includes usually at least two steps: restoring backup / copying database – you have both ways in Azure SQL Database – and realigning security context with specific users regarding your targeted environment (ACC / INT …). But the latter is not as trivial as you may expect if you opted to use either a SQL Login / User or a Service Principal to carry out this operation in your process. Indeed, in both cases creating an Azure AD User or Group is not supported, and if you try you will face this error message:

‘’ is not a valid login or you do not have permission.

All the stuff (either Azure automation runbook and PowerShell modules on-prem) done so far and described afterwards meets the same following process:

First, we used Invoke-SQCMD in Azure Automation runbook with T-SQL query to create a copy of a source database to the target server. T-SQL is mandatory in this case as per documented in the Microsoft BOL because PROD and ACC or INT servers are not on the same subscription. Here a simplified sample of code:

...
$CopyDBCMD = @{
'Database' = 'master'
'ServerInstance' = $TargetServerName
'Username' = $SQLUser
'Password' = $SQLPWD
'Query' = 'CREATE DATABASE '+ '[' + $DatabaseName + '] ' + 'AS COPY OF ' + '[' + $SourceServerName + '].[' + $DatabaseName + ']'
}

Invoke-Sqlcmd @CopyDBCMD
...

But as you likely know, Invoke-SQLCMD doesn’t support AAD authentication and because SQL Login authentication was the only option here, it led us dealing with an annoying issue about the security configuration step with AAD users or groups as you may imagine.

Then, because we based authentication mainly on trust architecture and our security rules require using it including apps with managed identities or service principals, we wanted also to introduce this concept to our database refresh process. Fortunately, service principals are supported with Azure SQL DBs since v12 with access token for authentication by ADALSQL. The corresponding DLL is required on your server or if you use it from Azure Automation like us, we added the ADAL.PS module but be aware it is now deprecated, and I advise you to strongly invest in moving to MSAL. Here a sample we used:

...
$response = Get-ADALToken `
-ClientId $clientId `
-ClientSecret $clientSecret `
-Resource $resourceUri `
-Authority $authorityUri `
-TenantId $tenantName

...

$connectionString = "Server=tcp:$SqlInstanceFQDN,1433;Initial Catalog=master;Persist Security Info=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;"
# Create the connection object
$connection = New-Object System.Data.SqlClient.SqlConnection($connectionString)
# Set AAD generated token to SQL connection token
$connection.AccessToken = $response.AccessToken

Try {
$connection.Open()
...
}
...

But again, even if the copy or restore steps are well managed, we still got stuck with security reconfiguration, because service principals were not supported for creating AAD users or groups so far …

In the meantime, we found out a temporary and interesting solution based on dbatools framework and the Invoke-dbaquery command which supports AAD authentication (Login + Password). As we may not rely on service principal in this case, using a dedicated AAD account was an acceptable tradeoff to manage all the database refresh process steps. But going through this way comes with some disadvantages because running Invoke-dbaquery in a full Azure automation mode is not possible with missing ADALsql.dll. Workaround may be to use hybrid-worker, but we didn’t want to add complexity to our current architecture only for this special case. Instead we decided to move the logic of the Azure automation runbook into on-prem PowerShell framework which already include logic for DB refresh for on-prem SQL Server instances.

Here a simplified sample of code we are using:

...
Try {
# Connect to get access to Key Vault info
Connect-AzAccount | Out-Null

[String]$user = (Get-AzKeyVaultSecret -VaultName $KeyvaultName -Name "AZSQL-SQLBCKUSER").SecretValueText
[System.Security.SecureString]$pwd = ConvertTo-SecureString (Get-AzKeyVaultSecret -VaultName $KeyvaultName -Name "AZSQL-SQLBCKPWD").SecretValueText -AsPlainText -Force
[String]$SourceServerName = (Get-AzKeyVaultSecret -VaultName $KeyvaultName -Name "AZSQL-NAME").SecretValueText
[String]$TargetServerName = (Get-AzKeyVaultSecret -VaultName $KeyvaultName -Name "AZSQL-TARGETNAME").SecretValueText + '.database.windows.net'

# DB Restore will be performed in the context of dedicated AAD account
$pscredential = New-Object -TypeName System.Management.Automation.PSCredential($user, $pwd)

Write-Host "Restoring DB:$DatabaseName from Source Server: $SourceServerName to Target Server: $TargetServerName"

$Query = "CREATE DATABASE [$DatabaseName] AS COPY OF [$SourceServerName].[$DatabaseName]"
Invoke-DbaQuery `
-SqlInstance $TargetServerName `
-Database master `
-SqlCredential $pscredential `
-Query $Query `
-EnableException

# Wait for DB online and ready ...
# Code should be implemented for this check

Write-Output "Applying security configuration to DB: $DatabaseName on Server:$TargetServerName"

$Query = "
DROP USER [az_sql_ro];CREATE USER [az_sql_ro] FROM EXTERNAL PROVIDER;
"
Invoke-DbaQuery `
-SqlInstance $TargetServerName `
-Database $DatabaseName `
-SqlCredential $pscredential `
-Query $Query `
-EnableException

}
Catch {
Write-Host "Error encountered: $($_.Exception.Message)"
}
...

Referring to the PowerShell code above, in the second step, we create an AAG group [az_sql_ro] on behalf of the AAD dedicated account with the CLAUSE FROM EXTERNAL PROVIDER.

Finally, with the latest news published by the SQL AAD team, we will likely consider using back service principal instead of dedicated Windows AAD account. This Microsoft blog post explains in details how it works and what you have to setup to make it work correctly. I don’t want to duplicate what is already explained so I will apply the new stuff to my context.

Referring to the above blog post, you need first to setup a server identity for your Azure SQL Server as below:

Set-AzSqlServer `
-ResourceGroupName sandox-rg `
-ServerName a-s-sql02 `
-AssignIdentity

# Check server identity
Get-AzSqlServer `
-ResourceGroupName sandox-rg `
-ServerName a-s-sql02 | `
Select-Object ServerName, Identity

ServerName Identity
---------- --------
a-s-sql02 Microsoft.Azure.Management.Sql.Models.ResourceIdentity

Let’s have a look at the server identity

# Get identity details
$identity = Get-AzSqlServer `
-ResourceGroupName sandox-rg `
-ServerName a-s-sql02

$identity.identity

PrincipalId Type TenantId
----------- ---- --------
7f0d16f7-b172-4c97-94d3-34f0f7ed93cf SystemAssigned 2fcd19a7-ab24-4aef-802b-6851ef5d1ed5

In fact, assigning a server identity means creating a system assigned managed identity in the Azure AD tenant that’s trusted by the subscription of the instance. To keep things simple, let’s say that System Managed Identity in Azure is like to Managed Account or Group Managed Account on-prem. Those identities are self-managed by the system. Then you need to grant this identity the Azure AD « Directory Readers « permission to get rights for creating AAD Users or Groups on behalf of this identity. A PowerShell script is provided by Microsoft here a sample of code I applied in my context for testing:

...
Try {
$DatabaseName = "test-DBA"

# Connect to get access to Key Vault info
Connect-AzAccount | Out-Null

[String]$user = (Get-AzKeyVaultSecret -VaultName $KeyvaultName -Name "AZSQL-SQLBCKAPPID").SecretValueText
[System.Security.SecureString]$pwd = ConvertTo-SecureString (Get-AzKeyVaultSecret -VaultName $KeyvaultName -Name "AZSQL-SQLBCKAPPSECRET").SecretValueText -AsPlainText -Force
[String]$SourceServerName = (Get-AzKeyVaultSecret -VaultName $KeyvaultName -Name "AZSQL-NAME").SecretValueText
[String]$TargetServerName = (Get-AzKeyVaultSecret -VaultName $KeyvaultName -Name "AZSQL-TARGETNAME").SecretValueText + '.database.windows.net'

# DB Restore will be performed in the context of dedicated AAD account
$pscredential = New-Object -TypeName System.Management.Automation.PSCredential($user, $pwd)

$adalPath = "${env:ProgramFiles}\WindowsPowerShell\Modules\Az.Profile.7.0\PreloadAssemblies"
# To install the latest AzureRM.profile version execute -Install-Module -Name AzureRM.profile
$adal = "$adalPath\Microsoft.IdentityModel.Clients.ActiveDirectory.dll"
$adalforms = "$adalPath\Microsoft.IdentityModel.Clients.ActiveDirectory.WindowsForms.dll"
[System.Reflection.Assembly]::LoadFrom($adal) | Out-Null
$resourceAppIdURI = 'https://database.windows.net/'

# Set Authority to Azure AD Tenant
$authority = 'https://login.windows.net/' + $tenantId

$ClientCred = [Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential]::new($clientId, $clientSecret)
$authContext = [Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext]::new($authority)
$authResult = $authContext.AcquireTokenAsync($resourceAppIdURI,$ClientCred)
$Tok = $authResult.Result.CreateAuthorizationHeader()
$Tok=$Tok.Replace("Bearer ","")

Write-host "Token generated is ..."
$Tok
Write-host ""

Write-Host "Create SQL connectionstring"
$conn = New-Object System.Data.SqlClient.SQLConnection

$conn.ConnectionString = "Data Source=$TargetServerName;Initial Catalog=master;Connect Timeout=30"
$conn.AccessToken = $Tok

Write-host "Connect to database and execute SQL script"
$conn.Open()

Write-Host "Check connected user ..."
$Query = "SELECT USER_NAME() AS [user_name];"
$command = New-Object -TypeName System.Data.SqlClient.SqlCommand($Query, $conn)
$Command.ExecuteScalar()
$conn.Close()

Write-Host "Restoring DB:$DatabaseName from Source Server: $SourceServerName to Target Server: $TargetServerName"

$conn.ConnectionString = "Data Source=$TargetServerName;Initial Catalog=master;Connect Timeout=30"
$conn.AccessToken = $Tok
$conn.Open()
$Query = "DROP DATABASE IF EXISTS [$DatabaseName]; CREATE DATABASE [$DatabaseName] AS COPY OF [$SourceServerName].[$DatabaseName]"
$command = New-Object -TypeName System.Data.SqlClient.SqlCommand($Query, $conn)
$command.CommandTimeout = 1200
$command.ExecuteNonQuery()
$conn.Close()

# Wait for DB online and ready ...
# Code should be implemented for this check

Write-Output "Applying security configuration to DB: $DatabaseName on Server:$TargetServerName"

$conn.ConnectionString = "Data Source=$TargetServerName;Initial Catalog=$DatabaseName;Connect Timeout=30"
$conn.AccessToken = $Tok
$conn.Open()
$Query = 'CREATE USER [az_sql_ro] FROM EXTERNAL PROVIDER;'
$command = New-Object -TypeName System.Data.SqlClient.SqlCommand($Query, $conn)
$command.ExecuteNonQuery()
$conn.Close()

}
Catch {
Write-Output "Error encountered: $($_.Exception.Message)"
}
...

Using service principal required few changes in my case. I now get credentials of the service principal (ClientId and Secret) from Azure Key Vault instead of the AAD dedicated account used in previous example. I also changed the way to connect to SQL Server by relying on ADALSQL to get the access token instead of using dbatools commands. Indeed, as far as I know, dbatools doesn’t support this authentication way (yet?).

The authentication process becomes as follows:

My first test seems to be relevant:

This improvement looks promise and may cover broader scenarios as the one I described in this blog post. This feature is in preview at the moment of this write-up and I hope to see it coming soon in GA as well as a potential support of preferred PowerShell framework DBAtools

See you!

Lorsqu’une recherche d’index n’est pas forcément adéquate

mikedavem — Thu, 13 Oct 2016 18:35:29 +0000

N’avez-vous jamais considéré une recherche d’index comme un problème? Laissez moi vous raconter une histoire avec un de mes clients avec un contexte simple: une requête spécifique qui n’était pas dans les valeurs acceptables de performance exigées (environ 200ms de temps d’exécution moyen). Le plan d’exécution associé de la requête était similaire à ce que vous pouvez voir ici

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

Les 24 heures du PASS (24HOP) – édition francophone

mikedavem — Sat, 24 Sep 2016 06:25:00 +0000

Cette année a eu lieu la première édition francophone des 24h du PASS (les 20 et 21 septembre 2016). L’idée était plutôt simple: proposer une série de 24 webinaires gratuits de 10h jusqu’à 22h (heure française) pendant 2 jours. C’était l’occasion de recevoir et d’échanger les dernières informations autour de l’administration et du développement des bases de données, des nouvelles tendances côté Business Intelligence et du Cloud.

Pour ma part, c’est avec plaisir que j’ai eu l’occasion d’échanger avec vous autour de 2 sujets: tempdb et bonnes pratiques ainsi que des columnstore et leur implication dans les nouvelles tendances d’architecture BI avec Thoi Dung TSP Microsoft Switzerland.

C’est encore l’occasion de remercier les sponsors et Isabelle sans qui ce type d’événement n’aurait certainement pas eu lieu.

Les slides et démos devraient arriver sous peu!

David Barbarin
MVP & MCM SQL Server

Chute de l’espérance de vie d’une page mais ne paniquez pas tout de suite!

mikedavem — Thu, 03 Mar 2016 05:37:22 +0000

Il y a quelques semaines de cela, j’ai eu une discussion intéressante avec un de mes clients a propos de la surveillance du fameux page life expectancy (ou PLE). Il me demandait si cela était une bonne pratique de surveiller sa valeur parce qu’il avait remarqué une grosse chute de ce dernier en dessous des recommandations en vigueur et ceci durant la nuit ou quelques fois pendant la journée. En plus, il craignait d’être déranger la nuit à cause de tâche de maintenance qui s’exécuterait sans impact sur l’activité business à ce moment là.

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

NOLOCK n’est définitivement pas ce que vous croyez

mikedavem — Fri, 19 Feb 2016 07:37:22 +0000

Pour ceux qui croient encore qu’utiliser le hint nolock se prémunit contre tout verrou, lisez la suite de ce billet. J’avais déjà écrit un billet précédent sur le sujet il y a 3 ans environ, lorsque j’étais chez un client et que nous avions eu une discussion intéressante sur ce type de hint placé dans les requêtes de Reporting. Cette fois, j’ai pu expérimenté chez un autre client, une problématique de verrouillage intéressante en utilisant ce même hint.

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

Dilemme entre les filtres dynamiques et les requêtes « kitchen sink »

mikedavem — Wed, 10 Feb 2016 18:51:02 +0000

Etre confronté aux filtres dynamiques est un scénario assez courant avec les applications de gestion ou les ERP. En effet, les utilisateurs voudraient avoir la flexibilité de filtrer and de trier leur données business comme ils veulent afin d’être le plus efficace possible. cibler et opérer rapidement sur les bonnes données est en phase avec les exigences de performances quotidiennes. Du moins c’est ce que j’ai pu noter chez différents clients.

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

Envie de faire un benchmark de votre stockage? Il est temps de passer à diskspd

mikedavem — Tue, 03 Nov 2015 13:21:51 +0000

Comme vous le savez certainement, SQLIO est officiellement dépréciée depuis quelques mois. Pour ma part, j’ai eu l’occasion (peut être la dernière) d’utiliser SQLIO pour un projet client dans le but d’effectuer un benchmark de son stockage et préparer une installation AlwaysOn et groupes de disponibilité. Il est maintenant temps de passer au prochain outil prévu à cet effet: DiskSpd.

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

Jointure, transitivité et simplification de requête

mikedavem — Mon, 02 Nov 2015 13:55:18 +0000

Au cours d’une discussion hier avec un de mes amis « oraclien » nous avons débattu sur les différents comportements d’optimisation entre Oracle et SQL Server sur un cas bien précis. Vous pouvez lire son billet ici et trouver un script pour reproduire le problème par vous même. J’ai donc décidé d’écrire un billet à mon tour parce que cette discussion avait introduit des concepts intéressants que je voulais approfondir côté SQL Server.

> Pour lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

sp_cursor_fetch et performance

mikedavem — Mon, 02 Nov 2015 13:39:09 +0000

Il y a quelques semaines lors d’un audit, mon client me parlait de problèmes de performances identifiés uniquement sur la phase de login de son application. Après des échanges divers entre le client et l’éditeur de logiciel, nous avons constaté que le problème ne se produisait pas lorsque l’application et l’instance SQL Server étaient installées sur le même serveur (moins d’une seconde avec la configuration de l’éditeur contre 10 secondes avec celle du client). A vrai dire, mon client ne possédait pas tout à fait la même configuration qui comprenait un serveur applicatif et un serveur de bases de données distant. Imaginez la déception du client lorsqu’il s’est aperçu que sa configuration matérielle était de loin plus puissante que celle de l’éditeur de logiciel pour ces tests avec un même volume données.

> Pour lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server