As a reminder, this Grafana dashboard gets information from Prometheus server and metrics related to MSSQL environments. For a sake of clarity, in this dashboard, environment defines one availability group and a set of 2 AG replicas (A or B) in synchronous replication mode. In other words, ENV1 value corresponds to availability group name and to SQL instance names member of the AG group with SERVERA\ENV1 (first replica), SERVERB\ENV1 (second replica).

In the picture above, you can notice 2 sections. One is for availability group and health monitoring and the second includes a set of black box metrics related to saturation and latencies (CPU, RAM, Network, AG replication delay, SQL Buffer Pool, blocked processes …). Good job for one single environment but what if I want to introduce more availability groups and SQL instances in the game?

The first and easiest (or naïve) way we went through when we started writing this dashboard was to copy / paste all the stuff done for one environment the panels as shown below:

After creating a new row (can be associated to section in the present context) at the bottom, all panels were copied from ENV1 to the new fresh section ENV2. New row is created by converting anew panel into row as show below:

Then I need to modify manually ALL the new metrics with the new environment. Let’s illustrate the point with Batch Requests/sec metric as example. The corresponding Prometheus query for the first replica (A) is: (the initial query has been simplified for the purpose of this blog post):

Same query exists for secondary replica (B) but with a different label value:

SERVERA:ENV1 and SERVERB:ENV1 are static values that correspond to the name of each SQL Server instance – respectively SERVERA\ENV1 and SERVERB\ENV1. As you probably already guessed and according to our naming convention, for the new environment and related panels, we obviously changed initial values ENV1 with new one ENV2. But having more environments or providing filtering capabilities to focus only on specific environments make the current process tedious and we need introduce dynamic stuff in the game … Good news, Grafana provides such capabilities with dynamic creation of rows and panels. and rows.

Generating dynamic panels in the same section (row)

Referring to the dashboard, first section concerns availability group health metric. When adding a new environment – meaning a new availability group – we want a new dedicated panel creating automatically in the same section (AG health).
Firstly, we need to add a multi-value variable in the dashboard. Values can be static or dynamic from another query regarding your context. (up to you to choose the right solution according to your context).

Once created, a drop-down list appears at the upper left in the dashboard and now we can perform multi selections or we can filter to specific ones.

Then we need to make panel in the AG Heath section dynamic as follows:
– Change the title value with corresponding dashboard (optional)
– Configure repeat options values with the variable (mandatory). You can also define max panel per row

According to this setup, we can display 4 panels (or availability groups) max per row. The 5th will be created and placed to a new line in the same section as shown below:

Finally, we must replace static label values defined in the query by the variable counterpart. For the availability group we are using sqlserver_hadr_replica_states_replica_synchronization_health metric as follows (again, I voluntary put a sample of the entire query for simplicity purpose):

You can notice the regex expression used to get information from SQL Instances either from primary (A) or secondary (B). The most interesting part concerns the environment that is now dynamic with $ENV variable.

Generating dynamic sections (rows)

As said previously, sections are in fact rows in the Grafana dashboard and rows can contain panels. If we add new environment, we want also to see a new section (and panels) related to it. Configuring dynamic rows is pretty similar to panels. We only need to change the “Repeat for section” with the environment variable as follows (Title remains optional):

As for AG Health panel, we also need to replace static label values in ALL panels with the new environment variable. Thus, referring to the previous Batch Requests / sec example, the updated Prometheus query will be as follows: (respectively for primary and secondary replicas):

…

The dashboard is now ready, and all dynamic kicks in when a new SQL Server instance is added to the list of monitored items. Here an example of outcome in our context:

Happy monitoring!

In my last blog post, I discussed about monitoring and how it should help to address quickly a situation that is going degrading. Alerts are probably the first way to raise your attention and, in my case, they are often in the form of emails in a dedicated folder. That remains a good thing, at least if you’re not focusing too long in other daily tasks or projects. In work office, I know I would probably better focus on new alerts but as I said previously, telework changed definitely the game.

I wanted to find a way to address this concern at least for main SQL Server critical alerts and I thought about relying on my existing home lab infrastructure to address the point. Reasons are it is always a good opportunity to learn something and to improve my skills by referring to a real case scenario.

My home lab infrastructure includes a cluster of Raspberry PI 4 nodes. Initially, I use it to improve my skills on K8s or to study some IOT stuff for instance. It is a good candidate for developing and deploying a new app for detecting new incoming alerts in my mailbox and sending notifications to my Lametric accordingly.

Lametric is a basically a connected clock but works also as a highly-visible display showing notifications from devices or apps via REST APIs. First time I saw such device in action was in a DevOps meetup in 2018 around Docker and Jenkins deployment with Eric Dusquenoy and Tim Izzo (@5ika_). In addition, one of my previous customers had also one in his office and we had some discussions about cool customization through Lametric apps.

Connection through VPN to my company network is mandatory to work from home and unfortunately Lametric device doesn’t support this scenario because communication is limited to local network only. So, I need an app that run on my local (home) network and able to connect to my mailbox, get new incoming emails and finally sending notifications to my Lametric device.

Here my setup:

There are plenty of good blog posts to create a Raspberry cluster on the internet and I would suggest to read that of Andrew Pruski (@dbafromthecold).

As shown above, there are different paths for SQL alerts referring our infrastructure (On-prem and Azure SQL databases) but all of them are send to a dedicated distribution list for DBA.

The app is a simple PowerShell script that relies on Exchange Webservices APIs for connecting to the mailbox and to get new mails. Sending notifications to my Lametric device is achieved by a simple REST API call with well-formatted body. Details can be found the Lametric documentation. As prerequisite, you need to create a notification app from Lametric Developer site as follows:

As said previously, I used PowerShell for this app. It can help to find documentation and tutorials when it comes Microsoft product. But if you are more confident with Python, APIs are also available in a dedicated package. But let’s precise that using PowerShell doesn’t necessarily mean using Windows-based container and instead I relied on Linux-based image with PowerShell core for ARM architecture. Image is provided by Microsoft on Docker Hub. Finally, sensitive information like Lametric Token or mailbox credentials are stored in K8s secret for security reasons. My app project is available on my GitHub. Feel free to use it.

Here some results:

– After deploying my pod:

– The app is running and checking new incoming emails (kubectl logs command)

When email is detected, notification is sendig to Lametric device accordingly

Geek fun good (bad?) idea to start this new year 2021

All messages are sent from the initiators to a single central target. Windows authentication for endpoint’s connection and messages are encrypted before to be sent to the remote service by using certificates mapped to database users.

Installing such infrastructure is not trivial and during this year, we often had to deploy and reinitialize service broker configurations because we added new initiators, or some initiators were replaced by other ones. Therefore, on the DBA side we worked on an automatic way to interact with service broker configuration and it turns out that the natural way for us was to write a PowerShell module to address it.

A couple of days, for a sake of curiosity I implemented a service broker event based XE session and I noticed a bunch of broker_message_undeliverable messages with the reason: The message could not be delivered because it is a duplicate. See below:

What does it mean exactly? Well, according to my Google-fu and very-explained blog from Remus Rusanu, it seems that the messages are accepted by the target, but the acks don’t reach back to the sender and therefore the sender is retrying the message again and again. But in my case, the sender ended up managing sending back the acknowledgement.

First step was to identify a misconfigured route between the target and the corresponding initiator. From broker instance value corresponds to the service broker id of the initiator. But surprisingly I didn’t find any misconfigured item. But I remembered the campaign of server rollout that had been going on for a few months where some servers were renamed following a new named convention. Furthermore, in our deployment process, dealing with new or renamed servers is exactly the same and we just initialize a new initiator and install a new route on the target regardless the scenario. But we forget one important point: because all routes are based on server name renaming a server leads to potentially to duplicate routes in this case. I verified my assumption by counting the number of existing routes on the target site:

The result was 148 and we expect to get 85 routes in the current architecture confirming we have duplicate items here. Next step consisted in identifying and fixing duplicate routes:

Here a sample of duplicate routes I founded. The renaming server kept its service broker id and I easily identified duplicates by grouping by the broker instance.

After fixing the issue, messages related to duplicates disappear from the XE output.
Hope this helps!

David Barbarin

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

Cette année a eu lieu la première édition francophone des 24h du PASS (les 20 et 21 septembre 2016). L’idée était plutôt simple: proposer une série de 24 webinaires gratuits de 10h jusqu’à 22h (heure française) pendant 2 jours. C’était l’occasion de recevoir et d’échanger les dernières informations autour de l’administration et du développement des bases de données, des nouvelles tendances côté Business Intelligence et du Cloud.

Pour ma part, c’est avec plaisir que j’ai eu l’occasion d’échanger avec vous autour de 2 sujets: tempdb et bonnes pratiques ainsi que des columnstore et leur implication dans les nouvelles tendances d’architecture BI avec Thoi Dung TSP Microsoft Switzerland.

C’est encore l’occasion de remercier les sponsors et Isabelle sans qui ce type d’événement n’aurait certainement pas eu lieu.

Les slides et démos devraient arriver sous peu!

David Barbarin
MVP & MCM SQL Server

Redémarrer un cluster à basculement dans un tel mode implique l’exécution de routines internes sur la base de données interne d’un cluster. Cette dernière utilise notamment un algorithme nommé Paxos pour garantir que les changements survenus sur un nœud soient répliqués de manière atomique sur le reste de membres de la topologie. Qu’est que Paxos exactement? Pour être honnête, j’avais déjà certains enregistrements dans le journal du cluster évoquant ce sujet et je n’avais jamais vraiment pris le temps de regarder plus précisément à quoi cela correspondait …

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server

> Lire la suite (en anglais)

David Barbarin
MVP & MCM SQL Server