If you deal with Cloud services and infrastructure, FinOps is a discipline you should get into for keeping under control your costs and getting actionable insights that could result in efficient cloud costs. Azure cost management provides visibility and control. Azure cost analysis is my favorite tool when I want to figure out costs of the different services and to visualize improvements after applying quick wins, architecture upgrades on the environment. It is also a good place to identify stale resources to cleanup. I will focus on Azure SQL DB here. From a cost perspective, Azure SQL DB service includes different meter subcategories regarding the options and the service tier you will use. You may have to pay for the compute, the dedicated storage for your database and for your backups (pitr or ltr) and so on … Cost Analysis allows drill-down analysis through different axis with aggregation or forecast capabilities.

In our context, we would like to know if moving from Azure SQL DB Azure Serverless compute tier (Pay-As-You-Go) to Provisioned Tier (+ Azure Hybrid Benefit + Reserved Instances for 3 years) has some good effects on costs. First look at the cost analysis section by applying correct filters and data aggregation on compute tier, confirmed our initial assumption that Serverless didn’t fit anymore with our context now. The chart uses a monthly-based timeframe daily aggregation. We switched to a different model mi-April as show below:

Real numbers are confidential but not so important here. We can easily notice a drop a daily cost (~ 0.5) between Serverless and Provisioned compute tier.

If we get a higher-level view of all services and costs for previous months, the trend is also confirmed for April with serverless + provisioned tier combined costs lower than serverless computer tier only for previous months. But we need to wait for next months to confirm the trend.

At the same time (and this is the focus on this write-up), we detected a sudden increase of backup storage cost in March that may ruin our optimization efforts made for compute, right? :). To explain this new trend, log analytics came to the rescue. As explained in the previous blog post, we configured streaming of Azure SQL DB telemetry into Log Analytics target to get benefit from solutions like SQL Insights and custom queries from different Azure logs.

Basic metrics are part of Azure SQL DB telemetry and stored in AzureMetrics table. We can use Kusto query to extract backup metrics and get an idea of different backup type trends over the time including FULL, DIFF and LOG backups. The following query shows backup trends within the same timeframe used for billing in cost management (February to May). It also includes a series_file_line function to draw a trendline in the time chart.

Full backup time chart

FULL backup size is relatively steady and cannot explain the sudden increase of storage backup cost in our case.

DIFF and LOG backup time chart

…

LOG and DIFF backup charts are more relevant and the trendline suggests a noticeable change starting mi-March. For the first part of the month, the trendline starts misaligning with backup size series.

At this stage, we found out the cause of the cost increase, but we were interested in understanding the reasons that may explain such trend. After investigating our ITSM system, we were able to find a correlation with the deployment of new maintenance tool – Ola Hallengren maintenance solution + custom scripts to rebuild columnstore indexes. The latter rebuilds aggressively 2 big fact tables with CCI in our DW (unlike the former tool) that explain the increase of DIFF and LOG backup sizes (~ 1TB).

This is where the collaboration with the data engineering team is starting to find an efficient and durable way to minimize the impact of the maintenance:

– Reviewing the custom script threshold may result to a more relax detection of fragmented columnstore indexes. However, this is only a piece of the solution because when a columnstore index becomes a good candidate for the next maintenance operation, it will lead to a resource-intensive and time-consuming operation (> 2.5h dedicated for these two tables). We are using Azure automation jobs with fair share to execute the maintenance and we are limited to 3h max per job execution. We may use a divide and conquer strategy to fit within the permitted execution timeframe, but it would lead to more complexity and we want to keep maintenance as simple as possible.

– We need to find another way to keep index and stat maintenance jobs execute time under a certain control. Introducing partition for these tables is probably a good catch and another piece of the solution. Indeed, currently concerned tables are not partitioned, and we could get benefit from partition-level maintenance for both indexes and statistics at the partition level.

Bottom line

Azure cost management center and log analytics are a powerful recipe in the FinOps practice. Kusto SQL language is a flexible tool for finding and correlate all kinds of log entries and events assuming you configured telemetry to the right target. I definitely like annotation-like system as we are using with Grafana because it makes correlation with external changes and workflows easier. Next step: investigate annotations on metric charts in Application insights?

See you!!

On-prem, DBAs often rely on well-established products but with Cloud-based architectures, often implemented through DevOps projects and developers, monitoring should be been redefined and include some new topics as:

1) Cloud service usage and fees observability
2) Metrics and events detection that could affect bottom line
3) Implementing a single platform to report all data that comes from different sources
4) Trigger rules with data if workload reaches over or drops below certain levels or when an event is enough relevant to not meet the configuration standard and implies unwanted extra billing or when it compromises the company security rules.
5) Monitoring of the user experience

A key benefit often discussed about Cloud computing, and mainly driven by DevOps, is how it enables agility. One of the meaning of term agility is tied to the rapid provisioning of computer resources (in seconds or minutes) and this shortening provisioning path enables work to start quickly. You may be tempted to grant some provisioning permissions to DEV teams and from my opinion this is not a bad thing, but it may come with some drawbacks if not under control by Ops team including database area. Indeed, for example I have in mind some real cases including architecture configuration drift, security breaches created by unwanted item changes, or idle orphan resources for which you keep being charged. All of these scenarios may lead either to security issues or extra billing and I believe it is important to get clear visibility of such events.

In my company, Azure built-in capabilities with Azure Monitor architecture are our first target (at least in a first stage) and seem to address the aforementioned topics. To set the context, we already relied on Azure Monitor infrastructure for different things including Query Performance Insight, SQL Audit analysis through Log Analytics and Azure alerts for some performance metrics. Therefore, it was the obvious way to go further by adding activity log events to the story.

In this blog post, let’s focus on the items 2) 4). I would like to share some experimentations and thoughts about them. As a reminder, items 2) 4) are about catching relevant events to help identifying configuration and security drifts and performing actions accordingly. In addition, as many event-based architectures, additional events may appear or evolve over the time and we started thinking about the concept with the following basic diagram …

… that led to the creation of the two following workflows:
– Workflow 1: To get notified immediately for critical events that may compromise security or lead immediately to important extra billing
– Workflow 2: To get a report of other misconfigured items (including critical ones) on schedule basis but don’t require quick responsiveness of Ops team.

Concerning the first workflow, using alerts on activity logs, action groups and webhooks as input of an Azure automation runbook appeared to be a good solution. On another side, the second one only requires running an Azure automation workbook on schedule basis. In fact, this is the same runbook but with different input parameters according to the targeted environment (e.g. PROD / ACC / INT). In addition, the runbook should be able to identity unmanaged events and notified Ops team who will decide either to skip it or to integrate it to runbook processing.

Azure alerts which can be divided in different categories including metric, log alerts and activity log alerts. The last one drew our attention because it allows getting notified for operation of specific resources by email or by generating JSON schema reusable from Azure Automation runbook. Focusing on the latter, we had come up (I believe) with what we thought was a reasonable solution.

Here the high-level picture of the architecture we have implemented:

1- During the creation of an Azure SQL Server or a database, corresponding alerts are added with Administrative category with a specific scope. Let’s precise that concerned operations must be registered with Azure Resource Manager in order to be used in Activity Log and fortunately they are all including in the Microsoft.Sql resource provider in this case.
2- When an event occurs on the targeted environment, an alert is triggered as well as the concerned runbook.
3- The execution of the same runbook but with different input parameters is scheduled on weekly basis to a general configuration report of our Azure SQL environments.
4- According the event, Ops team gets notified and acts (either to update misconfigured item, or to delete the unauthorized item, or to update runbook code on Git Repo to handle the new event and so on …)

The skeleton of the Azure automation runbook is pretty similar to the following one:

Some comments about the PowerShell script:

1) Input parameters should include either the Webhook data or specific parameter values for a complete Azure SQL DB check.
2) The first section should include your own functions to respond to different events. In our context, currently we drew on DBAChecks thinking to develop a derived model but why not using directly DBAChecks in a near future?
3) When an event is triggered, a JSON schema is generated and provides insight. The point here is you must navigate through different properties according to the operation type (cf. BOL).
4) The increase of events to manage could be a potential issue making the runbook fat especially if we keep both the core functions and event processing. To mitigate this topic, we are thinking to move functions into modules in Azure automation (next step).

Bottom line

Thanks to Azure built-in capabilities we improved our visibility of events that occur on the Azure SQL environment (both expected and unexcepted) and we’re now able to act accordingly. But I should tell you that going this way is not a free lunch and we achieved a reasonable solution after some programming and testing efforts. If you can invest time, it is probably the kind of solution you can add to your study.

See you

In this blog post I would like to focus on the authentication architecture design to connect to a SQL Azure DB. I already played with some SQL Azure DB with a simple authentication protocol that consisted in using SQL Logins and opening some firewall ports to expose the DB service on the internet. I guess this is a basic scenario for lot of people (including me) who want to play with such service. But what about enterprise-class scenarios? I had the chance to get involved to the implementation of the end-to-end Integrated Windows Authentication (IWA) between SSRS data sources on-premises and one of SQL Azure DB. SQL Login is likely the most common method used to connect for its simplicity, but like on-premises scenarios, it is not the best one in terms of security.

So where to start? First of all, let’s say that Microsoft provides some configuration clues in the BOL (Azure SQL Database and AAD section).

From an architecture standpoint we must meet the following prerequisites:

Sounds complicated right? It could be regarding your context and which components you have access.

Let’s show a very simplified chart of the authentication flow:

Usually shops which own a hybrid environment with Azure already implemented the federation which was the case for me. So as DBA you will likely delegate some Active Directory stuff to right team.
Let’s start with the easiest parts of this big cake: Installing ADALSQL library on SSRS servers is pretty straightforward so need to talk further about it. Once the library is installed, it gives access to a new Microsoft Azure SQL Database connection type in SSRS data source as show below:

=> Kerberos delegation

If you are already confident with Kerberos delegation with your SQL Server environment, configuring Kerberos delegation on ADFS didn’t raise special difficulties. You must still configure correctly the SPNs and then configure the Kerberos delegation. In this case, The SSRS service must be configured to delegate the authentication to the ADFS service.

=> AAD authentication with SQL DB Azure

AAD authentication must be enabled on the SQL Azure DB before using IWA. As you know, SQL DB Azure provides different ways to connect including:

Well, you may notice that enabling AAD authentication is not trivial So let’s dig further to the AAD authentication method. It provides 2 non-interactive ways Active Directory – Password and Active Directory – Integrated authentication that are suitable for many applications based on ADO.NET, JDCB, ODC used by SSRS data-sources. This is because these methods never result in pop-up dialog boxes which can be used. Other method includes interactive method like Active Directory – Universal with MFA suitable for administrative accounts (including DBAs) for obvious security reasons. These methods are illustrated below:

After authentication, authorization comes into play. As DBA we must be aware of the different security model implied by this Azure service. In addition, regarding the login / user type, the access path will be different accordingly as follows:

Compared to on-premises SQL databases, there are only 2 unrestricted accounts including a Server Admin (SQL based Login) and an AAD admin created in master. These accounts are automatically mapped to the dbo user of each SQL Azure DB and are implicitly DB owner. Even these accounts are considered unrestricted, in fact it is not because they are not member of sysadmin server role (that is not available in SQL DB Azure by the way). For more details the BOL is your friend

Here a PowerShell script sample that helps identifying each of them:

Other administrative roles are dbmanager and loginmanager which are respectively able to create Azure databases and to create logins. Members of these roles must be created in the master database.
Finally, non-administrative users don’t need to access to the master database and may be SQL or AAD contained authentication-based users (making your database portable)
Creation of AAD login / user requires using the clause FROM EXTERNAL PROVIDER in the CREATE USER TSQL command. Like on-premises environments, you can rely on AAD group for your security strategy.
The sys.database_principals DMV is still available to get a picture of different users in each database:

…

Username values are not important here Note the type and authentication_type column values. In my context, we have a mix of AAD groups and users (EXTERNAL_USER/GROUP). Users with INSTANCE authentication type are server-level users with a correspond entry in master database. In the first line, the user is part of the dbmanager role.

…

Referring to this quick overview of SQL DB Azure authentication and permissions, in the context of my project we only need to configure a dedicated AD account to get access the data in Azure. Because the security standard is group-based we only need to move this user to the corresponding group that is member of db_datareader role on the SQL DB Azure side.

For DBAs the previous topics are familiar. However, the biggest part of the cake: the underlying AD and federation infrastructure is likely less. For modern DBAs in the cloud, authentication and connectivity are part of their new skills to gain. I don’t pretend to be an expert on this topic so the idea here is to share my experience I had on an issue we faced during the IWA implementation and drove me to improve my skills on the different components of the federation infrastructure including the ADFS server, the AD Azure connect component and the Azure Active Directory. After configuring the different components of the authentication infrastructure, we tried to configure the dedicated SSRS service account with the new SSRS data source and here the message we got when attempting to connect to the SQL Azure DB:

Could not discover a user realm. (System.Data)

Before continuing, let’s precise that the SSRS service account name (or User Principal Name) we configured on the AD on-premises side was in the form of domain.local\ssrs_account. After the AD Azure sync comes into play, on the AAD side the correspond AAD identity was ssrs_account@domain.onmicrosoft.com

At this stage, I didn’t pay attention of this UPN and I naively believed that creating the user referring this identity on the SQL DB Azure will be the final step of the authentication architecture implementation:

But as you guess, it was not the case and the authentication process failed. I spent some times to figure out what could be wrong.I admit my first DBA’s habit led me to try to look at an eventual SQL Server error log, but you know, we are dealing with a different beast and there is no SQL Server error log anymore to look at … In fact, to get a status of SQL Server connections SQL DB Azure provides auditing capabilities at different levels (server and db) with a default policy that includes FAILED_DATABASE_AUTHENTICATION_GROUP action to monitor successful and failed logins. In addition, consuming these events depends on the target storage (Storage Account, Logs Analytics). For instance, here a sample of results I got from our dedicated Log Analytics workspace with a search from the SQLSecurityAuditEvents category:

But nothing relevant and related to my SSRS service account … After reading the BOL more carefully, I finally figured out that if I want to get a chance to find (maybe) something related to the AAD identity I had to look at the AAD side instead as stated to the follow section of the documentation:

When using AAD Authentication, failed logins records will not appear in the SQL audit log. To view failed login audit records, you need to visit the Azure Active Directory portal, which logs details of these events.

But again, the AAD audit trail didn’t contain any relevant records of my issue. So where to look at in this case? Well, I decided to review the authentication process in the federation part. In the error message it was about a missing realm … realm names come from the Kerberos authentication protocol and they serve practically the same purpose as domains and domain names. In my context because we are in a hybrid authentication infrastructure, in order to make the authentication process working correctly a federation must be configured and it was effective in my case. As said previously, the federation part includes an ADFS server (provides SSO capabilities and IWA authentication for applications), an AD Sync component (To sync identities between AD and AAD) and a configured AAD as shown below:

A basic configuration we may retrieve in many shops. However, the remarkable thing was the different form of the SSRS realm between AD on-premises and AAD after the synchronization stuff of the AD Azure connect component. Getting support of my team, it became quickly obvious the root cause was on the federation infrastructure side. In normal case, with verified users we normally get the same UPN on the both side as shown below:

domain.com\user (AD on-premises) => domain.com\user (AAD)

Something important to bear in mind is that the Azure AD Connect tool comes with a troubleshooting section that was helpful in my case because it helped to validate our suspicion:

…

The tool diagnosed there was a mismatch between the userPrincipalName attribute value between the AD on-premises and the AAD because the UPN suffix was not verified by the AAD Tenant.
Using the Get-AzureDDomain cmdlet confirmed the only domain name: domain.com that was federated as shown below:

As a reminder, the SSRS account was created with an UPN domain suffix (domain.local) which was not verified by the AAD (domain.local is different from domain.com). As stated by the troubleshooting section when the UPN suffix is not verified with the AAD Tenant, Azure AD takes different inputs into account in the given order to calculate the UPN prefix in the cloud resulting to create the wrong one in my case (ssrs_account@domain.onmicrosoft.com).

Finally changing the correct UPN suffix to domain.com fixed this annoying issue!

See you!

Mais avant de parler de la prochaine version de SQL Server, attardons nous à la version courante – SQL Server 2016 – qui offre d’ores et déjà des améliorations et nouvelles possibilités intéressantes dans plusieurs domaines qu’il s’agisse de la performance, la haute disponibilité, la sécurité et bien d’autres. Par ailleurs, une autre nouvelle importante qui intéressera la plupart de nos clients est la sortie récente du SP1 de SQL Server 2016 et qui permet une homogénéisation de la surface des fonctionnalités entre les différentes éditions. Il sera notamment possible d’utiliser la compression, le partitionnement ou les index columnstore avec une édition standard par exemple. Je ne suis pas devin mais je pense ne pas me tromper en disant que cette année je verrais pousser quelques infrastructures 2016 dans les écosystèmes de nos clients!

En tout cas, pour commencer du bon pied avec cette nouvelle version de SQL Server, un événement francophone à inscrire absolument dans vos agendas est le MS Cloud Summit qui se déroulera à Paris à partir 23 janvier prochain. Cet événement se veut beaucoup plus large que les journées SQL Server que vous connaissez certainement déjà. Le MS Cloud Summit, c’est 600 participants attendus, 7 tracks avec 72 sessions autour du cloud, des scénarios hybrides et on-premises.  Pour ma part, je serai présent avec dbi services et j’aurai le plaisir de vous présenter les nouveautés dont vous bénéficierez en termes de haute disponibilité et plan de récupération avec le couple gagnant Windows 2016 et SQL Server 2016.

Au plaisir de vous y retrouver. D’ici là je vous souhaite une très bonne année 2017 !

David Barbarin
MVP & MCM SQL Server