A couple of weeks ago, we studied opportunities to replace existing clustered indexes (CI) with columnstore indexes (CCI) for some facts. To cut the story short and to focus on the right topic of this write-up, we prepared a creation script for specific CCIs based on the Niko’s technique variation (no MAXDOP = 1 meaning we enable parallelism) in order to get a better segment alignment.

Before deploying those indexes in our SQL DB Azure environment, we staged a first scenario in on-premises instance and the creation of all indexes took ~ 1h. It is worth noting that our tests are based on the same database with the same data in all cases. But guess what, the story was different in Azure and I got feedbacks from another team who was responsible to deploy indexes in Azure, the creation script was a bit longer (~ 4h).
I definitely enjoyed this story because we got a deeper understanding of DB Azure performance topic.

=> Moving to the cloud means we’ll get slower performance?

Before drawing conclusions to quickly a good habit to get is to compare specifications between environments. It’s not about comparing oranges and apples. Well let’s set my own context: from one side, the on-premises virtual SQL Server environment specification includes 8vCPUs (Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz), 64 GB of RAM and a high-performance storage array with micro latency device dedicated to our IO intensive workloads. From the vendor specifications, we may except very interesting IO performance with a general throughput greater than 100 KIOPs (Random) or 1GB/s (sequential). On another side, the SQL DB Azure is based on the service pricing tier General Purpose: Serverless Gen5, 8 vCores. We use the vCore purchasing model and referring to the Microsoft documentation, hardware generation 5 includes a compute specification based on Intel E5-2673 v4 (Broadwell) 2.3-GHz and Intel SP-8160 (Skylake) processors. Added to this, the service pricing tier comes with a remote SSD based storage including IO latency around 5-7ms and 2560 IOPs max. Given the opportunity of the infrastructure elasticity, we could scale to up 16 vCores, 48GB of RAM and 5120 IOPs for data. Obviously, latency remains the same in this case.

As illustration, creation of all indexes (CI + CCI + NCIs) performed in our on-premises environment gave the following storage performance figures: ~ 700MB/s and 13K IOPs for maximum values that were an aggregation of DATA + LOG activity on D: drive. Rebuilding indexes are high resource consuming operations in terms of CPU as well and we obviously noticed CPU saturation at different steps of the operation.

…

As an aside, we may notice the creation of CCI is a less intensive operation in terms of resources and we retrieve the same pattern in Azure below. Talking of which, let’s compare with our SQL Azure DB. There are different ways to get performance metrics including the portal which enables monitoring performance through easy-to-use interface or DMVs for each Azure DB like sys.dm_db_resource_stats. It is worth noting that in SQL Azure DB metrics are expressed as percentage of the service tier limit, so you need to adjust your analysis with the tier you’re using. First, we observed the same resource utilization pattern for all steps of the creation script but within a different timeline – duration has increased to 4h (as mentioned by another team). There is a clear picture of reaching the limit of the configured service tier, especially for Log IO (green line) and we already switched from GP_S_Gen5_8 to GP_S_Gen5_16 service tier

In addition, Wait stats gave interesting insights as well:

Excluding the traditional PAGEIOLATCH_xx waits, the LOG_RATE_GOVERNOR wait type appeared in the top waits and confirms that we bumped into the limits imposed on transaction log I/O by our performance tier.

=> Scaling vs Upgrading the Service for better performance?

With SQL DB Azure PaaS, we may benefit from elastic architecture. Firstly, scaling the number of CPUs is a factor of improvement and there is a direct relationship with storage (IOPs), memory or disk space allocated for tempdb for instance. But the order of magnitude varies with the service tier as shown below:

For General Purpose ServerLess Generation 5 service tier – Resources per Core

Something relevant here because even performance increases with the number of vCores provisioned, we can deduce Log IO saturation from our test in Azure (especially in the first step of the CI creation) results of max log rate limitation that doesn’t scale in the same way. This is especially relevant here because as said previously index creation can be an resource intensive operation with a huge impact on the transaction log.

What would be a solution to speed-up this operation?

First viable solution in our context would be to switch to SIMPLE recovery model that fits perfectly with our scenario because we could get minimally-logged capabilities and a lower impact on the transaction log and because it is suitable for DW environments. Unfortunately, at the moment of this write-up, this is not supported and I suggest you to vote on feedback Azure if you are interested in.
From an infrastructure standpoint, improving max log rate throughput is only possible by upgrading to a higher service tier (but at the cost of higher fees obviously). For a sake of curiosity, I did a try with the BC_Gen5_16 service tier specifications:

Even if this new service tier seems to be a better fit (suggested by the relative percentage of resource usage) …

… there are important notes here:

1) Business Critical Tier is not available for Serverless architecture

2) Moving to a different service is not instantaneous and it may require several hours according to the database size (~ 3h for a total size of ~500GB database size in my case). Well, this is not viable option even if get better performance. Indeed, if we add the time to upgrade to a higher service tier (3h) + time to run the creation script (3h or 25% of performance gain compared to the previous GP_S_Gen5_16 service tier). We may obviously upgrade again to reach performance closer to our on-premises environment but does it worth fighting for here only for an index creation script?

Concerning our scenario (Data Warehouse), it is generally easy to schedule a non-peak hours time frame that doesn’t overlap with the processing-oriented workload but it could not be the case for everyone!

See you!

In this blog post I would like to focus on the authentication architecture design to connect to a SQL Azure DB. I already played with some SQL Azure DB with a simple authentication protocol that consisted in using SQL Logins and opening some firewall ports to expose the DB service on the internet. I guess this is a basic scenario for lot of people (including me) who want to play with such service. But what about enterprise-class scenarios? I had the chance to get involved to the implementation of the end-to-end Integrated Windows Authentication (IWA) between SSRS data sources on-premises and one of SQL Azure DB. SQL Login is likely the most common method used to connect for its simplicity, but like on-premises scenarios, it is not the best one in terms of security.

So where to start? First of all, let’s say that Microsoft provides some configuration clues in the BOL (Azure SQL Database and AAD section).

From an architecture standpoint we must meet the following prerequisites:

Sounds complicated right? It could be regarding your context and which components you have access.

Let’s show a very simplified chart of the authentication flow:

Usually shops which own a hybrid environment with Azure already implemented the federation which was the case for me. So as DBA you will likely delegate some Active Directory stuff to right team.
Let’s start with the easiest parts of this big cake: Installing ADALSQL library on SSRS servers is pretty straightforward so need to talk further about it. Once the library is installed, it gives access to a new Microsoft Azure SQL Database connection type in SSRS data source as show below:

=> Kerberos delegation

If you are already confident with Kerberos delegation with your SQL Server environment, configuring Kerberos delegation on ADFS didn’t raise special difficulties. You must still configure correctly the SPNs and then configure the Kerberos delegation. In this case, The SSRS service must be configured to delegate the authentication to the ADFS service.

=> AAD authentication with SQL DB Azure

AAD authentication must be enabled on the SQL Azure DB before using IWA. As you know, SQL DB Azure provides different ways to connect including:

Well, you may notice that enabling AAD authentication is not trivial So let’s dig further to the AAD authentication method. It provides 2 non-interactive ways Active Directory – Password and Active Directory – Integrated authentication that are suitable for many applications based on ADO.NET, JDCB, ODC used by SSRS data-sources. This is because these methods never result in pop-up dialog boxes which can be used. Other method includes interactive method like Active Directory – Universal with MFA suitable for administrative accounts (including DBAs) for obvious security reasons. These methods are illustrated below:

After authentication, authorization comes into play. As DBA we must be aware of the different security model implied by this Azure service. In addition, regarding the login / user type, the access path will be different accordingly as follows:

Compared to on-premises SQL databases, there are only 2 unrestricted accounts including a Server Admin (SQL based Login) and an AAD admin created in master. These accounts are automatically mapped to the dbo user of each SQL Azure DB and are implicitly DB owner. Even these accounts are considered unrestricted, in fact it is not because they are not member of sysadmin server role (that is not available in SQL DB Azure by the way). For more details the BOL is your friend

Here a PowerShell script sample that helps identifying each of them:

Other administrative roles are dbmanager and loginmanager which are respectively able to create Azure databases and to create logins. Members of these roles must be created in the master database.
Finally, non-administrative users don’t need to access to the master database and may be SQL or AAD contained authentication-based users (making your database portable)
Creation of AAD login / user requires using the clause FROM EXTERNAL PROVIDER in the CREATE USER TSQL command. Like on-premises environments, you can rely on AAD group for your security strategy.
The sys.database_principals DMV is still available to get a picture of different users in each database:

…

Username values are not important here Note the type and authentication_type column values. In my context, we have a mix of AAD groups and users (EXTERNAL_USER/GROUP). Users with INSTANCE authentication type are server-level users with a correspond entry in master database. In the first line, the user is part of the dbmanager role.

…

Referring to this quick overview of SQL DB Azure authentication and permissions, in the context of my project we only need to configure a dedicated AD account to get access the data in Azure. Because the security standard is group-based we only need to move this user to the corresponding group that is member of db_datareader role on the SQL DB Azure side.

For DBAs the previous topics are familiar. However, the biggest part of the cake: the underlying AD and federation infrastructure is likely less. For modern DBAs in the cloud, authentication and connectivity are part of their new skills to gain. I don’t pretend to be an expert on this topic so the idea here is to share my experience I had on an issue we faced during the IWA implementation and drove me to improve my skills on the different components of the federation infrastructure including the ADFS server, the AD Azure connect component and the Azure Active Directory. After configuring the different components of the authentication infrastructure, we tried to configure the dedicated SSRS service account with the new SSRS data source and here the message we got when attempting to connect to the SQL Azure DB:

Could not discover a user realm. (System.Data)

Before continuing, let’s precise that the SSRS service account name (or User Principal Name) we configured on the AD on-premises side was in the form of domain.local\ssrs_account. After the AD Azure sync comes into play, on the AAD side the correspond AAD identity was ssrs_account@domain.onmicrosoft.com

At this stage, I didn’t pay attention of this UPN and I naively believed that creating the user referring this identity on the SQL DB Azure will be the final step of the authentication architecture implementation:

But as you guess, it was not the case and the authentication process failed. I spent some times to figure out what could be wrong.I admit my first DBA’s habit led me to try to look at an eventual SQL Server error log, but you know, we are dealing with a different beast and there is no SQL Server error log anymore to look at … In fact, to get a status of SQL Server connections SQL DB Azure provides auditing capabilities at different levels (server and db) with a default policy that includes FAILED_DATABASE_AUTHENTICATION_GROUP action to monitor successful and failed logins. In addition, consuming these events depends on the target storage (Storage Account, Logs Analytics). For instance, here a sample of results I got from our dedicated Log Analytics workspace with a search from the SQLSecurityAuditEvents category:

But nothing relevant and related to my SSRS service account … After reading the BOL more carefully, I finally figured out that if I want to get a chance to find (maybe) something related to the AAD identity I had to look at the AAD side instead as stated to the follow section of the documentation:

When using AAD Authentication, failed logins records will not appear in the SQL audit log. To view failed login audit records, you need to visit the Azure Active Directory portal, which logs details of these events.

But again, the AAD audit trail didn’t contain any relevant records of my issue. So where to look at in this case? Well, I decided to review the authentication process in the federation part. In the error message it was about a missing realm … realm names come from the Kerberos authentication protocol and they serve practically the same purpose as domains and domain names. In my context because we are in a hybrid authentication infrastructure, in order to make the authentication process working correctly a federation must be configured and it was effective in my case. As said previously, the federation part includes an ADFS server (provides SSO capabilities and IWA authentication for applications), an AD Sync component (To sync identities between AD and AAD) and a configured AAD as shown below:

A basic configuration we may retrieve in many shops. However, the remarkable thing was the different form of the SSRS realm between AD on-premises and AAD after the synchronization stuff of the AD Azure connect component. Getting support of my team, it became quickly obvious the root cause was on the federation infrastructure side. In normal case, with verified users we normally get the same UPN on the both side as shown below:

domain.com\user (AD on-premises) => domain.com\user (AAD)

Something important to bear in mind is that the Azure AD Connect tool comes with a troubleshooting section that was helpful in my case because it helped to validate our suspicion:

…

The tool diagnosed there was a mismatch between the userPrincipalName attribute value between the AD on-premises and the AAD because the UPN suffix was not verified by the AAD Tenant.
Using the Get-AzureDDomain cmdlet confirmed the only domain name: domain.com that was federated as shown below:

As a reminder, the SSRS account was created with an UPN domain suffix (domain.local) which was not verified by the AAD (domain.local is different from domain.com). As stated by the troubleshooting section when the UPN suffix is not verified with the AAD Tenant, Azure AD takes different inputs into account in the given order to calculate the UPN prefix in the cloud resulting to create the wrong one in my case (ssrs_account@domain.onmicrosoft.com).

Finally changing the correct UPN suffix to domain.com fixed this annoying issue!

See you!