Blog
How to create a shared access signature for a specific folder in Azure Storage

How to create a shared access signature for a specific folder in Azure Storage

As we move into a serverless wold where devlopment work is done less on generic multipurpose servers that require patching both to the OS and to the application installed on them, things become easier and in some cases cheaper.

For instance I am currently working on a project that needs to send and recieve XML files from different sources and with no other requirements for a server, Azure storage seemed the perfect fit. As it's just a storage service you pay for what you need and nothing more. All the patching and maintencance of API's is provided by Microsoft and we have an instant place to store out files. Ideal!

However new technology always comes with new challenges. My first issue came with the plan on how we would be getting files in and out with partners. This isn't the first time I've worked with Azure File Storage, but it is the first time I did it without an opps person setting things up. Previously we used FTP, which while dated works with a lot of applications. Like a headphone jack it's not the most glamorus of connection, but it works, everyone knows how it works and everyone has things that work with it. However it transpires that despite Azure storage being 10 years old and reciving requests for FTP from the start, Microsoft have decided to go the same route as Apple with the headphone jack and not have it. Instead the only option for an integration is REST. As it transpires the opps people I had worked with in the past when faced with this issue had just put a VM infront of it, which kind of defeats the point of using Azure storage in the first place!

So we're going with REST and Microsoft provide quite a straightforward REST API all good so far, but how do we limit access? Well there's a guide to Using the Azure Storage REST API which contains a section on creating an authorisation header. It's long and overly complex, but does point you in the direction that to do this you need a Shared Access Signature. The other option is an access key, but this is something you should never give away to a third party.

Shared Access Signature

After a bit more digging through the documentation (and just clicking the thing that sounded right in the potal) I found this documentation on creating an Account SAS which sounded like what I wanted (it wasn't, but it's close).

With a shared access signature you can say what kind of service should be allowed, what permissions they should have, IP address's, start and end dates. All awesome things.

Once I had this I could then use the REST API, but there was a problem. I could access every folder in the storage account and there was no way to stop this! For integrating with 2 third partys they would both be able to access each others stuff, and our own private stuff.

There is also no way to revoke the SAS once it's been generated other than refreshing the access keys which would affect everyone.

Folder Level Shared Access Signature

After a bit more research I found what I was looking for. How to create a shared access signature at a folder or item level and how to link it to a policy.

The first thing you need is Azure Storage Explorer. Once your set up with this you will be able to view all your storage accounts.

From here you are able to browse to the folder you want to share right click it and choose Manage Access Policies.

This will open a dialoge to manage the policies for this specific object rather than the account.

Here you can set all the same permissions as you could for a signature at an account level but now for a specific object and against a policy rather than an actual signature, meaning the policy can be updated in the future with no change to the signature.

Better still you can remove the policy which will then invalidate any signature using it.

For the actual signature key right click the same folder and click Get Shared Access Signature.

Then in the dialoge select the policy from the drop down rather than spcifying the individual permissions.

Click create and you can copy the keys.

You now have an access key that is limited to a specific folder rather than the entire account.

This is only possible to do though one of the code/scripting interfaces. e.g. PowerShell or the storage explorer. The azure portal will only let you get signatures at an account level.

Debugging Sitecore 9 Analytics Issues

Debugging Sitecore 9 Analytics Issues

With Sitecore 9 the way analytics is recorded and processed changed. Rather than everything being done by the IIS application, the architecture changed to include:

  1. A second IIS application called xconnect
  2. A windows services called Sitecore XConnect Search Indexer

As well as this Mongo was replaced by:

  1. SQL Server databases
  2. A SOLR XDB Core

Sitecore 9 also introduced data being secure in transit as well as at rest which means all traffic is encrypted using certificates.

While all these peices can be considered good, it does also create more points of failure that becomes harder to debug. So after spending a decent amount of time debugging why no analytic reports were loading and then why no data was appearing in the reports, I've made a checklist to go through.

Debugging Sitecore Analytics Checklist

1. XConnect Connection String

The main Sitecore application has connection strings for where it will find the XConnect service. They will look like this:

1 <add name="xconnect.collection" connectionString="https://mysite.xconnect" />
2  <add name="xconnect.collection.certificate" connectionString="StoreName=My;StoreLocation=LocalMachine;FindType=FindByThumbprint;FindValue=EF7F38B623E6664359110F2C6EB6DA00D567950F" />

One gives the path to the XConnect service and the other gives the path to find the certificate.

Firstly check that the XConnect path is correct and that you can access it and secondly check that the thumbprint corresponds with a certificate in the certificate store.

You can see what certificates are on your machine using this PowerShell script:

1Get-ChildItem -Path "cert:\LocalMachine\Root" | Format-Table Subject, FriendlyName, Thumbprint
2Get-ChildItem -Path "cert:\LocalMachine\My" | Format-Table Subject, FriendlyName, Thumbprint
3Get-ChildItem -Path "cert:\CurrentUser\My" | Format-Table Subject, FriendlyName, Thumbprint

2. Check certificate expiry date

Having a certificate is a good start, but it could still have expired.

Find the certificate in the certificate store by hitting start > manage computer certificates and find it in one of the folders.

Check the expiration date. If it's in the past then it's not going to work. This is common because the installation script for Sitecore 9 will set the expiration date to a year after install by default.

If it has expired you will need a new cert, you can create this by using the same script that you used to install sitcore origionally (Just the certificate bit).

1#Switch to correct vesion of SIF
2Remove-Module -Name SitecoreInstallFramework
3Import-Module -Name SitecoreInstallFramework -RequiredVersion 1.2.1
4

5#define parameters
6$prefix = "SitePrefix"
7$PSScriptRoot = "C:\resourcefiles9.0"
8$XConnectCollectionService = "$prefix.xconnect"
9$sitecoreSiteName = "$prefix.sc"
10

11#install client certificate for xconnect
12$certParams = @{
13 Path = "$PSScriptRoot\xconnect-createcert.json"
14 CertificateName = "$prefix.xconnect_client"
15 RootCertFileName = "SIF121Root"
16}

3. Check security permissons on the certificate

If you have a certificate and it's still valid then it could be that the app pool the site is running in doesn't have access to read the certificate.

To check this:

  1. Right click the certificate
  2. All Tasks
  3. Manage Private Keys
  4. Check that the app pool user for your site is listed in the list of users and that it has read permission. If it's not there add it using the name
    IIS APPPOOL\app pool name

4. Check License Files

Partner licenses only last for a year so if your using one of those it may have expired.

We're all used to checking the license file in the Sitecore application but XConnect has a license too.

These will be in:
sitename.xconnect\App_data\
sitename.xconnect\App_data\jobs\continuous\IndexWorker\App_data
sitename.xconnect\App_data\jobs\continuous\AutomationEngine\App_Data

Only the first will be used when your viewing the site, but it's worth knowing about the others to, incase you ever run a job manually.

5. Check manual rebuild of indexes

You can trigger a manual rebuild of the xDB index by following these instructions:

https://doc.sitecore.com/developers/90/sitecore-experience-platform/en/rebuild-the-xdb-index-in-solr.html

Remember in point 4 that it has it's own license file. It also has it's own connection strings.

6. Check XConnect site works in a browser

If you open XConnect in a browser you should recieve no certificate errors and a timestamp saying how long XConnect had been running for.

7. Check certificates are in the right store

This stack overflow post was a big help for me (https://stackoverflow.com/questions/26247462/http-error-403-16-client-certificate-trust-issue ). I was at the point where everything seemed right, but moving the certificates as shown here got it to the point of the analytics reports loading.

Windows 2012 introduced stricter certificate store validations. According to KB 2795828: Lync Server 2013 Front-End service cannot start in Windows Server 2012, the Trusted Root Certification Authorities (i.e. Root) store can only have certificates that are self-signed. If that store contains non-self-signed certificates, client certificate authentication under IIS returns with a 403.16 error code.

To solve the problem, you have to remove all non-self-signed certificates from the root store. This PowerShell command will identify non-self-signed certificates:

1Get-Childitem cert:\LocalMachine\root -Recurse | 
2    Where-Object {$_.Issuer -ne $_.Subject}

In my situation, we moved these non-self-signed certificates into the Intermediate Certification Authorities (i.e. CA) store: 

1Get-Childitem cert:\LocalMachine\root -Recurse | 
2    Where-Object {$_.Issuer -ne $_.Subject} | 
3    Move-Item -Destination Cert:\LocalMachine\CA

Checklist for data not going into Analytics

If you've got to the point of the analytics reports working, but not showing any data, this is my checklist for making sure data goes in. In my case I was trying to log site searches as per my article from a few years ago https://himynameistim.com/2017/09/13/populating-the-internal-search-report-in-sitecore/ there wern't any errors, but no data ever showed.

1. Enable Analytics Debugging

In Sitecore.Analytics.Tracking.config there is a setting to set the analytics logging level to debug. You will also need to set the log level on log4net root in Sitecore.config to debug.

2. Disable Robot Detection

In my case Sitecore thought I was a robot. Changing these settings will disable that:

1<setting name="Analytics.AutoDetectBots" set:value="false" />
2<setting name="Analytics.Robots.IgnoreRobots" set:value="false" />

3. Test analytics is tracking something

One of the hardest parts about analytics is it's not instant. The initial tracking only goes into the DB at the end of the users session and that's only for collection. It won't appear in the reports until processing has happened. So to speed this up:

Create a page called kill.aspx as follows. This will end the users session and trigger the data to be fed into the DB.

1<%@ Page language="c#" %>
2

3<!DOCTYPE html>
4<html>
5  <head>
6

7  </head>
8  <body>
9

10<div>Session Abandoned</div>
11<% Session.Abandon(); %>
12

13  </body>
14</html>

Next do something on the site that will cause some tracking to get added to the DB. In my case it was the search. Then go to kill.aspx to force session abondon.

Check the logs. You should see something like this...

125016 11:19:13 DEBUG [Analytics]: The CommitSession pipeline, ProcessSubscriptions is skipped - there is no subscriptions for location id: 4ebd0208-8328-5d69-8c44-ec50939c0967

Check the DB an entry should have gone into a shard db for [xdb_collection].[Interactions] table

To speed up processing, restart the main sitecore application.

Config file transforms with Azure Devops

Config file transforms with Azure Devops

For a long time now our primary CI setup has been based around Team City and Octopus deploy, but as reliable as it is there are things I don't like about it:

  1. It's not a SASS setup meaning there's a VM to occassionaly think about and updates to install. While Octopus is now availiable as a SASS option, Team City is not and moving Octopus will only solve half the problem.
  2. That VM they both sit on every so often gets and issue with it's hard disk being full.
  3. It's complicated to recommend the same setup to clients. You end up having to go through multiple things they need to buy which then require some installation and ongoing maintenance. Ideally we would have a setup thats easy for them to replicate and own themselves with minimal maintenance.

So when we took over a site recently that typically came with no existing CI setup in place, I decided to take a look at using Azure Devops instead. You can use Azure Devops with Octopus Deploy but as it claims to be able to manage releases as well as builds we went for doing the whole thing just in Azure Devops.

Getting a build set up was relatively straight forward so I'm going to skip past that bit, but in short we ended up with a build that will create a web deploy package and publish it as an artifact. Typical msbuild type stuff.

File transforms and variable substitution

The first real tricky point came with replacing variables in config files during a release to each envrionment. We were using the IIS Web App Deploy task to deploy the application to IIS on a VM (no new Azure Web App Services in this setup :( as I said we took over the site and this was just to get automated deploys of what they already have). A simple starting point with this is some built in functionality for XML Variable Substituion in the IIS Web App Deploy task.

Quite simply you can add all your varibles to the variable list, set the scope for which envrionment you want it to apply to and the during the deploy they are replaced in your config. Unlike some tag replacement tools I've used in the past this one actually uses the name of the connecting string or app setting you need to set, so if you need to set a connection string named web, the variable name will be web.

This is also where my problem stated. The description for what XML variable substitution does is:

This was a Sitecore solution and for Sitecore most of your config settings are in Sitecores own Sitecore section of the config file. So in other words the connection string will get updated but the rest won't.

Parameter and SetParameter XML files

My next issue was trying to find a solution is actually quite hard. Searching for this problem either gave me a lot of results for setups using ARM templates (as I said, this was a solution we took over and that kind of change is not on the agenda), or you just get the easy bit above. Searching for Sitecore and Azure Devops also leads you to a lot of results on a cloud infrastructure setup (again not what we're doing here, at least in the short term). Everything that was coming up felt far more complicated than the solution should be.

However the documentation on the XML variable substitution did have one interesting sentance.

A parameters.xml file isn't something I've used before which makes this sentance a bit cryptic. The first half says I can do what I want with an xml file, but the second half says I'll need something else to actually do it.

After a bit of research this all comes back to web deploy. When you do a build that outputs a web deploy package, you get 5 files.

A zip file containing the actual site, a command file which has the script to do the deploy and a set parameters file which is used to set config variables during the deploy. The others aren't so imporant.

To have different config set on different envrionments you just need to edit the set parameters file. But first you need to have the parameter in the set parameters file so that you can actually change it and this is where the parameters.xml file comes in.

Creating the parameter files

Add a file called parameters.xml file to the root of your project and then add parameters as follows.

1<?xml version="1.0" encoding="utf-8" ?>
2<parameters>
3  <parameter name="DataFolderLocation" defaultvalue="#{dataFolder}">
4    <parameterEntry kind="XmlFile" scope="App_Config\\Include\\Z.Project\\DataFolder\.config$" match="/configuration/sitecore/sc.variable[@name='dataFolder']/patch:attribute/text()" />
5  </parameter>
6</parameters>

Some important parts:

default value - The value that the config setting will get set to

scope - The path to the file containing the setting

match - An XPath expression for find the part of the config file to update

Once you have this the build will start producing a SetParameters.xml file containing the extra parameters.

1<?xml version="1.0" encoding="utf-8"?>
2<parameters>
3  <setParameter name="IIS Web Application Name" value="Default Web Site/SiteCore.Website_deploy" />
4  <setParameter name="DataFolderLocation" value="#{dataFolder}" />
5</parameters>

Note: I've set the value to be something I intend to replace in the release process.

Replacing the tokens

With our SetParameters.xml file now contining all the config we need to update, we need a step in the release process that will replace all the tokens with the correct values.

To do this I used a replaced tokens task https://marketplace.visualstudio.com/items?itemName=qetza.replacetokens

Config options need to be set for:

Root Directory - Path to the folder containing the SetParameters.xml file

Target files - A list of files to have replacements done in. In our case this was SiteCore.Website.SetParameters.xml

Token prefix - The prefix on tokens to be search for. Ours was #{

Token suffix - The suffix to denote the end of a token. Ours was }

Lastly in the IIS Web App Deploy step the SetParameters file needed to be selected and the new variables added to the variable list in Azure Devops. The variable names need to be called the bit between your prefix and suffix. i.e. #{datafolder} would be called datafolder.

If you don't set the variables then the log's will show warning for each one it couldn't find.

12019-09-24T17:17:21.6950466Z ##[section]Starting: Replace tokens in SiteCore.Website.SetParameters.xml
22019-09-24T17:17:23.9831695Z ==============================================================================
32019-09-24T17:17:23.9831783Z Task         : Replace Tokens
42019-09-24T17:17:23.9831816Z Description  : Replace tokens in files
52019-09-24T17:17:23.9831861Z Version      : 3.2.1
62019-09-24T17:17:23.9831891Z Author       : Guillaume Rouchon
72019-09-24T17:17:23.9831921Z Help         : v3.2.1 - [More Information](https://github.com/qetza/vsts-replacetokens-task#readme)
82019-09-24T17:17:23.9831952Z ==============================================================================
92019-09-24T17:17:27.2703037Z replacing tokens in: C:\azagent\A1\_work\r1\a\PublishBuildArtifacts\SiteCore.Website.SetParameters.xml
102019-09-24T17:17:27.3133832Z ##[warning]variable not found: dataFolder
112019-09-24T17:17:27.3179775Z ##[section]Finishing: Replace tokens in SiteCore.Website.SetParameters.xml

With all this set our config has it's variables configured within Azure Devops for each environment,

Sitecore strange language switching

Sitecore strange language switching

The other day we started experiencing a strange issue on one of our test sites. Pages were starting to error and looking at the logs the errors were happening in view's which hadn't been updated in a long time. Some of these seemed like the code wasn't robust enough to handle when a datasource hadn't been set (the classic object not set to an instance of an object error), but fixing these just resulted in an error in another. It also didn't explain why this suddenly started happening, not to mention the components in question should have all had data.

Then we noticed that on the first view of any page of the site with a new session the site would display fine even though these were server errors happening. Navigating to any other page or refreshing though caused the server error to return.

This led us to look at the differences between the request headers and we had our answer. On all subsequent requests the language context had been changed to some random thing which there is no content for. The site itself is not a multilanguage site and only has content for en. Adding the language code into the URL would force the context language back and the page would work again.

How language works in Sitecore

To understand what is happening it's good to know how languages in Sitecore work.

Sitecore is built as a platform which can server content in multiple languages. By default you start with one language in the editor (en) but are able to add more. You can read my blog post from a few years ago on how to do this here (https://himynameistim.com/2015/06/30/sitecore-adding-languages-for-a-multilingual-site/).

Sitecore will recognise which language should be displayed based on a language code in a URL. e.g. himynameistim.com/en-gb/ for English - Great Britian. This is done through the strip language pipeline which picks up these languages and sets the context language.

Config for the link manager then controls if links are generated with these language codes in the URL or not. On a single language site you would have this set to never resulting in URL's without a language code and the default language will be used.

The flaw in all of this though is the strip language pipeline always runs, even when your pages only have one lauange. The pipeline also doesn't check if the language it finds in the url is set up as a language on the site, so quite a lot of two letter combinations will work and change the language context. When this is changed, it is changed for the users session meaning it is possible for a url to inadvertedly cause the language context to change for a user on the site when the site only has one language.

Disabling Strip Language

As the site in question only has one language the fix is quite simple. For multilingual sites the solution is a bit harder.

For a single language site you can simply turn off the strip language functionality. You can do this using a patch config file as follows:

1<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
2 <sitecore>
3   <settings>
4    <setting name="Languages.AlwaysStripLanguage">
5       <patch:attribute name="value">false</patch:attribute>
6    </setting>
7   </settings>
8 </sitecore>
9</configuration>