With Sitecore 9 the way analytics is recorded and processed changed. Rather than everything being done by the IIS application, the architecture changed to include:

1. A second IIS application called xconnect
2. A windows services called Sitecore XConnect Search Indexer

As well as this Mongo was replaced by:

1. SQL Server databases
2. A SOLR XDB Core

Sitecore 9 also introduced data being secure in transit as well as at rest which means all traffic is encrypted using certificates.

While all these peices can be considered good, it does also create more points of failure that becomes harder to debug. So after spending a decent amount of time debugging why no analytic reports were loading and then why no data was appearing in the reports, I've made a checklist to go through.

Debugging Sitecore Analytics Checklist

1. XConnect Connection String

The main Sitecore application has connection strings for where it will find the XConnect service. They will look like this:

One gives the path to the XConnect service and the other gives the path to find the certificate.

Firstly check that the XConnect path is correct and that you can access it and secondly check that the thumbprint corresponds with a certificate in the certificate store.

You can see what certificates are on your machine using this PowerShell script:

2. Check certificate expiry date

Having a certificate is a good start, but it could still have expired.

Find the certificate in the certificate store by hitting start > manage computer certificates and find it in one of the folders.

Check the expiration date. If it's in the past then it's not going to work. This is common because the installation script for Sitecore 9 will set the expiration date to a year after install by default.

If it has expired you will need a new cert, you can create this by using the same script that you used to install sitcore origionally (Just the certificate bit).

3. Check security permissons on the certificate

If you have a certificate and it's still valid then it could be that the app pool the site is running in doesn't have access to read the certificate.

To check this:

1. Right click the certificate
2. All Tasks
3. Manage Private Keys
4. Check that the app pool user for your site is listed in the list of users and that it has read permission. If it's not there add it using the name
   IIS APPPOOL\app pool name

4. Check License Files

Partner licenses only last for a year so if your using one of those it may have expired.

We're all used to checking the license file in the Sitecore application but XConnect has a license too.

These will be in:
sitename.xconnect\App_data\
sitename.xconnect\App_data\jobs\continuous\IndexWorker\App_data
sitename.xconnect\App_data\jobs\continuous\AutomationEngine\App_Data

Only the first will be used when your viewing the site, but it's worth knowing about the others to, incase you ever run a job manually.

5. Check manual rebuild of indexes

You can trigger a manual rebuild of the xDB index by following these instructions:

https://doc.sitecore.com/developers/90/sitecore-experience-platform/en/rebuild-the-xdb-index-in-solr.html

Remember in point 4 that it has it's own license file. It also has it's own connection strings.

6. Check XConnect site works in a browser

If you open XConnect in a browser you should recieve no certificate errors and a timestamp saying how long XConnect had been running for.

7. Check certificates are in the right store

This stack overflow post was a big help for me (https://stackoverflow.com/questions/26247462/http-error-403-16-client-certificate-trust-issue ). I was at the point where everything seemed right, but moving the certificates as shown here got it to the point of the analytics reports loading.

Windows 2012 introduced stricter certificate store validations. According to KB 2795828: Lync Server 2013 Front-End service cannot start in Windows Server 2012, the Trusted Root Certification Authorities (i.e. Root) store can only have certificates that are self-signed. If that store contains non-self-signed certificates, client certificate authentication under IIS returns with a 403.16 error code.

To solve the problem, you have to remove all non-self-signed certificates from the root store. This PowerShell command will identify non-self-signed certificates:

In my situation, we moved these non-self-signed certificates into the Intermediate Certification Authorities (i.e. CA) store:

Checklist for data not going into Analytics

If you've got to the point of the analytics reports working, but not showing any data, this is my checklist for making sure data goes in. In my case I was trying to log site searches as per my article from a few years ago https://himynameistim.com/2017/09/13/populating-the-internal-search-report-in-sitecore/ there wern't any errors, but no data ever showed.

1. Enable Analytics Debugging

In Sitecore.Analytics.Tracking.config there is a setting to set the analytics logging level to debug. You will also need to set the log level on log4net root in Sitecore.config to debug.

2. Disable Robot Detection

In my case Sitecore thought I was a robot. Changing these settings will disable that:

3. Test analytics is tracking something

One of the hardest parts about analytics is it's not instant. The initial tracking only goes into the DB at the end of the users session and that's only for collection. It won't appear in the reports until processing has happened. So to speed this up:

Create a page called kill.aspx as follows. This will end the users session and trigger the data to be fed into the DB.

Next do something on the site that will cause some tracking to get added to the DB. In my case it was the search. Then go to kill.aspx to force session abondon.

Check the logs. You should see something like this...

Check the DB an entry should have gone into a shard db for [xdb_collection].[Interactions] table

To speed up processing, restart the main sitecore application.

Out the box Sitecore ships with a number of reports pre-configured. Some of these will show data without you doing anything. e.g. The pages report will automatically start showing the top entry and exit pages as a page view is something Sitecore can track.

Other's like the internal search report will just show a message of no data to display, which can be confusing/frustrating for your users. Particularly when they've just spent money on a license fee to get great analytics data only to see a blank report.

The reason it doesn't show any information is relatively straight forward. Sitecore doesn't know how your site search is going to work and therefore it can't do the data capture part of the process. That part of the process however is actually quite simple to do.

Sitecore has a set of page events that can be registered in the analytics tracker. Some of these like Page Visited will be handled by Sitecore. In this instance the one we are interested in is Search and will we have to register it manually.

To register the search event use some code like this (note, there is a constant that references the item id of the search event). The query parameter should be populated with the search term the user entered.

Now after triggering the code to be called a few times, your internal search report should start to be populated like this.

If you need to rebuild your Path Analyzer maps, Sitecore have a knowledge base article on how to do it here - https://kb.sitecore.net/articles/766858

In the FAQ section it gives some info on how long you should expect it to take

What it doesn't tell you though is how you find out how many interactions your site has to process. The data is stored in analytics collection in your Mongo DB, so go there an then run...

One of the new features in Sitecore 7.5 was the xDB, a new analytics database that rather than using SQL Server was Mongo based. If your main job is as a Sitecore dev, like me this may be the first time you've ever had to use Mongo, or a least the first time you've ever had to commercially use it.

Installing Mongo DB is a relatively painless experience, Mongo provide a decent enough guide to installing on Windows. Half way through you may be wondering why its running in a console window, but eventually you get to instructions for setting it up as a service.

However there are a couple of gotchas:

1. Sitecore doesn't work with the latest version of Mongo. At time of writing the latest version of Sitecore is 8.0 rev. 150427 and Mongo is 3.0. Sitecore however only supports version 2.6.x of Mongo though.
2. None of the installation instructions include any details on authentication. Which while this is fine for your local dev machine is not so great for a live site.

So here's my guide for getting up and running with Mongo DB and Sitecore.

Installing Mongo

First off get Mongo DB installed as a service on your machine. I'm not going to include instructions here for this bit, just follow the guide on the Mongo DB website.

One thing to note though, the Windows installer for Mongo 2.6 will by default install Mongo to C:\Program Files\MongoDB 2.6 Standard\. However the instructions for setting up Mongo include example commands expecting it to be installed in C:\mongodb\

Setting Up Authentication

Once you've got Mongo up and running you'll need to create a user.

1. Open a command prompt and connect to MongoDB
   
   C:\Program Files\MongoDB 2.6 Standard\bin\mongo.exe
   
2. Switch to the admin db by doing
   
   use admin
   
3. Create the user using
   
   db.createUser({user: "admin_mongo",pwd: "your password",roles: [ { role: "userAdminAnyDatabase", db:"admin" }, { role: "root", db:"admin" } ] })
   
4. To verify the user has been created you can use the following commands
   
   db.auth("admin_mongo", "your password")
   db.getUsers()
   

Next update your config file to require authentication by adding auth=true. You will also need to restart the service for the change to take effect.

logpath=c:\data\log\mongod.log
dbpath=c:\data\db
auth=true

Sitecore uses 4 db's in Mongo; Analytics, tracking_live, tracking_history and tracking_contact. We need to create a user in each of them.

Open a mongo shell again, switch to the admin db and connect with your admin login.

Now create each of the users as follows:

use analytics
db.createUser({user: "mongo_user",pwd: "your password",roles: [ { role: "readWrite", db:"analytics" } ]  })

use tracking_live
db.createUser({user: "mongo_user",pwd: "your password",roles: [ { role: "readWrite", db:"tracking_live" } ]  })

use tracking_history
db.createUser({user: "mongo_user",pwd: "your password",roles: [ { role: "readWrite", db:"tracking_history" } ]  })

use tracking_contact
db.createUser({user: "mongo_user",pwd: "your password",roles: [ { role: "readWrite", db:"tracking_contact" } ]  })

Sitecore Connection Strings

All that's left now is to update the connection strings in Sitecore.

The default connection strings Sitecore give you look like this:

The format for a connection string with authentication is

mongodb://[username:password@]host1[:port1]/database

One of Sitecores features is the ability to assign profile cards or profile key scores to pages. As visitors browse the website the profile key scores from each page are added to the visitors profile. Visitors are then assigned a pattern card using N-Dimonsional Euclidean Distance to calculate the closest matching pattern card to the visitors profile.

However before a pattern card is assigned to a visitor they must visit a minimum number of pages. The default for this is set to 3.

Reasons for changing this could be that you actually want to be more certain of a users profile before triggering personalisation this could be to avoid visitors landing pages on the site heavily dictating which persona they initially become profiled as, which in turn could influence personalisation that keeps them in that persona.

Or it could be because you want to lower the number of pages for a more immediate effect.

Whatever the reason to change the default from 3 you need to add the following setting to your config:

Sitecore's Engagement Analytic's engine automatically tracks all page requests. When you assign profile cards to items this also triggers data about a persons interests to start being built up against their visit record, which can then be used to create a personalized site experience.

However what if you want items that are not pages to also contribute to a users profile?

This scenario came about with a recent site we took over that the client wanted to add personalisation too. The site (for unknown reasons) had been built with one product page which looked at a querystring parameter to determine the product information to be displayed (the querystring format was hidden from end users through the use of a URL rewrite). Product data was being stored as Sitecore items allowing them to have profile cards assigned, but as the item was never visited the profile cards values were never applied to the users profile.

After a bit of searching through the Sitecore.Analytics.dll I stumbled across the TrackingFieldProcessor class. This class contains a process function that takes an item parameter an in turn triggers all the functionality for processing campaigns, profiles and events related to the item.

To use it your code would look like this:

This will cause the profile key values to be updated but in the visitor history it still looks like the visitor was only looking at the one page. To change those values we can do this:

Note: I was unable to find any documentation on these functions, or any official way of doing this. Use at your own risk!