Building a Continuous Integration Environment for Sitecore Part 8 – Using Unicorn for Sitecore Item Syncronization

This is part 8 of the series of post that I am doing about building a Continuous Integration Environment using Sitecore, TeamCity, Octopus Deploy, and Unicorn.

Part 1 – Setting Up TeamCity
Part 2 – Setting up OctopusDeploy
Part 3 – Setting up SQL Server for Sitecore
Part 4 – IIS
Part 5 – Octopus Environment and Deployment Configuration
Part 6 – TeamCity Project
Part 7 – OctopusDeploy Project
Part 8 – Sitecore Item Synchronization using Unicorn
Part 9 – Displaying Build number on Sitecore Home Page & Tagging Git
Part 10 – config transformations using Octopus Deploy
Part 11 – Deploying to a Sitecore Multi-Instance Configuration

Thoughts so far

I deliberately put off including anything to do with Sitecore item synchronization until I had my build and deployment process. I wanted to get that working first so that I can resolve any issues before throwing Sitecore into the mix. If I had a failure in either TeamCity or Octopus I wanted to narrow it down quickly and not worry about Sitecore.

Why Unicorn

If you have done any development work with Sitecore, you know that one of the biggest issues is synchronizing items. Templates, layouts, etc

To get around it,

• All your dev’s are working on a single shared Sitecore instance.
• Every time a dev does a code checkin they have to also generate a Sitecore Package of all the changes they made
• Use a tool

I am not a fan of people working on a shared database, even when not using Sitecore so that option is out.

Requiring people to create a package, you are going to end up with 100’s of package files, tracking them becomes a task in itself

Which leaves the last option, use a tool. The big daddy of the market is Hedgehog’s TDS. I have used TDS in the past. I know I never used the full capabilities of the product, but I never felt 100% happy using the product. I feel that it suffers from some of the issues with having dev’s create Sitecore packages, in that it requires a human to remember to create the .item files and then remember to sync. There is a lot of upfront project creation, a project for master, a project for core. Want to take a guess at how many times I forgot to create an item, or sync after doing a merge and then caused problems. Too many times to count.

And if your merge tool resolves a conflict in an .item file that TDS controls, and you and several team members have then sync’ed that file, you will have fun reversing out whatever you got left with. Once again this happened to me, spent way to much time resolving a template problem because of an item being merged “incorrectly.”

I never liked having to use Sitecore Courier either to deploy changes as that basically needs you to create a Sitecore Package which includes changes that need to be made. Trying to work out changes made on a multi-team project, with multiple check ins on git, and even with tools it was not fun trying to work out why a change was not present. Once again not having fun resolving issues because of missing/corrupt templates within Sitecore because the courier package was missing something.

I was told about Unicorn and I have been using it for several months and it does appear to be resolving some of the issues that I have experienced in the past.

• Want to have all your templates sync’ed to disk when you maintain them in Sitecore – Yes
• After doing a merge, automatically sync changes made – Yes using Transparent Sync
• Not having to worry about creating packages to deploy via Sitecore Courier – Yes

Unicorn ticked these and many more boxes for me.

I am sure that anybody who has worked on Sitecore for more than 5 minutes has their pain points in regards to item synchronization

Getting Started

I am not going to go into too much detail because the readme on the home page details a lot, I am just going to detail what I did to get it working. I am also going to assume you have already set up your solution within Visual Studio.

From your web project, start the package manager console and type in the following

Install-Package Unicorn

Or just do it from the NuGet Package manager dialog. At the time of this post, I am using Unicorn 3.2.

Unicorn uses Yaml for item sync what is soo much easier for resolving merge conflicts than resolving Sitecore .item file conflicts ever where.

Unicorn comes with a Unicorn.configs.default.example. As with everything Sitecore, remove the .example and its ready to use. I initially started using this file to get my feet wet, but I feel that the default config file supplied wants to put too much under Source Control. Convention that I work by is that only items that are created for the solution should be put under source control. Actual content should never be placed under source control. Items included with the default Sitecore database should never be put under source control. Items created via packages should never be put under source control

This is the sample supplied to put items under the layout under Unicorn’s control.

<!-- Master Layout items (and renderings, sublayouts, etc) -->
<include database="master" path="/sitecore/layout">
	<!-- NOTE: You cannot use excludes with Transparent Sync. See https://github.com/kamsar/Unicorn/wiki/The-Transparent-Sync-Guide -->
	<exclude path="/sitecore/layout/Simulators" />
	<exclude path="/sitecore/layout/Layouts/System" />
	<exclude path="/sitecore/layout/Renderings/App Center Sync" />
	<exclude path="/sitecore/layout/Renderings/System" />
	<exclude path="/sitecore/layout/Renderings/Social MVC" />
	<exclude path="/sitecore/layout/Sublayouts/App Center Sync" />
	<exclude path="/sitecore/layout/Sublayouts/Social" />
	<exclude path="/sitecore/layout/Placeholder Settings/App Center Sync" />
</include>

I wanted to use Transparent Sync for development, the comment says you cannot use excludes’ with Transparent Sync

Even if I was not going to use transparent sync, I feel that you have to include too many exceptions using an exclude node. As most of development is done using MVC, all the renderings should be placed within a single folder within the Renderings folder, and then that single folder is put under Source Control

To include this within Unicorn all that is needed is

<include name="Layout.Renderings.ProjectName" database="master" path="/sitecore/layout/Renderings/ProjectName" />

The hardest part I discovered was creating meaningful unique names for each item you want to put under source control.

If you follow the convention of creating a custom folder, and then adding in the relevant items into the custom folder, there are not many folders that actually need to be put under Unicorn’s control if you follow the convention of creating a sub folder.

This is my full unicorn config file

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <unicorn>
      <configurations>
        <configuration name="Project" description="Custom configuration for CompanyName">
          <predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
            <include name="Home" database="master" path="/sitecore/layout/Controllers/Home" />
            <include name="Layout.Layouts" database="master" path="/sitecore/layout/Layouts/Project" />
            <include name="Layout.Models" database="master" path="/sitecore/layout/Models/Project" />
            <include name="Layout.PlaceholderSettings" database="master" path="/sitecore/layout/Placeholder Settings/Project" />
            <include name="Layout.Renderings.Project" database="master" path="/sitecore/layout/Renderings/Project" />
            <include name="System.Dictionary.Project" database="master" path="/sitecore/system/Dictionary/Project" />
            <include name="System.Languages" database="master" path="/sitecore/system/Languages" />
            <include name="System.Workflows" database="master" path="/sitecore/system/Workflows" />
            <include name="Templates.Project" database="master" path="/sitecore/templates/Project" />
            <include name="System.HtmlEditorProfiles" database="core" path="/sitecore/system/Settings/Html Editor Profiles/Project" />
          </predicate>
        </configuration>
      </configurations>
    </unicorn>
  </sitecore>
</configuration>

With the exception of Languages and Workflows, all items are created by default in their own custom folder. I am using Project here but this can be replace with any relevant name.

As you can see, Workflow and Language exceptions aside, none of the items supplied by Sitecore are placed under source control.

There were a few more changes that I needed to make. All done through my own patch file so that I don’t have any problems during an upgrade.

    <unicorn>
      <defaults>
        <targetDataStore type="Rainbow.Storage.SerializationFileSystemDataStore, Rainbow" useDataCache="false">
          <patch:attribute name="physicalRootPath">~/Unicorn/$(configurationName)</patch:attribute>
        </targetDataStore>
        <dataProviderConfiguration type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn">
          <patch:attribute name="enableTransparentSync">true</patch:attribute>
        </dataProviderConfiguration>
      </defaults>
    </unicorn>
    <settings>
      <setting name="Rainbow.SFS.SerializationFolderPathMaxLength">
        <patch:attribute name="value">110</patch:attribute>
      </setting>
    </settings>

Rainbow.Storage.SerializationFileSystemDataStore Is to set the folder where all the yaml files are to be created. My gitignore file will not include anything within the data folder, so I had to specify another location than the one supplied by default so that it would be automatically picked up as I didn’t have to put in ignore rule exception in my .gitignore file. As you can guess not a big fan of exceptions

Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration is for enabling Transparent Sync.

Another change that was necessary for me was modify the setting Rainbow.SFS.SerializationFolderPathMaxLength. I set this value to 110. This was done purely because of my development environment and you may not need to alter this from the default value. This setting was changed via my own .config file and not by directly modifying rainbox.config

First Run

After reading the documentation, all you need to do is run /unicorn.aspx. This will display the Unicorn Control Panel.

Click Sync to get your first initial files. You could have already switched on Transparent Sync but I initially had it disabled. The xml above is from my final config file, and not what I used initially.

Once that has completed, go to the config files and enable Transparent Sync.

If you had enabled transparent sync before going to the Unicorn Control Panel, you will get a warning about no files and that you will need to do an initial sync. Quickest way to do this after sync. Delete the unicorn folder.

Testing

I wanted to make sure everything was working, so I created a template, then I did the following

• Rename the .yaml file. Change automatically applied
• Copied the yaml file, and modified the ID. New template created automatically
• Deleted the yaml file. Template automatically deleted.

I performed more tests than the ones I have listed, and no matter what I did, Unicorn kept my Sitecore system in site.

Now that this is finished, its time to modify my build and deployment process.

TeamCity

The only change I had to make was to update my .nuspec file to include the Unicorn folder. No other changes where required to TeamCity.

  <files>
    <file src="App_Config\**\*.config" target="App_Config" />
    <file src="Assets\**\*.*" target="Assets" />
    <file src="bin\*.dll" target="bin" />
    <file src="Views\**\*" target="Views" />
    <file src="default.css" target="" />
    <file src="default.htm.sitedown" target="" />
    <file src="default.js" target="" />
    <file src="Global.asax" target="" />
    <file src="Web.*.config" target="" />
    <file src="Web.config" target="" />
    <file src="webedit.css" target="" />
    <file src="emails\**\*.*" target="emails" />
    <file src="*.ps1" target="" />
    <file src="Unicorn\**\*.*" target="Unicorn" />
  </files>
</package>

This is my updated .NuSpec file. I only had to add in the line

<file src="Unicorn\**\*.*" target="Unicorn" />

Once the Unicorn files where included in the TeamCity artefact everything else will be performed by Octopus.

Octopus Deploy

I needed to do a little bit more work with my Octopus Deploy project to deal with Unicorn, than with what I had to change with TeamCity. I performed these changes over multiple steps, and I broke the build because I wanted to do a small change and verify changes made manually rather than doing one massive big bang approach and then spending time working out what broke.

Moving Unicorn Files out of Webroot

As you can see from above, based on the Octopus Deploy project created in Step 7 Unicorn’s yaml files are going to be written to D:\WebSites\ProjectName.CI\Website\Unicorn when the Octopus step “DeploySite” is executed. I did not want to have the sync files within the web root. So the first task I set myself was to move the files out of the web root.

When I deploy to CI this is the folder structure being used

D:\WebSites\ProjectName.CI\data
D:\WebSites\ProjectName.CI\Website

The Unicorn yaml files where stored in D:\WebSites\ProjectName.CI\Website\Unicorn.

I just decided that I would move the Unicorn folder from D:\WebSites\ProjectName.CI\Website\ to D:\WebSites\ProjectName.CI\Data\

Step Template – Move Unicorn files out of webroot

This first thing I needed to do was to create a new custom step template that will

• Copy the files to the new location
• Delete the files from the old location

I just decided on doing a copy command instead of a move. Feel free to change if required.

This is the script to be imported into Octopus

{
  "Id": "ActionTemplates-62",
  "Name": "Move Unicorn files out of webroot",
  "Description": null,
  "ActionType": "Octopus.Script",
  "Version": 9,
  "Properties": {
    "Octopus.Action.Package.NuGetFeedId": "feeds-builtin",
    "Octopus.Action.Script.Syntax": "PowerShell",
    "Octopus.Action.Script.ScriptSource": "Inline",
    "Octopus.Action.RunOnServer": "false",
    "Octopus.Action.Script.ScriptBody": "$dataFolder = $OctopusParameters['data.folder']\r\n$websiteFolder  = $OctopusParameters['website.folder']\r\n\r\nWrite-Output \"dataFolder: \" $dataFolder\r\nWrite-Output \"websiteFolder: \" $websiteFolder\r\n\r\nif((Test-Path -Path $dataFolder\\Unicorn )){\r\n    Write-Output \"[io.directory]::delete(\"\"$dataFolder\\Unicorn\"\")\"\r\n    Get-ChildItem -Path \"$dataFolder\\Unicorn\\*\" -Recurse | Remove-Item -Force -Recurse\r\n    remove-item -path $dataFolder\\Unicorn -Recurse -Force\r\n}    \r\nWrite-Output \"copy-item -path $websiteFolder\\Unicorn -destination $dataFolder -recurse -force\"\r\ncopy-item -path $websiteFolder\\Unicorn -destination $dataFolder -recurse -force\r\n\r\nWrite-Output \"[io.directory]::delete(\"\"$websiteFolder\\Unicorn\"\")\"\r\n#[io.directory]::delete(\"$websiteFolder\\Unicorn\", true)\r\n    Get-ChildItem -Path \"$websiteFolder\\Unicorn\\*\" -Recurse | Remove-Item -Force -Recurse\r\n    remove-item -path $websiteFolder\\Unicorn -Recurse -Force\r\n\r\n"
  },
  "Parameters": [],
  "$Meta": {
    "OctopusVersion": "3.3.1",
    "Type": "ActionTemplate"
  }
}

No parameters are needed as it will be using variables already defined

Update Deployment Process

Within Octopus Deploy web portal, load the project, and select Process. Click Add Step

Scroll to select “Move Unicorn files out of webroot”

Ensure that the roles CD and CD are selected. No other changes are required. Save the process

The newly created step needs to be moved. Within the process, click Reorder steps

Move the step “Move Unicorn files out of webroot” from being the last step to being the step immediatelty after Deploy Site.

At this point I saved everything and triggered a build from TeamCity. Everything worked as I expected. Remember that I am not finished so even if the Deployment Process was broken I expected this to happen.

I needed to let Unicorn know where its yaml files where located.

In Visual Studio I needed to update my patch file

<unicorn>
  <defaults>
    <targetDataStore type="Rainbow.Storage.SerializationFileSystemDataStore, Rainbow" useDataCache="false">
      <patch:attribute name="physicalRootPath" xdt:Transform="Replace">#{data.folder}\Unicorn\$(configurationName)</patch:attribute>
    </targetDataStore>
    <dataProviderConfiguration type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn" xdt:Transform="Replace">
      <patch:attribute name="enableTransparentSync">false</patch:attribute>
    </dataProviderConfiguration>
  </defaults>
</unicorn>

I disabled transparent sync as reading the web this was shown to be a good practice on non development environments. I will be detailing how I got unicorn sync’ing working shortly

#{data.folder} is an OctopusDeploy variable, and when the site is being deployed, it will be replaced with the variable’s value

Pushed to git and triggered a new Build / Deployment processes. Once it had finished I checked everything and was satisfied with progress. The files where in the new location, and Unicorn knew where the files where. Time to get the Deployment process to perform the synchronization.

Sync’ing from Octopus

As I had disabled transparent sync. I needed to use the Automated Tool Api. The documentation on the site is very comprehensive so I don’t see the need to duplicate it all here.

On the Unicorn / doc / Powershell Remote Scripting page, you need to download MicroCHAP.dll. Place this within the Tools folder on the IIS Server. I eventually placed this file in folder D:\Tools\UnicornSync. Make a note of where you saved this file. This file is needed to authenticate with Unicorn

Environment – Variable Sets

Within the Octopus Deploy Web Portal, select Library | Variable Sets | External Software Paths

Just add in a new line that will specify the location of MicroCHAP.dll. I added in a variable named MicroChap.path, and a value of D:\Tools\UnicornSync\MicroCHAP.dll. This is important as this variable is used in new Step Template that will be created

Don’t forget to save the page

Step Template – Unicorn Sync

This step is pretty much a copy of the sample.ps1 file that is in the same folder on GitHub as MicroCHAP.dll

This step will call Unicorn’s API and tell it to perform a sync. In short you call the Unicorn Control Panel and tell it to perform a sync.

There are a few variables that need to be passed to perform the sync successfully.

Pass in the SharedSecret so that Unicorn can authenticate itself
The list of Unicorn configurations to sync
The name of the website

This is the step template to import

{
  "Id": "ActionTemplates-61",
  "Name": "Unicorn Sync",
  "Description": "Calls Unicorn sync",
  "ActionType": "Octopus.Script",
  "Version": 8,
  "Properties": {
    "Octopus.Action.Package.NuGetFeedId": "feeds-builtin",
    "Octopus.Action.Script.Syntax": "PowerShell",
    "Octopus.Action.Script.ScriptSource": "Inline",
    "Octopus.Action.RunOnServer": "false",
    "Octopus.Action.Script.ScriptBody": "$ErrorActionPreference = 'Stop'\r\n\r\n$microchap = $OctopusParameters['MicroChap.path']\r\n\r\nAdd-Type -Path $microchap\r\n\r\nFunction Sync-Unicorn {\r\n    Param(\r\n        [Parameter(Mandatory=$True)]\r\n        [string]$ControlPanelUrl,\r\n\r\n        [Parameter(Mandatory=$True)]\r\n        [string]$SharedSecret,\r\n\r\n        [Parameter(Mandatory=$True)]\r\n        [string[]]$Configurations,\r\n\r\n        [string]$Verb = 'Sync'\r\n    )\r\n\r\n    # PARSE THE URL TO REQUEST\r\n    $parsedConfigurations = ($Configurations) -join \"^\"\r\n\r\n    $url = \"{0}?verb={1}&configuration={2}\" -f $ControlPanelUrl, $Verb, $parsedConfigurations\r\n\r\n    Write-Host \"Sync-Unicorn: Preparing authorization for $url\"\r\n\r\n    # GET AN AUTH CHALLENGE\r\n    $challenge = Get-Challenge -ControlPanelUrl $ControlPanelUrl\r\n\r\n    Write-Host \"Sync-Unicorn: Received challenge: $challenge\"\r\n\r\n    # CREATE A SIGNATURE WITH THE SHARED SECRET AND CHALLENGE\r\n    $signatureService = New-Object MicroCHAP.SignatureService -ArgumentList $SharedSecret\r\n\r\n    $signature = $signatureService.CreateSignature($challenge, $url, $null)\r\n\r\n    Write-Host \"Sync-Unicorn: Created signature $signature, executing $Verb...\"\r\n\r\n    # USING THE SIGNATURE, EXECUTE UNICORN\r\n    $result = Invoke-WebRequest -Uri $url -Headers @{ \"X-MC-MAC\" = $signature; \"X-MC-Nonce\" = $challenge } -TimeoutSec 10800 -UseBasicParsing\r\n\r\n    $result.Content\r\n}\r\n\r\nFunction Get-Challenge {\r\n    Param(\r\n        [Parameter(Mandatory=$True)]\r\n        [string]$ControlPanelUrl\r\n    )\r\n\r\n    $url = \"$($ControlPanelUrl)?verb=Challenge\"\r\n\r\n    $result = Invoke-WebRequest -Uri $url -TimeoutSec 360 -UseBasicParsing\r\n\r\n    $result.Content\r\n\r\n}\r\n\r\n$websiteName = $OctopusParameters['website.name']\r\n$SharedSecret = $OctopusParameters['SharedSecret']\r\n$Configurations = $OctopusParameters['Sync.Configurations']\r\n\r\nWrite-Output \"websiteName\" $websiteName\r\nif ( $SharedSecret ) {\r\n    Write-Output \"Shared secret supplied\"\r\n} \r\nelse {\r\n    Write-Output \"No Shared Secret Supplied\"\r\n}\r\nWrite-Output \"Configurations\" $Configurations\r\n\r\nif ( $Configurations ) {\r\n    $configs = $Configurations.split(\"`n\")\r\n}\r\nelse {\r\n    $configs = \" \"\r\n}\r\n\r\nSync-Unicorn -ControlPanelUrl \"http://$websiteName/unicorn.aspx\" -SharedSecret $SharedSecret -Configurations $configs\r\n"
  },
  "Parameters": [
    {
      "Name": "SharedSecret",
      "Label": "Shared Secret",
      "HelpText": null,
      "DefaultValue": null,
      "DisplaySettings": {
        "Octopus.ControlType": "Sensitive"
      }
    },
    {
      "Name": "Sync.Configurations",
      "Label": "List of Configurations to Sync",
      "HelpText": null,
      "DefaultValue": null,
      "DisplaySettings": {
        "Octopus.ControlType": "MultiLineText"
      }
    },
    {
      "Name": "website.name",
      "Label": "Website Name",
      "HelpText": null,
      "DefaultValue": null,
      "DisplaySettings": {
        "Octopus.ControlType": "SingleLineText"
      }
    }
  ],
  "$Meta": {
    "OctopusVersion": "3.3.1",
    "Type": "ActionTemplate"
  }
}

Update Deployment Process

Within Octopus Deploy web portal, load the project, and select Process. Click Add Step

Scroll to select “Unicorn sync”

Make a note of what you use for the shared secret, as we will need to store this in a patch file. To generate the shared secret I used https://www.passwordgenerator.com and generated a 128 digit password to be used as the shared secret.

If you only have a single configuration file, you do not need to supply the List of Configurations to Sync. It can be left blank.

Save the step. There is no need to change the step order in the deployment process. I am happy having this as the last step run in the process.

In Visual Studio I needed to update my patch file

    <unicorn>
      <authenticationProvider type="Unicorn.ControlPanel.Security.ChapAuthenticationProvider, Unicorn">
        <SharedSecret>MySharedSecretGeneratedFromPasswordsGeneratorDotCom</SharedSecret>
      </authenticationProvider>
    </unicorn>

Save the changes and push to git.

After watching the whole process ( it takes a full 10 minutes ) I logged into Sitecore and yes, my custom template that I had created on my development environment was there. As with my local machine I did a few tests. Make sure that it would rename a Sitecore Item template, would delete a Sitecore Item template. Everything I did correctly updated the Sitecore instance on my CI environment.

Thoughts

Now after 8 steps I now have a system that will make sure that any change that is made, either to Sitecore or my custom code, is deployed to my CI environment.

So what’s the first thing I did. I broke it.

How?

I added a new template, and forgot to run “git add” before running “git commit” I know how ironic especially after what I said at the start of this post about relying on developers to do things manually.

Upgrading Unicorn

I initially was using Unicorn 3.1 and while building this process, 3.2 was released. Because I did not modify any of the original files, NuGet had no problem upgrading to the newest version of Unicorn.

Next Steps

First a break. I initially did all this over 4 days. I has taken me nearly two weeks to blog about this.

Right now I am looking to create a deploy process to satisfy the following:

Your environment may be different from mine, either in name or number.

Next I need to extend Octopus to deploy to the next environment, which is my instance is QA.

About Darren

My musing about anything and everything

‹ Building a Continuous Integration Environment for Sitecore Part 7 – Octopus Deploy project

Displaying Build number on Sitecore Home Page & Tagging Git ›

Tagged with: CI, Continuous Integration, Octopus, Octopus Deploy, sitecore, TeamCity, Visual Studio
Posted in C#, CI, Continuous Integration, Octopus, Octopus Deploy, Sitecore, TeamCity, Unicorn, Visual Studio