If you’re working with Umbraco, the chances are that at some point you’re going to be asked to integrate with a CRM (customer relationship management) solution, such as Microsoft Dynamics.
Sigma was approached to do just this for one of our clients, to allow us to track new members and purchase orders. This article aims to cover some of the integration options we explored and hopefully provides some useful hints and tips. When integrating there are several things to consider from a technical perspective, including:
- Architecture for the database and the code
- At what point in the process the integration should take place
- Error handling
- Understanding the Dynamics model
- Integration options and which are bested suited for this development
Also, developers and clients should consider:
- GDPR - as Dynamics and Umbraco will store data, GDPR should always be considered
- Development, test, and live environments - it is preferable to separate the different environments to ensure that the live environment is not amended whilst development and testing are underway
Architecture
Data
One of the important decisions to make when integrating with Dynamics is what data will be captured and stored where. In this model, we were using Dynamics to generate invoices. In this instance, an invoice has a contact and product details attached. This meant members who have registered need to have a record in Dynamics, and the products on sale also needed to exist in Dynamics.
To search Dynamics products via the website would have resulted in poor performance on the website, as the site would have needed to query Dynamics each time. The products also needed to be categorised and tagged for display on the web, data which Dynamics did not require. Products on this site don’t change often and new products are rarely created so we decided to dual key data into the website and Dynamics, with the website holding the Dynamics ID for the related Dynamics product.
Member data in the website is modeled as contacts in Dynamics. Whilst Umbraco does offer the facility to swap out the membership provider, in this case the data held between the two is very different, so we added a Dynamics ID to the member data to relate the member record to their Dynamics record.
And finally, because only successful orders are invoiced in Dynamics we opted to maintain the order data via the website, and only push out successful orders to Dynamics using a Dynamics ID on the record to indicate the order had been pushed into Dynamics.
Code
We created two separate projects for the Dynamics integration:
- Dynamics.Service - this managed the integration points to Dynamics
- Dynamics.Domain - this allowed us to model the Dynamics data in much the same way as a developer would when using an Object-relational mapping (ORM) such as Entity Framework
Integration Points
If the integration is for user registration or to record an order being completed, it may feel natural to add the call to the CRM at the point in the code that the data has been saved to Umbraco (for example, within the member save event or the order pipeline). But doing so at this point we found can slow the user journey down on the website.
We resolved this by adding a Dynamics ID to both the order data and the member data, and then using a scheduled task to pass the member and order data to Dynamics which didn’t have the ID. There are some options for using scheduled tasks in Umbraco:
- Umbraco 7 supports scheduled tasks: https://our.umbraco.com/Documentation/Reference/Config/umbracoSettings/#scheduledtasks (as long as the site has enough traffic to keep the scheduled task running)
- Umbraco 8 supports scheduling using BackgroundTask runner: https://our.umbraco.com/documentation/reference/Scheduling/
- Or explore HangFire as a scheduled task runner in Umbraco: https://cultiv.nl/blog/using-hangfire-for-scheduled-tasks-in-umbraco/
Note: When running in a load-balanced environment, ensure that only one server (the admin server) is configured to run the scheduled tasks.
Also, in Umbraco 8 Callum Whyte has been working on a project called “Sidecar” which allows Umbraco to run as a console, function or service: https://github.com/callumbwhyte/sidecar.
Error Handling
By using a scheduled task for the integrations, all errors are hidden from the end-user (and from the client), so the next thing to consider is how to make them visible and to whom. This means it’s important to log the errors. In Umbraco 8, errors in the log file are surfaced in the logviewer: https://our.umbraco.com/documentation/getting-started/Backoffice/LogViewer/. For Umbraco 7, you can use Dan Booth’s Diplo Trace Log Viewer: https://our.umbraco.com/packages/developer-tools/diplo-trace-log-viewer/. If you followed Seb’s HangFire suggestions it’s possible to surface the errors in the HangFire dashboard.
In a load-balanced environment, as long as your admin server is the only one running the scheduled tasks, then the errors are visible in the back office.
However, this means the user needs to log in to see errors. So anyone managing the data in Dynamics needs to have an Umbraco login, and also remember to log in to see any errors. They may also see errors that are not related to the integration.
You can of course notify users via email when an error occurs, but this needs building and an interface needs developing, which details who should receive the emails, what the emails should contain, etc. You also need to consider how not to spam the recipient when the connection goes down or there is a data error.
Because the site is hosted in an Azure App Service, we have made use of application insights to notify of errors: https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview. This means control over frequency of email notifications and full drill-down of the logs is supported in Azure.
Understand the Dynamics Model
As an Umbraco developer, the chances are you have been asked to integrate with Dynamics after the Dynamics system has been built.
To understand what entities and associated properties are to be populated, book a meeting with the client and the Dynamics developers to take you through them, in Dynamics.
Dynamics Entities and Properties
Exploring the Data Model
It may be that the Dynamics developers are unavailable to discuss the setup. In which case, once the client has demonstrated the forms and the fields to be maintained you will need to find the entities and properties they map to. To do this go to settings -> customizations -> customize the system.
For each entity, it is possible to view the Dynamics Forms related to it:
Exploring the Dynamics Form
If you already know the dynamics names for the entities you’re working with and you just want to know a name for a property on the form. Whilst in that form launch the form editor from the toolbar:
Integration Options
The settings -> customizations section has a developer resources section that will list out the different ways to connect to the instance of Dynamics 365 you are working with.
The developer resources section will also list out the endpoints available for your service.
OData
OData is short for the Open Data Protocol and allows for RESTful CRUD management of the Dynamics data. The API endpoint for this is detailed in the developer resources section under “Instance Web API”.
I used this for exploring the entities. There is an example project found here: https://<yourinstance>/api/data/v9.1
This URL will list out all the entities.
From here, you can navigate and search them using the entity name: https://<yourinstance>/api/data/v9.1/accounts
And this can be queried using: https://<yourinstance>/api/data/v9.1/accounts?$select=name
https://<yourinstance>/api/data/v9.1/contacts?$filter=emailaddress1%20eq%20%27name%40email.com%27
The API supports the filter operators defined at: https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/webapi/query-data-web-api#standard-filter-operators
And CRUD samples can be found here: https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/webapi/web-api-samples-csharp
Whilst this is a quick and easy way to explore the data held in Dynamics out of the box, there are no models to work with. OData querying and support are limited. For example, it’s not possible to query across entities.
Organization Service
The organization service provides a more structured way to connect to the data. The Service is found in the NuGet package Microsoft.CrmSdk.CoreAssemblies.
There are many ways to instantiate the connection. Here I will explore two ways:
OrganizationServiceProxy
This method uses the following code:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var creds = new ClientCredentials();
creds.UserName.UserName = <userName>;
creds.UserName.Password = <password>;
var proxy= new OrganizationServiceProxy(new Uri(<Url>), null, creds, null);Replace <userName> with the service username, <password> with the service password and <Url> with a reference to the Organization.svc endpoint to query.
The proxy returns an “IsAuthenticated” value to indicate if the channel has been authenticated.
If you’re used to connecting to Dynamics using this method then it was deprecated on 4th February 2020.
CrmServiceClient
This replaces the OrganizationServiceProxy.
As well as connecting to Dynamics it comes with a utility CrmServiceClient.MakeSecureString which must be called to ensure that the password is encrypted correctly.
Connection string
The SDK and the documentation indicates that it is possible to connect to the CrmServiceClient via a connection string as shown below (in all honesty, I never got it to work.):
var connectionString = "AuthType=Office365;Username=<userName>Password=" +
CrmServiceClient.MakeSecureString(<password>) + ";Url=<Url>";
var client = new CrmServiceClient(connectionString);Login with user-supplied settings
var client = new CrmServiceClient(<user>,
CrmServiceClient.MakeSecureString(<password>),
"<region>", "<organisationName>", isOffice365: true,
useSsl:true);Where <region> is the region Dynamics is provisioned in and <organisationName> is the unique name of the instance, found in the developer resources section.
More constructor options can be found here: https://docs.microsoft.com/en-us/dotnet/api/microsoft.xrm.tooling.connector.crmserviceclient?view=Dynamics-xrmtooling-ce-9#constructors
Connection Errors
The CrmServiceClient may not error on connect As well as handling errors as you would normally, it is important to test the IsReady flag after connecting. If this is false LastCrmError and LastCrmException will be populated.
Migrating from OrganizationServiceProxy to CrmServiceClient
Migrating from the OrganizationServiceProxy to CrmServiceClient is detailed here. I did have to replace the instructions for changing the connection string to using Login with user-supplied settings as detailed above: https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/authenticate-office365-deprecation
Important dates on the deprecation (taken from the article):
- Effective October 2020, the authentication protocol will be retired for all new tenants.
- Effective April 2021, the authentication protocol will be retired for all new environments within a tenant.
- Effective April 2022, the authentication protocol will be retired for all new and existing environments within a tenant.
Common Data Service
FetchXML is a proprietary query language that is used by Dynamics.
The XML is built out as below:
<fetch distinct="false" mapping="logical" output-format="xml-platform" version="1.0">
<entity name="<entity>">
<attribute name="<property>"/>
<order descending="false" attribute="name"/>
</entity>
</fetch>Where <entity> is the name of the entity you wish to query, and <property> is the name of the property to retrieve. To retrieve multiple properties you can add multiple <attribute> elements:
<fetch distinct="false" mapping="logical" output-format="xml-platform" version="1.0">
<entity name="account">
<attribute name="name"/>
<attribute name="accountid"/>
<order descending="false" attribute="name"/>
</entity>
</fetch>Here the query is returning the name and accountid for accounts, and they are ordered by name. The OrganizationServiceProxy or the CrmServiceClient can then be queried using: connection.RetrieveMultiple(new FetchExpression(<xml>))
This returns an object of type EntityCollection where each item in the collection is of type Entity.
All properties queried are then accessed via attributes on the entity collection:
account.Attributes["name"].ToString();
And to create an entity in Dynamics, find the name of the entity using the forms we explored above, and the properties and then you can do this:
var contact = new Entity("contact")
{
["emailaddress"] = emailAddress,
["telephone"] = phoneNumber,
["firstname"] = forename,
["lastname"] = surname
};With the OrganizationServiceProxy or the CrmServiceClient you can call the Create method: connection.Create(contact);
For simple queries and simple objects this works well. For more information see: https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/use-fetchxml-construct-query
For objects where we wish to query by related objects or create or update an object which has foreign keys on it the XML can become unwieldy and the setting of attributes difficult.
There has to be an easier way!
Where’s the ORM?
As developers, we are used to using ORMs like EntityFramework, NHibernate or NPoco.
With a bit of legwork, the same can be done using the Microsoft.CrmSdk.
The SDK ships with crmsvcutil.exe and this provides a method for generating an early bound model based on the properties of the Dynamics instance being integrated with.
The crmsvcutil.exe provides a method for creating an OrganizationServiceContext which enables, among other things, access to the Microsoft Dynamics CRM Linq provider. Full details of the OrganizationServiceContext are available here: https://docs.microsoft.com/en-us/dotnet/api/microsoft.xrm.sdk.client.organizationservicecontext?view=Dynamics-general-ce-9.
There are a number of properties which can be used when calling crmsvcutil.exe and these were the ones used in this project:
crmsvcutil.exe /url:<url> /out:<file> /codewriterfilter:<type>,<object> /password:<password> /username:<username> /serviceContextName:<context>
- /url:<url> - this is the endpoint for the Organization.svc
- /out:<file> - this specifies the file name for storing the Dynamics models in
- /codewriterfilter:<type>,<object> - this allows specific objects to be written to the Dynamics models. Without this the <file> can be very large. For more information on implementing this see: http://erikpool.blogspot.com/2011/03/filtering-generated-entities-with.html
- /password:<password> - service password
- /username:<username> - service username
- /serviceContextName:<context> - OrganisationServiceContext name, used just like a database context
A full list of parameters for crmsvcutil.exe can be found here: https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/org-service/extend-code-generation-tool.
Copy the cs file generated to a new project. In this case, we used Dynamics.Domain. By using a separate project, not only are you following DDD principles, but you will avoid potential name conflicts, where the model contains the same names as objects already created/referenced in the existing solution.
Then add a reference to your Dynamics project.
If the connection to Dynamics is via OrganizationServiceProxy then the following line of code will need to be added at the point the connection is retrieved:
proxy.EnableProxyTypes();
If connection is via CrmServiceClient no further amends are required.
Now you can build a model as normal:
var invoice = new Invoice{
Name = "Web order”,
CustomerId = new EntityReference("contact", customerId)
};The above code shows how to create a reference between the invoice object and the customer object.
To create an object on the service use:
connection.Create(invoice);
connection.Update(invoice);
connection.Delete(invoice);To query the service:
var context = new <context>(proxy);
var invoice = context.InvoiceSet.FirstOrDefault(x => x.InvoiceId == DynamicsId);Where <context> is the name of the context used when calling crmsvcutil.exe and proxy is defined as above.
The OrganizationServiceContext <context>
Caution 1: crmsvcutil.exe And Option Sets
Just as Umbraco has drop-down lists as picklists, Dynamics has option sets.
The crmsvcutil.exe will map option sets as enums:
public enum languages
{
[System.Runtime.Serialization.EnumMemberAttribute()]
English = 857250000,
[System.Runtime.Serialization.EnumMemberAttribute()]
French = 857250001,
[System.Runtime.Serialization.EnumMemberAttribute()]
Spanish = 857250002,
[System.Runtime.Serialization.EnumMemberAttribute()]
German = 857250003,
[System.Runtime.Serialization.EnumMemberAttribute()]
Italian = 857250004,
[System.Runtime.Serialization.EnumMemberAttribute()]
MandarinChinese = 857250005,
[System.Runtime.Serialization.EnumMemberAttribute()]
Russian = 857250006,
[System.Runtime.Serialization.EnumMemberAttribute()]
Portuguese = 857250007,
[System.Runtime.Serialization.EnumMemberAttribute()]
Hindi = 857250008,
[System.Runtime.Serialization.EnumMemberAttribute()]
Arabic = 857250009,
}Because option sets can hold different values in different instances of Dynamics, when running the same code against different instances, do not trust the enum value without verifying it is the same on the different instances. Without access to the database, a quick way to do this is to run crmsvcutil.exe against each instance and compare the values that will be used in the code.
Further reading: https://develop1.net/public/post/2013/09/09/Do-you-understand-MergeOptions.
Setting The State
And finally, it’s not possible to update the state of an entity at the same time as you update/create the entity. This is done via a separated SetStateRequest call:
var savedinvoice = context.InvoiceSet.FirstOrDefault(x => x.InvoiceId == DynamicsId);
var request = new SetStateRequest
{
EntityMoniker = new EntityReference
{
LogicalName = savedinvoice.LogicalName,
Id = savedinvoice.Id
},
Status = optionSet,
State = new OptionSetValue(Convert.ToInt32(InvoiceState.Active))
};
context.Execute(request);Summary
When integrating with Dynamics, business processes and error handling should be considered just as much as the code required to integrate.
The Microsoft.CRM.Sdk provides a method for querying and returning objects using FetchXML or via Early Bound objects using the OrganizationServiceContext.
The OrganizationServiceContext will allow bulk processing of data and will track entity state.
The OrganizationServiceProxy/CrmServiceClient will only allow individual service calls but is good to use when you are only maintaining a few non-complex records and want to reduce the overhead on the service.
Note
Earlier versions of Dynamics required the context to have the object added and saved:
var account = new Account
{
AccountId = Guid.NewGuid(),
Name = "Test account",
};
ctx.AddObject(account);
ctx.SaveChanges();https://pederwagner.wordpress.com/2015/06/29/working-with-the-organizationservicecontext/
Useful Links
More information on configuring log 4 net can be found here: https://our.umbraco.com/Documentation/Getting-Started/Code/Debugging/Logging/index-v7.
Umbraco 8 uses SeriLog, more information on configuring logging for SeriLog can be found here: https://our.umbraco.com/Documentation/Getting-Started/Code/Debugging/Logging/.
More about Odata can be found here: https://www.odata.org/.