ServiceNow Instance Synchronisation: Migrating Related Tickets Together

Scope

This article applies to ServiceNow instance migration. It details how to approach a common requirement, which is how to develop a project that will migrate a specified set of one type of ticket, but to also include all the tickets related to it. For example, at the top level, we are migrating a set of incidents. However, we also want to migrate any Problems and Change Requests associated with these incidents.

The assumption will be that all foundation, CI and other referenced data has already been migrated correctly. If this is not the case, you are advised to look at synchronisation of this data first, particularly (though not exclusively) users, groups, companies, CI's and services.

Identify Requirements

The approach to this use case will be different depending on the requirements you have, the amount of data being migrated each time the project is run, the amount of data existing on the target already and whether the intention is to run this project repeatedly and only pick up records changed and added since the last migration.

Create the Projects Individually for Each Ticket Type

The basic approach will be the same for all projects.

Identify the top level ticket type, for which you want to include data for. In this example this will be 'Incident'
Identify the related ticket types, and also identify how they are related. In this example, we will include related change requests, related problems and child incidents.

As always, it is recommended that you start with a template. In our case, we will be making use of the Incident, Problem and Change Request templates.

Create a project for each of these templates. All of these templates are constructed to allow migration of a set of tickets (defined by a source query) and multiple related records such as journal entries, affected CI's, tasks, sla tasks etc.

For each of these projects, adjust the template to fit your systems. The things you will need to look at are:

Removing Form mappings that you do not need in your use case (for example, remove those related to workflow if these are not required).
Identifying any situations where referenced foundation data or CI's do not or may not have the same sys_id on source and target. If this is found to be the case, the recommended approach is to use a reference mapping or ID List mapping to map these fields.
Identify fields that have not been mapped by the template, particularly custom fields, but there may be OTB fields as well.
Run the validation report for the project. This will identify any missing fields. If missing target fields are found, you should remove the mapping for them. If missing source fields are found, you might also remove the affected field mappings, or you might elect to set the target to a default value.
Test the projects on a smaller set of data, by defining a source filter on the top-level task form mapping only (e.g. incident, problem or change request) only. All the other form mappings are defined as 'children; of this one so there is no need to change the source filtering for these.

Once the above is done, you should have a working project for Incident, Problem and Change Request.

Linking the Projects together

Now there is a choice to make. Should you join these projects together to run as a whole or run them individually in sequence? The choice you make here will define how you ultimately set the source filtering for each task type.

You should consider joining the projects together if all the following are true, since this is usually simpler to manage:

You have removed a large number of the mappings for each project (particularly the workflow mappings)
The total number of tickets to be migrated in one execution if the project will be relatively small (<100k incidents, problems and changes combined).

Otherwise, it is sometimes best to keep the project separate and execute them in the correct order.

The next 2 sections describe both options.

Option 1 - One Single Project:

Create a duplicate of the the top level project (e.g. incident) using the Save As function. We will be adding the mappings to the other related projects to this one.

Use Project-> Import Form Mappings to import the mappings for Problem and Change Request to the current project. A few notes on this:

You do NOT need to migrate the disabled reference form mappings as long as you already have a similar mapping on the current project. For example, in the Incident template there is a mapping 'User Reference Mapping' which defines the correlation to be used when migrating user references. The Problem template has the same reference mapping. In this case, there is no need to migrate the User Reference Form Mapping from Problem. However, if you had a reference form mapping on Problem that was not defined on Incident, you would need to include it. Best practice is to add disabled reference mappings at the top of the projects form mapping set.
If there is an identifier clash between an existing form mapping and one being imported a suffix will be added to the existing one. It is recommended that after importing the mappings, you go through all form mappings and update the identifier to give clarity in what each does.
If you want to include child incidents, you can import form mappings from the original Incident project project. Of course in this case, you will get identifier clashes for all form mappings, so should update the identifiers to clarify what each one is for. For example, the imported form mapping for incident might be imported as Task: Incident_001. You could change the Identifier to Child Incidents.
Run project validation after each import and fix issues reported.

You should now have all the form mappings that you need. Now you need to link them together to ensure that only changes, problems and child incident related to the parent incident are migrated.

This will use an inclusion Filter. The example below looks at Change requests related to Incidents.

In this case, there is a field on the incident 'Change Request' that identifies the related change request. We can define an inclusion filter that will:

Collate a list of all the change request sys_id's for the incidents that get successfully migrated
Use this list to only migrate change requests with a sys_id in this list.

We want to include all change requests that have a sys_id that matches one of those.

This Inclusion Filter (On the Change Request mapping) will look like this:

Note that any source filter query should be removed. Include ALL change requests related to the migrated incidents.

For Child incidents, a similar mapping can be used:

Option 2 - Separate Projects:

This option is a little more tricky to manage. Firstly, you need to ensure that when migrating the projects, the top level Incident records get migrated first. We will look at how to migrate the set of related change requests.

Since these projects are separate, the Change Request project is not aware of the set of incidents previously migrated. We therefore need to construct this list using a query. In Precision Bridge, this is termed as using an 'Ad-hoc' query inclusion filter to define the set of parent incidents.

This again uses the inclusion filter function, but this time the list of related change requests will come from a query on the source or target incident table. It is usually safest to query the target, since this way only incidents that already exist on the target can be included in the list.

The query that is used should 'match' that used initially to migrate incidents. This means that is you change the incident query, you will need to make the same change to the query used for change requests.

In our example, let's say we want to migrate closed incidents that were last updated on or after a given date/time.

Our query for incident migration might look like this:

In the change request project, we should set the inclusion filter like this:

For this to work correctly, the important thing to note is that the adhoc query MUST match the source filter used in the incident project. If you have used variables (as in the example above) they must be given the same value when executing each project.

Note that for Child incidents, you can duplicate the incident project and simply change the query to use an Adhoc Target Query inclusion filter similar to that below (remember to remove the source filter query):

Other Considerations - Delta Projects

If you want to define a project that only migrates records changed since the last execution, a different and more complex approach, since it might be possible that a change request got updated since the last execution, but the top level incident was not. In this situation, you might consider the following approach:

Removing the inclusion filter from the change request
Adding a query to the change request to only migrate records updated since last execution (using the LAST_EXACUTION_TIMESTAMP variable)
Adding an existence check variable in the change request form mapping. This would be set using a lookup onto the target incident to ensure that at least one incident references this change request. The lookup mapping should be set to 'skip' if no records are found.

Note that for delta projects, the source filters for related records in general may no longer be appropriate if a related record can be changed or added without the top level project changing. A similar approach to that outlined above can be used here too, to ensure that only related records that relate to an existing 'parent' get migrated, thus avoiding migrating orphan related records.

Importing or Copying a Form Mapping

Project Variables Overview

Filtering By Inclusion Using a Previous Migration Set

Filtering By Inclusion Using an Ad-Hoc Query

Reference Form Mappings - Overview and Worked Example

Changing the Mapping Identifier