Defining the Set of Source Records to Migrate

Project Variables

Before we look at the options for defining which records get migrated, we shall look at Project Variables. These allow you to define flexible filtering for multiple form mappings, enabling the set of records to easily be changed where the project will be executed multiple times.

For example, let us look at a use case where a migration is being performed on a pre-company basis. So all the data for one company will be migrated and validated before moving on to another company.

In this situation, it would be advised to create a project variable, holding the name of the company for which data will be migrated.

To create and edit project variables, use Project->Variables to open the Project Variable editing dialog

Precision Bridge provides a 'special' project variable, LAST_EXECUTION_TIMESTAMP that is of particular use in synchronisation projects, or any situations where a 'delta' migration needs to be run to synchronise the target with any changes made on the source since the last migration. The value of this variable will be updated each time the project runs, allowing source filter conditions to be defined that only migrate records created or updated since the last execution.

Defining Source Filters

These are defined on the Source Filter tab of the Form Mapping.

A source filter is a condition applied when the source data is extracted that restricts the records that are included based on a query (or filter) This filter can include Source fields, constant values, relational operators (=, !=, <, > etc), Project Variables and conditional operators (ADD, OR)

The filter is a string value, usually written in the Precision Bridge generic query format. However, for many Adaptor Types it is possible to use a query written in the native query language for the server.

For more information on constructing source queries, see: Creating a Source Filter Query

Inclusion Filtering - Value in List

Inclusion filters are intended to allow records related to some 'parent record' to be migrated for each parent record that gets migrated. In most cases, the native query language does not support this, so Precision Bridge includes the concept of an 'Inclusion Filter'. This feature is supported by the majority of adaptors.

Let's look at an example to clarify how this can be used. Our use case requires us to migrate all 'active' incidents for a given company. So far so good, we can easily put together a source filter to do this. However, the requirement is to also migrate all the tasks related to these incidents. We can add a second form mapping to migrate the tasks, but unless we add a source filter, all tasks will be migrated, regardless of whether their parent incident was. Tasks contain a reference to their parent incident, so it would be possible to get the set of tasks for a single incident. However, it is not possible in most adaptors to get the tasks for a set of incidents using a query string.

This is the situation that inclusion filters were made for.

Let's look at how filtering requirement might be met using an inclusion filter in the Incident Tasks mapping.

It is worth noting, that if an incident fails to migrate for some reason, then its ID will not be included in the list of IDs used by the inclusion filter so the incident's tasks will also NOT be migrated.

Mor information on Inclusion Filtering can be found here: Filtering By Inclusion Using a Previous Migration Set. It is also possible to define a query to get the set of values that the source field must have for the record to be migrated. See Filtering By Inclusion Using an Ad-Hoc Query.

Execution Time Setting of Variables

There will be more information about the settings that you can specify at execution time later in this section, but Variable Values are an important part of these settings. For each Execution, you should check that the variables have the correct values to define the set of records you want to migrate.

Note that when the project has been executed, the variable values are saved for the next time. So, you should make it a habit to check these are correct before every execution.