Scheduling Data Syncs

This article related to the configuration and implementation of data-syncs between ServiceNow instances.

To schedule a data sync, first create a project in Precision Bridge which includes all the mappings for the application tables that you want to sync.  For OOTB applications you can use the templates provided or if you have custom tables (or a lot of custom fields), you can use the Generate Form Mappings wizard to auto-create the required mappings.

Once the mappings are created, check to make sure that the source filtering matches your requirements for record selection. You can check the source filtering setting under the Source Filtering tab on each of the form mappings.  In most cases the filter query should be set to:

[sys_updated_on] > #LAST_EXECUTION_TIMESTAMP#

If the table does not have the [sys_updated_on] field (for example sys_audit, sys_journal_field) this indicates that records on this table are only created and never updated. Use this filter query instead:

[sys_created_on] > #LAST_EXECUTION_TIMESTAMP#

This filter will only select records that have been created or updated since the previous execution.

When you are happy with the mappings and filtering, select the option to include attachments if needed, under the Options tab for each form mapping.

Then select Execute Open Project from the Project menu and from the Scheduling tab, set the frequency for the data sync using the Repeat menu (from monthly down to 2 minutes):

Note for large volumes of data it is recommended that the execution should have time to complete syncing before the next execution is due to start.

For the initial execution, at execution time, on the Variable Values tab, set the LAST_EXECUTION_TIMESTAMP to the earliest date for which you want to retrieve records; it will then only select records created or updated since this date.   If there is a lot of data to migrate for the initial run it is recommended that this be executed without the repeat and that the repeat option only be used for subsequent executions.

You can then execute the project to initiate the syncing process.  Alternatively after saving the project, you can use the command line option to initiate the data sync by specifying the project name and frequency using the -srp option.  For more information on the command line configuration, see this article.

If the frequency is very high (< 10 minutes) we do not recommend using the Windows Task Manager to manage the sync because this will force the project to re-initialise each time it runs.  For high frequency syncs, use the command line -srp option instead to avoid this.

Bi-Directional Syncing

If you have a requirement to sync in both directions you must create a separate project to sync in the opposite direction.  Follow the steps outlined above to set this up.  

To avoid looping, the following configuration is required for both projects:

1. Create a ServiceNow user account with admin rights that can act as the migration user or use the account that is currently used to connect Precision Bridge to the target ServiceNow instance (pb user);

2. In each of the form mappings in your data-sync projects, set the sys_updated_by to the pb user.

3. In the source filtering for each mapping, add the condition:

AND [sys_updated_by] != 'pb_user'

where pb_user is the name of the account identified in step 1.

This configuration should prevent any looping but ensure that updated records by other users are still migrated.

Each project should be timed to execute with an offset so that they do not execute at the same time.  For example if both projects are set to sync every 10 minutes, set the 2nd project to start 5 minutes after the first.

Handling Conflicting Updates

In a bi-directional sync, it is important to consider how to handle the situation where the record is updated on both instances by different users, between a synchronisation. If this is a possibility, you might consider the following approach to prevent an earlier update overwriting a later one.

• On the form mapping, create a temporary variable of type String named LaterTargetUpdate
• Map this variable using a target lookup. The target lookup should retrieve the sys_id of the target record with both a matching sys_id and a sys_updated_on date that is equal or later than that of the source record.
• Create a second temporary variable named SkipIfLaterTargetUpdate. Map this using an assignment mapping and use the PERFORM_COND_ACTION function to skip the record if the variable LaterTargetUpdate is not equal to NULL
• The update will now be skipped if the target record was updated after the source record.