Defining Advanced Execution Options

The Advanced Execution Migration options are presented like this:

Migration Chunk Size

The migration chunk size defines the number of records that should be retrieved from the source server for processing before continuing with the transformation and migration of these records.  Once one chunk of records has been migrated (including attachments) The next chunk is retrieved and processed and so on until all records matching the source filter have been processed.

If the chunk size is set too high then there is a possibility that the machine where Precision Bridge is installed could run out of memory before the chunk processing is completed.  This would only occur if the records were especially large (for example if they had a large number of attachments) and the available memory on the host machine was low, (the recommended minimum is 8GB RAM).  In this case, the chunk size might need to be reduced below the default of 10,000 records.

Note that the chunk size can also be set on individual form mappings where necessary (see Form Mapping Options Tab).  The form mapping chunk-size setting takes precedence over the Execution chunk size setting defined here.

NOTE: This option will only be applied to form mappings where an inclusion filter HAS NOT been used.

Migration Inclusion Chunk Size

The Migration Inclusion Chunk Size defines the number of 'parent' records that 'child' records will be retrieved for before continuing with the transformation and migration of the retrieved records. Once one chunk of records has been migrated (including attachments) The next chunk is retrieved and processed and so on until all records matching the source filter have been processed. See the example below for clarification.

Care should be taken in making this value too high, though if the average number of child records per parent is very high this value should be reduced from its default 1000 to avoid memory issues.

As for the Migration Chunk Size above, this can be set on specific form mappings as well, with the form mapping value taking precedence over the execution options value.

The default value is 1000.

NOTE: This option will only be applied to form mappings where an inclusion filter HAS been used.

Chunking Example

The requirement is to migrate 5000 incidents (the parent) and all the Journal Entries for these incidents (the children). There are an estimated average of 10 journal entries per Incident. The project is configured to use an inclusion rule to ensure that only the journal entries for the migrated incidents are processed. The Migration Chunk Size is set to 1000 and The Migration Inclusion Chunk Size is set to 500.

At execution time, the Incidents are migrated in 5 chunks of 1000 records each.

Then, the Journal Entries are migrated in 10 chunks (each chunk processing the Journal entries for 500 records). Approximately 5000 (500x10) Journal Entries will be migrated in each chunk.

Verify Source IDs

This option only applies to migrations with a Service Now source server.

Performs additional verification to ensure all source records are retrieved when migrating from an active source server where source records could be created, updated or deleted during the migration leading to potentially missed records.  

If this option is selected, any missed records are captured and migrated at the end of the execution run.

This option requires additional memory during processing.

Reload Cache on Execution

If this option is checked, server information such as form/field properties that were retrieved at design time, are re-loaded.  This will slow the initialisation stage of the migration but may be required if changes have been made to form/field definitions on the server since the project was loaded.

Retrieval Batch Size

The Retrieval Batch Size defines the maximum number of records that can be retrieved from the source server in any single call.  On some servers, a limit is placed on the number of records that may be retrieved in a single call, and errors can occur if this is breached.

Also, with very large records, timeouts and other issues can occur during data retrieval. On the other hand, sometimes you may have minimal records and can improve performance by increasing the retrieval size without issue.  If left blank, the maximum retrieval size will depend on the source server type and will be 1000 records (Database, Remedy and Salesforce), 500 (ServiceNow and RSAArcher) and 100 for Jira and HP Service Manager.

Note that the max data retrieval size can also be set on individual form mappings where necessary (see Form Mapping Options Tab).  The form mapping setting takes precedence over the Execution setting defined here.

Submit Batch Size

This option only applies to migrations that support Batch Insert/Update operations (for example service Now).

The Batch Submit Size defines how many records are submitted for create or update when processing in a batch.

If using a single thread, the batch size may be increased dynamically during the migration to improve performance. For example, if you configure a batch size of 100, it may increase to as high as 500.

When using multiple threads for submitting data to the target server, the configured batch size remains persistent.

Source Delete Batch Size

This option only applies to migrations that support source deletion operations (for example Service Now -> Database).

The Source Delete Batch Size defines how many records are processed for source deletion at a time. The source deletion process involves validation checked on both the source and target server as well as the delete operation itself.

Setting this too high can result in timeouts when the records are deleted since often the target server also performs cascade deletions. The maximum value that can be set is 100.

To improve performance, it is usually the case that increasing the number of threads for source deletion will be more effective than increasing the batch size.

Import Set and Scripted Web Service Application

This option only applies to Service Now Target migrations.

This is the application where any referenced import sets should reside.  If they do not exist, they will be created in this application, provided the ‘Allow Import Set Web Service Synchronisation’ option is enabled.

Select Target Application

This option only applies to Service Now Target migrations.

Sets the application scope for all records created on the target server.  The application scope is set (for the admin user) when the migration starts and is reset to the original value after the migration completes.

Note for Select Target Application, the Application Scope is set for the login ID being used to connect to the target server. Note that this may impact existing sessions for the same login id, therefore it is recommended to ensure no other user or process is connecting with this login during migration.

Allow Import Set Web Service Synchronisation

This option only applies to Service Now Target migrations.

If selected, Precision Bridge will automatically create the required Import Sets, if either the ‘Import Set (Field Mappings)’ or ‘Import Set (Scripted)’ migration methods are also selected (see previous article).

If the Import Set already exists, it will be synchronised with any custom fields added to the corresponding table mapping.

If this option is de-selected and an Import Set is not found at execution time then a fatal error will be reported.

Force Import Set Synchronisation

This option only applies to Service Now Target migrations.

If this option is checked, the import sets and transform maps that Precision Bridge uses will always be updated on the target server during the initialisation of the execution. This option should not generally be required, since a check is made during initialisation of all transform maps used in execution to see if synchronisation might be required. If it is, the synchronisation will be performed for those import sets/ transform maps as needed.

This option can be checked if, for some reason, the earlier check fails to trigger an update following changes to the fields on the target form. If checked, the load/initialisation of the project for execution will take longer.