Adding a Source and Destination Connection

In order to extract your data, you need to add a source and a destination connection.

  • Adding a source connection specifies the data source on which your data will be retrieved through the extraction process. Once the data is taken from the source connection, it needs a location to be stored, hence the destination connection.
  • Adding a destination connection specifies the place in which your data will be integrated in order to be read once transformed.

To add a source or a destination connection:

  1. Click the Connections icon on the left-hand side.

  2. Click New next to Source Connections or Destination Connections.

  3. Select the type of data source you want to use.

    Source connections

    Destination connections

    Active Directory

    Acumatica

    Custom API

    Flat File

    Google Analytics

    Google Sheets

    GraphQL

    HubSpot

    LDAP

     

     

    MongoDB

    Microsoft Access

    Microsoft Excel

    MySQL

    NetSuite

    OData

    ODBC

    Oracle

    PostgreSQL

     

     

    Quickbooks Desktop

    Quickbooks Online

    Sage Intacct

    Sage 50 UK

    Salesforce

    SAP HANA

    SQL Azure

    SQL RDS

    SQL Server

     

     

    Flat File

    Oracle

    SQL Azure

    SQL RDS

    SQL Server

    Note:  Versions prior to SQL Server 2008 R2 are not supported.

  4. The New Connection dialog box is displayed. Enter the parameters in the following panels: Connection Properties, Additional Connection Properties and Advanced Settings as described below.

  5. Click Save to finish.
Setting Description

Connection Properties

According to the connection selected, the fields will be different from one connection to another due to their characteristics.

For more information about each source connection type, see Source Connector Settings.

Tool tips located next to each field are available to help you setting up some fields. Feel free to take a look to get more details if needed.

Additional Connection Properties

This panel can be used to add specific properties to the connection string.

Add Property

Click this button if you want to specify some properties for this connection.

For each property you add, you can define if you want it to be encrypted or not (i.e. visible). If you choose Encrypted, the value of the property (ex: password) will not be shown on the screen and will be encrypted in the back end.

Property Enter a connection string which defines the action you want to perform (ex: READ).
Value

According to the property you entered, you can set the state (for example : TRUE or FALSE).

Encrypt

Select this option if you want to encrypt data related to the property.

Delete

Click on the X sign to delete the property.

Advanced Settings

This panel is available for some source connections.

Tracking Type

Date: the synchronization between the source and the destination will be incremental as it will be based on the latest update values of the DateTime entry of each row.

Note:  Sage X3 for instance used the field UPDDATTIM_0 as DateTime while in Sage 300 the DateTime is divided into 2 separate fields (AUDTDATE and AUDTTIME). For the case of Sage 300, you will have to create a calculated field first to merge these fields then to convert them into a "real" DateTime field in order to work properly.

Time Zone

(if you select Date as tracking type

Defines the region and time zone that will be used by the source connection.

Time Offset

(if you select Date as tracking type)

Adds an offset to the source connection that will be used for Incremental Load job (Minimal value is 0 and maximum 3600 seconds).

Batch Size

The batch size represents the quantity of data read, transform and load in the destination at the same time.

To set the value, remember that the bigger the batch size is the bigger the memory usage will be and the faster it will be up to an extend. The default value is 2000 and the maximum value should not be set to more than 10 000.

Based on your network speed and disk performance, you can experiment which size of block would be faster for you. Usually the default value (2000 records) works fine.

Note

If DataSync is not installed on the same server as your ERP application, please check that the minutes and seconds in the two servers match. Use the Time Offset feature (in seconds) if there is a difference. You can enter a number greater than the difference to ensure optimal data retrieval during an Incremental Load task.

Example

A client imports an Excel file that has a lot of transactions (1000 rows).

While processing the transactions with a job, a small time gap exists during the time period between when the job has finished submitting the rows and when the system has stamped the last success date (DateTime.Now (meaning the current time)).

During that time period, imagine that a few rows have been added (for example, 5 rows, so there are now 1005 rows) and the client does a refresh (Incremental Load). Since the refresh is based on the last success date, it will skip the 5 new rows that came after the job has finished but before the timestamp was done by the system.

Adding an offset to the connection can fix that problem because it removes the time gap.