Configure Flat File External Source Settings

DataSync allows you to connect to local and remote CSV resources, including cloud storage providers and file servers. For each source type, you set the URI parameter to the CSV resource location, along with any required authentication properties.

Supported connection types

Connection type	Description
Basic Authentication	Connect to local folders, HTTP/HTTPS streams, SharePoint, FTP/FTPS, or SFTP.
Amazon S3	Connect to Amazon Simple Storage Service (S3) buckets and folders.
Azure Storage	Connect to Azure Blob Storage or Azure Data Lake Store (Gen1/Gen2).
Oracle Cloud Storage	Connect to Oracle Cloud Storage buckets and folders.
Wasabi	Connect to Wasabi cloud object storage.

Basic Authentication

Local file (URI format)

Property	Requirement
URI	Required — local folder path or file URI

1. Set URI to the folder containing CSV files.
2. Combine multiple CSVs with the same schema:

   URI=C:\folder; AggregateFiles=True;

3. List each file as a separate table:

   URI=C:\folder; AggregateFiles=False;

HTTP/HTTPS

Property	Requirement
URI	Required — http://remoteStream or https://remoteStream
User	Optional
Password	Optional
AuthScheme	Optional

1. Set URI to the HTTP or HTTPS URL of the CSV:

   URI=http://www.host1.com/streamname1;

2. Add AuthScheme and credentials if necessary. You can also set CustomHeaders or CustomUrlParams.
3. Query the streamed table:

   SELECT * FROM streamedtable;

SharePoint Online SOAP

Property	Requirement
URI	Required — sp://remotePath
User	Required
Password	Required

1. Set URI to the document library containing the CSV files.
2. Specify User, Password, and the full SharepointUrl to authenticate.

   URI=sp://Documents/folder1; User=user1; Password=password1; SharepointUrl=https://subdomain.sharepoint.com;

FTP/FTPS

Property	Requirement
URI	Required — ftp://server:port/remotePath or ftps://server:port/remotepath
User	Required
Password	Required

1. Set URI to the FTP/FTPS server and the folder path.
2. Provide User and Password for authentication:

   URI=ftps://localhost:990/folder1; User=user1; Password=password1;

SFTP

Property	Requirement
URI	Required — sftp://server:port/remotePath
User	Optional
Password	Optional

1. Set URI to the SFTP server and folder location.
2. Include User and Password if authentication is required:

   URI=sftp://server:22/folder1; User=user1; Password=password1;

Amazon S3

Property	Requirement
URI	Required — s3://bucket/folder/
AWSAccessKey	Required — your AWS Access Key (user name)
AWSSecretKey	Required — your AWS Secret Key
AWSRegion	Optional

1. Set URI to the S3 bucket and folder location:

   URI=s3://bucket1/folder1;

2. Provide AWSAccessKey and AWSSecretKey for authentication:

   URI=s3://bucket1/folder1; AWSAccessKey=token1; AWSSecretKey=secret1; AWSRegion=OHIO;

3. Optionally set AWSRegion to specify the region for the bucket.

Example — connecting to S3-compatible services

You can also connect to S3-compatible storage services by setting a custom base URL in the connection properties. Here is an example base URL settings

URL=http://s3.%region%.myservice.com
Region=region-1

With these settings, DataSync will generate request URLs in one of the following formats:

UseVirtualHosting	Generated URL format
False	https://s3.region-1.myservice.com/bucket/...
True	https://bucket.s3.region-1.myservice.com/...

Azure Storage

Azure Blob Storage

Property	Requirement
URI	Required — azureblob://mycontainer/myblob/
AzureAccount	Required
AzureAccessKey	Required — your Access Key (user name)

1. Set URI to the container and blob name.
2. Provide AzureAccount and AzureAccessKey for authentication:

   URI=azureblob://mycontainer/myblob/; AzureAccount=myAccount; AzureAccessKey=myKey;

Azure Data Lake Store Gen 1

Property	Requirement
URI	Required — adl://remotePath or adl://Account.azuredatalakestore.net@remotePath
InitiateOAuth	Required
OAuthClientId	Optional
OAuthClientSecret	Optional
OAuthAccessToken	Optional
AuthScheme	Required
AzureAccount	Required

1. Set URI to the Gen1 Data Lake path.
2. Configure OAuth parameters if required.
3. Specify AzureAccount and required authentication properties.

Azure Data Lake Store Gen 2

Property	Requirement
URI	Required — abfs://myfilesystem/remotePath
AzureAccount	Required
AzureAccessKey	Required — your Access Key (user name)

1. Set URI to the file system and folder containing your CSV files:

   URI=abfs://myfilesystem/folder1; AzureAccount=myAccount; AzureAccessKey=myKey;
   URI=abfss://myfilesystem/folder1; AzureAccount=myAccount; AzureAccessKey=myKey;

Azure Data Lake Store Gen2 with SSL

Property	Requirement
URI	Required — abfss://myfilesystem/remotePath
AzureAccount	Required
AzureAccessKey	Required — your Access Key (user name)

1. Set URI to the file system and folder location using the secure abfss:// protocol.
2. Provide AzureAccount and AzureAccessKey for authentication.

Oracle Cloud Storage

Property	Requirement
URI	Required — os://bucket/remotePath
AccessKey	Required — your Access Key (user name)
SecretKey	Required — your Secret Key
Region	Optional
OracleNamespace	Required

1. Set URI to the Oracle Cloud bucket and folder.
2. Provide AccessKey, SecretKey, and OracleNamespace for authentication.
3. Optionally specify Region:

   URI=os://bucket/remotePath/; AccessKey=token1; SecretKey=secret1; OracleNamespace=myNamespace; Region=us-ashburn-1;

Wasabi

Property	Requirement
URI	Required — wasabi://bucket1/remotePath
AccessKey	Required — your Access Key (user name)
SecretKey	Required — your Secret Key
Region	Optional

1. Set URI to the Wasabi bucket and folder.
2. Provide AWSAccessKey and AWSSecretKey for authentication.
3. Optionally specify AWSRegion:

   URI=wasabi://bucket1/folder1; AWSAccessKey=token1; AWSSecretKey=secret1; AWSRegion=OHIO;