Amazon S3 Name and File Size Requirements for Inbound Data Files

Describes the required fields, syntax, naming conventions and file sizes you need to follow when sending data to Audience Manager. Set the names and sizes of your files according to these specifications when you send data to an Audience Manager / Amazon S3 directory.

Contents:

Note: The text styles (monospaced text, italics, brackets [ ] ( ), etc.) in this document indicate code elements and options. See Style Conventions for Code and Text Elements for more information.

File Name Syntax

S3 file names contain the following required and optional elements:

  • S3 prefix: s3n://AWS_directory/partner_name/date=yyyy-mm-dd/
  • File name elements: ftp_dpm_DPID[_DPID_TARGET_DATA_OWNER]_TIMESTAMP(.sync|.overwrite)[.SPLIT_NUMBER][.gz]

For other accepted file name formats, see Custom Partner Integrations.

Note: Audience Manager only processes ASCII and UTF-8 encoded files.

Name Elements

The table defines the elements in an S3 file name.

Name Element Description

AWS_directory

The path to and name of your Amazon S3 bucket. Contact your Account Manager for your S3 directory name, path, and credentials.

date=yyyy-mm-dd

A timestamp (based on UTC time) of when you send the files to your S3 bucket.

DPID

The Data Provider ID (DPID) is an identifier that tells Audience Manager if a data file contains your own user IDs or Android or iOS IDs. Accepts the following options:

Data Partner ID

This is a unique ID Audience Manager assigns to your company or organization. Use this assigned ID in a file name when sending in data that contains your own user IDs. For example, ...ftp_dpm_21_123456789.sync tells Audience Manager that a partner with ID 21 sent the file and it contains user IDs assigned by that partner.

Android IDs (GAID)

Use ID 20914 as the DPID in a data file name if the file contains Android IDs. When you use ID 20914 as the DPID, you still need to identify your company to Audience Manager. This means the file name must use the _DPID_TARGET_DATA_OWNER parameter to hold your company ID. For example, say you're passing in files with Android IDs and your Data Provider ID is 21. In this case, the file name would look like ...ftp_dpm_20914_21_123456789.sync. This tells Audience Manager the file contains Android IDs and is from a partner identified by ID 21.

iOS IDs (IDFA)

Use ID 20915 as the DPID in a data file name if the file contains iOS IDs. When you use ID 20915 as the DPID, you still need to identify your company to Audience Manager. This means the file name must use the _DPID_TARGET_DATA_OWNER parameter to hold your company ID. For example, say you're passing in files with Android IDs and your Data Provider ID is 21. In this case, the file name would look like ...ftp_dpm_20915_21_123456789.sync. This tells Audience Manager the file contains iOS IDs and is from a partner identified by ID 21.

Note: Do not mix ID types in your data files. For example, if your file name includes the Android identifier, don't put iOS IDs or your own IDs in the data file.

See also the _DPID_TARGET_DATA_OWNER entry below.

_DPID_TARGET_DATA_OWNER

A placeholder for an ID. For example, you could set it to your Audience Manager ID if you set the DPID to a data source ID or an Android or iOS ID. This lets Audience Manager link the file data back to your organization.

For example:

  • ...ftp_dpm_33_21_1234567890.sync shows a partner with ID 21 has sent in data from a data source that uses ID 33.
  • ...ftp_dpm_20914_21_1234567890.sync shows a partner with ID 21 has sent in data that contains Android IDs.
  • ...ftp_dpm_20915_21_1234567890.sync shows a partner with ID 21 has sent in data that contains iOS IDs.

parnter_name

The company or organization name you use in Audience Manager.

TIMESTAMP

A 10-digit, UTC UNIX timestamp in seconds. The timestamp helps make each file name unique.

(.sync|.overwrite)

Synchronization options that include:

  • sync: Normal scenario when third-party data providers send traits on a per-user basis to be added or removed in the Audience Manager system.
  • overwrite: Lets data providers send a list of traits on a per-user basis that should overwrite all of this user's existing third-party traits for this data provider in the Audience Manager. You do not need to include all of your users in an overwrite file. Include only those users that you want to change.

[SPLIT_NUMBER]

An integer. Used when you split large files into multiple parts to improve processing times. The number indicates which part of the original file you're sending in.

For efficient file processing, split your data files as indicated:

  • Uncompressed: 1 GB
  • Compressed: 200-300 MB

See the first 2 file name examples below.

[.gz]

When sending files to Amazon S3, use gzip compression only. When compressed, these files get the .gz extension. Do not use .zip compression.

Compressed files must be 3 GB or smaller. If your files files are larger, please talk to Customer Care. Although Audience Manager can handle large files, we may be able to help you reduce the size of your files and make data transfers more efficient. See File Compression for Inbound Data Transfer Files.

File Name Examples

The following examples show properly formatted file names. Your file names could look similar.

  • s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_1366545717.sync.1.gz
  • s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_1366545717.sync.2.gz
  • s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_1366545717.sync
  • s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_567_1366545717.sync.gz
  • s3n://<AWS_Bucket>/<partner_name>/date=2016-05-09/ftp_dpm_478_1366545717.overwrite

You can download the sample file if you want additional examples. This file has been saved with the .overwrite file extension. Open it with a simple text editor.

Accepted File Sizes

Consider the figures below for fastest/earliest processing of your files as well as for file size limitations when you send data to an Audience Manager / Amazon S3 directory.
File Type Optimal Size Maximum Size
Compressed

200-300 MB

3 GB

Uncompressed

1 GB

5 GB

Note: The inbound data validation process will mark empty files as invalid and will not process them.