File Transfer Implementation (S3)

Unified Guide for KMS and SSE-S3 Encryption

File Transfer — Illustrated


This guide outlines how file transfers between Cardlytics and Publishers are implemented using Amazon S3 Replication, supporting both AWS KMS (Key Management Service) and SSE-S3 (Amazon S3-managed keys) encryption.



How It Works

File transfers are performed via Amazon S3 Replication:

  • Publishers send files to CDLX: via replication from Publisher’s S3 bucket to CDLX’s Source bucket.
  • CDLX sends files to Publishers: via replication from CDLX’s Destination bucket to Publisher’s Source bucket.

Separate setups are repeated for Pre-Prod and Prod environments.

For reference: AWS S3 Replication Documentation


Encryption Options

  • KMS Encryption (SSE-KMS): Used when customer-managed encryption keys are required.
  • SSE-S3 Encryption: Used for simplified setup without customer-managed keys.

Setup Overview

Cardlytics Will Provide:

  • AWS Account Number
  • Incoming S3 Bucket ARN (CDLX_Source_BUCKET_NAME)
  • Outgoing S3 Bucket ARN (CDLX_Destination_BUCKET_NAME)
  • CDLX IAM Principal ARN (CDLX_IAM_ARN)
  • CDLX KMS Key ARN (CDLX_KMS_ARN)KMS encryption only

Publisher Will Provide:

  • Publisher Source S3 Bucket ARN
  • Publisher Destination S3 Bucket ARN
  • IAM Role ARN (IAM_ROLE_ARN) for replication
  • Publisher KMS Key ARN (KMS_ARN)KMS encryption only

Bucket Configuration

Destination_BUCKET (Publisher → CDLX)

  • Versioning: Enabled
  • Encryption: SSE-KMS (provide KMS_ARN) or SSE-S3
  • Bucket Policy: Enforce HTTPS only
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSSLRequestsOnly",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::Destination_BUCKET_NAME/*",
        "arn:aws:s3:::Destination_BUCKET_NAME"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}

Source_BUCKET (CDLX → Publisher)

  • Copy configuration from Destination_BUCKET
  • Additional Bucket Policy for replication:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SetPermissionsForObjects",
      "Effect": "Allow",
      "Principal": {"AWS": "CDLX_IAM_ARN"},
      "Action": ["s3:ReplicateObject", "s3:ObjectOwnerOverrideToBucketOwner"],
      "Resource": "arn:aws:s3:::Source_BUCKET_NAME/*"
    },
    {
      "Sid": "SetPermissionsOnBucket",
      "Effect": "Allow",
      "Principal": {"AWS": "CDLX_IAM_ARN"},
      "Action": ["s3:GetBucketVersioning", "s3:PutBucketVersioning"],
      "Resource": "arn:aws:s3:::Source_BUCKET_NAME"
    }
  ]
}

IAM Role for Replication

Trust Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "s3.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}

IAM Policy

  • For KMS encryption:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowKMSKeyUsage",
      "Effect": "Allow",
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey"],
      "Resource": ["CDLX_KMS_ARN"]
    },
    {
      "Sid": "DestinationBucketPermissions",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetObjectVersionTagging", "s3:GetObjectVersionAcl", "s3:GetObjectVersionForReplication", "s3:GetReplicationConfiguration", "s3:GetObjectRetention", "s3:GetObjectLegalHold", "s3:ReplicateObject", "s3:ReplicateTags", "s3:ObjectOwnerOverrideToBucketOwner"],
      "Resource": ["arn:aws:s3:::Destination_BUCKET_NAME/*", "arn:aws:s3:::Destination_BUCKET_NAME"]
    },
    {
      "Sid": "ExternalBucketPermissions",
      "Effect": "Allow",
      "Action": ["s3:GetObjectVersionTagging", "s3:ReplicateObject", "s3:ReplicateTags", "s3:ObjectOwnerOverrideToBucketOwner"],
      "Resource": ["arn:aws:s3:::CDLX_Source_BUCKET_NAME/*"],
      "Condition": {
        "StringLikeIfExists": {
          "s3:x-amz-server-side-encryption": ["aws:kms"],
          "s3:x-amz-server-side-encryption-aws-kms-key-id": ["CDLX_KMS_ARN"]
        }
      }
    }
  ]
}
  • For SSE-S3 encryption: Same as above but remove all KMS-related statements and conditions.

Replication Rule (Destination Bucket)

Configure replication from Publisher’s Destination_BUCKET_NAME to CDLX_Source_BUCKET_NAME:

  • Destination: CDLX_Source_BUCKET_NAME
  • Account ID: CDLX_ACCOUNT_NUMBER
  • Change object ownership: Enabled
  • IAM Role: IAM_ROLE_ARN
  • Encryption:
    • For KMS: select "Replicate objects encrypted with AWS KMS" and provide CDLX_KMS_ARN
    • For SSE-S3: no action required

File Path Structure

Use the following format for file naming:
{Root}/{YYYY-MM-DD}/CDLX_<file-type>_<date>_<epoch>_1of2.jsonl


Summary

  • KMS Encryption offers fine-grained control and secure key management.
  • SSE-S3 simplifies the setup by removing the need to manage KMS keys.
  • Both approaches use S3 Replication with enforced SecureTransport and object ownership transfer to ensure secure, automated file delivery between Publisher and Cardlytics.