File Transfer — Illustrated

How It Works

File transfers between Cardlytics and Publishers are performed via Amazon S3 Replication.

At a high level, CDLX will create two S3 buckets, one for incoming files and one for outgoing files:

Publishers can send us files by S3 Replication.
CDLX pushes files out for outgoing S3 buckets. The files are sent using S3 Replication API^{[^1]}

The above steps will be repeated for the Pre-Prod environment.

For reference: https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-replication.html

Setup Overview

Cardlytics Setup

Cardlytics will provide the Publisher with the following information:

AWS account number
- Referred to as CDLX_ACCOUNT_NUMBER in the code sections below
Outgoing S3 Bucket ARN. Used by Publisher to receive files from CDLX
- Referred to as CDLX_EGRESS_BUCKET_NAME in the code sections below
Incoming S3 Bucket ARN. Used by Publisher to send files to CDLX
- Referred to as CDLX_INGRESS_BUCKET_NAME in the code sections below
CDLX S3 Principal ARN. Used by Publisher to add to the Publisher-side bucket that we copy files to, from our Outgoing Files bucket
- Referred to as CDLX_IAM_ARN in the code sections below
CDLX KMS key ARN. Used by Publisher to send files with encryption in transit
- Referred to as CDLX_KMS_ARN in the code sections below

Publisher Setup

The Publisher will need to provide CDLX with the following information:

Publisher-side S3 Bucket ARN for Publisher to send files to CDLX
- See EGRESS_BUCKET_NAME in the code sections below
Publisher-side S3 Bucket ARN for Publisher to receive files from CDLX
- See INGRESS_BUCKET_NAME
Publisher-side IAM Principal. This Publisher-side Principal will transfer files into the CDLX Incoming bucket. CDLX will configure the S3 bucket to allow write access from this Publisher-side IAM principal
- See IAM_ROLE_ARN
Publisher-side KMS key ARN. Used by CDLX to send files with encryption in transit
- See KMS_ARN

Setup Instructions

Follow the instructions below to set up each resource. The Publisher will need to replace the references in the code with the resource information from CDLX, and with the resource information that the Publisher is creating.

KMS_ARN

Used by the Publisher to encrypt their objects at rest. Used by CDLX to send files with encryption in transit.

Setup configuration

Key type: symmetric
Key usage: encrypt and decrypt
Key policy:

{
    "Id": "ReplicationPolicy",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowUseOfKey",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "IAM_ROLE_ARN",
                    "arn:aws:iam::CDLX_ACCOUNT_NUMBER:root"
                ]
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "KMS_ARN"
        },
        {
            "Sid": "AllowAttachmentOfPersistentResources",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "IAM_ROLE_ARN",
                    "arn:aws:iam::CDLX_ACCOUNT_NUMBER:root"
                ]
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "KMS_ARN",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

EGRESS_BUCKET

Used by the Publisher to send files to CDLX.

Setup configuration

Bucket Versioning: enabled
Encryption: SSE-KMS (select your KMS key)
Bucket Key: enable
Edit bucket policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSSLRequestsOnly",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::EGRESS_BUCKET_NAME/*",
                "arn:aws:s3:::EGRESS_BUCKET_NAME"
            ],
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        }
    ]
}

INGRESS_BUCKET

Used by Publisher to receive files from CDLX

Setup configuration

Copy setting from outgoing bucket
Edit bucket policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SetPermissionsForObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "CDLX_IAM_ARN"
            },
            "Action": [
                "s3:ReplicateObject",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": "arn:aws:s3:::INGRESS_BUCKET_NAME/*"
        },
        {
            "Sid": "SetPermissionsOnBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "CDLX_IAM_ARN"
            },
            "Action": [
                "s3:GetBucketVersioning",
                "s3:PutBucketVersioning"
            ],
            "Resource": "arn:aws:s3:::INGRESS_BUCKET_NAME"
        },
        {
            "Sid": "AllowSSLRequestsOnly",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::INGRESS_BUCKET_NAME/*",
                "arn:aws:s3:::INGRESS_BUCKET_NAME"
            ],
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        }
    ]
}

IAM_ROLE_ARN

Used by the Publisher to replicate objects to CDLX

Setup configuration

Trust Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [  {
            "Sid": "AllowKMSKeyUsage",
            "Effect": "Allow",
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": [
                "CDLX_KMS_ARN"
            ]
       },
       {
            "Sid": "EgressBucketPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionForReplication",
                "s3:GetReplicationConfiguration",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:ReplicateObject",
                "s3:ReplicateTags",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": [
                "arn:aws:s3:::EGRESS_BUCKET_NAME/*",
                "arn:aws:s3:::EGRESS_BUCKET_NAME"
            ]
        },
        {
            "Sid": "ExternalBucketPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectVersionTagging",
                "s3:ReplicateObject",
                "s3:ReplicateTags",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": [
                "arn:aws:s3:::CDLX_INGRESS_BUCKET_NAME/*"
            ],
            "Condition": {
                "StringLikeIfExists": {
                    "s3:x-amz-server-side-encryption": [
                        "aws:kms"
                    ],
                    "s3:x-amz-server-side-encryption-aws-kms-key-id": [
                        "CDLX_KMS_ARN"
                    ]
                }
            }
        }
    ]
}

Egress Bucket Replication Rule

Used by the Publisher to automatically transfer files to CDLX

Setup configuration

Replication rule should be added to your EGRESS_BUCKET_NAME with the following settings;

Destination: specify a bucket in another account
- Account id: CDLX_ACCOUNT_NUMBER
- Bucket name: CDLX_INGRESS_BUCKET_NAME
- “Change object ownership to destination bucket owner”: enabled
IAM Role: IAM_ROLE_ARN (the IAM from your account)
Encryption: Replicate objects encrypted with AWS Key Management Service
- AWS KMS ARN: CDLX_KMS_ARN (the KMS we provide)