Build your own CDP: Activating audiences in Google Ads Customer Match using Google Tentacles

In this tutorial we will show you how to activate audiences in Google Ads Customer Match lists using the Google Tentacles framework. The framework can be used activate your (first-party) data in marketing and advertising platforms. Basically building a headless Customer Data Platform (CDP) within an (existing) cloud / data environment without an expensive CDP tool or suite

• Automate the audience push to Google Ads (Customer Match lists), based on (hashed) email address or other available identifiers.
• Google Tentacles runs "serverless" within the Google Cloud Platform and starts running when a new file is uploaded in a specific Storage bucket.
• These files are new line delimited JSONs, containing the create and remove to audience operations. These files can be generated by your data warehouse (e.g. Google BigQuery or Dataflow).
• Can be used for a lot of common use cases (pushing audiences or conversion events to external platforms).
• The framework itself is focused on Google products (what a suprise!), but connectors for non-Google products can be developed easily (e.g. Facebook Conversion API or Custom Audience connector).

What is Tentacles?

Tentacles is an (unofficial) Google framework, operating serverless using different services within the Google Cloud Platform, like Cloud Functions & PubSub. The integration framework is able to handle large amounts of data to be send to other, mostly marketing related Google products (like Google Analytics data upload, Google Campaign conversions or Google Ads Custom Audiences). Next to all the Google related connectors, the framework is handling tasks like queueing, batching, rate-limits and (error) logging and therefore can be used perfectly for building custom connectors, like a Facebook Custom Audience connector we've build.

More background information on the framework in this article.

Architectural layout of the serverless integration framework. Source: Google

1. Preparation

If not present already, create a Google Cloud project (instructions) where you can run Tentacles. Tentacles provides you with a bash install script, that will check and activate all services / APIs needed. It assumes you're an project administrator, so if it's not possible to grant administrator rights, at least you need the following permissions / roles;

• Service Account User
• Cloud Functions Developer
• Pub/Sub Editor
• Storage Admin
• Service Usage Admin
• Cloud Datastore User
• Service Management Administrator

The following services / APIs are used by Tentacles, if not activated already, the install script will enable it for you;

In new GCP projects, you have to select a Datastore / Firestore operation mode. For Tentacles, choose the Firestore Native mode. You can't change this setting later on, so make sure you set it to Firestore Native mode.

• Google Ads API (you have to enable this manually in your project)
• Google Cloud Storage
• Firestore (Native mode)
• Google Cloud Functions
• Google Pub/Sub

Create an OAuth Client

1. In GCP navigate to Google APIs & Services > Credentials
2. If not present already, configure the consent screen and fill out the required fields.
3. Create new OAuth 2.0 Client credentials. When creating, select the Desktop app Create )Desktop app
4. Write down the Client-ID and Client-Secret, you will need it during the installation.

Google Ads API basic access

In Google Ads, create an API developer token and apply for API basic access. This will take some days. See documention

2. Installation / deployment

Let's move on the deployment of Tentacles.

1. In your Google Cloud project, open up the Cloud shell (top right).

2. Clone the repository to your Cloud shell instance;

1git clone https://github.com/GoogleCloudPlatform/cloud-for-marketing.git

3. Run the install script and follow the instructions.

1cd cloud-for-marketing/marketing-analytics/activation/gmp-googleads-connector
2bash deploy.sh

• The namespace chosen will be the prefix of all services created by Tentacles
• When asked for the APIs (connectors) to install, choose "Google Ads API for Customer Match Upload" (option 5).
• Enter the Client-ID and Cliend-Secret you've created in the earlier steps, this will generate a verification link that results in a response code you need to paste back into the install script.

The install script will enable all the services if not already, creating the Pub/Sub topics, Cloud Functions, Cloud bucket and triggers. Next step is to configure the Google Ads Customer Match connector.

3. Configure the Google Ads connector

The Google Ads connector uses the Google Ads API (not the old Google Adwords API). It expects;

• Two separate JSON files per audience, co (per audience), with one of the available customer identifiers, like (hashed) email or telephone. See this page for all available identifiers.
• One file containing the create (add) to audience operations
• One file containining the remove from audience operations

Be aware that you're uploading customer data to Google, so make sure you have the correct consent / permissions from those users.

Connector configuration

Every Google Ads audience needs to be defined in the config_api.json file. For the same audience you have to create 2 configurations, one for create operators and one for remove operators.

As I'm told, this is because Tentacles mimics the behavior of the Google Ads API. The Google Ads API separates create and delete operations. Next to that, using multiple identifiers (like hashed email and phone) also results in additional configurations, because the Google Ads API also separates these operations.

Example config_api.json file.

ACM: Ads Customer Match configurations (the config_api.json can also contain configurations for the other connectors

1{
2  "ACM": {
3    "audience_1_add": {
4        "developerToken": "<GOOGLE-ADS-API-DEVELOPER TOKEN>",
5        "customerMatchConfig": {
6            "customer_id": "<ADS-CUSTOMER-ID>",
7            "login_customer_id": "<ADS-MCC-CUSTOMER-ID>",
8            "list_id": "<CUSTOM-AUDIENCE-ID>",
9            "list_type": "hashed_email",
10            "operation": "create"
11        }
12    },
13    "audience_1_remove": {
14        "developerToken": "<GOOGLE-ADS-API-DEVELOPER TOKEN>",
15        "customerMatchConfig": {
16            "customer_id": "<ADS-CUSTOMER-ID>",
17            "login_customer_id": "<ADS-MCC-CUSTOMER-ID>",
18            "list_id": "<CUSTOM-AUDIENCE-ID>",
19            "list_type": "hashed_email",
20            "operation": "remove"
21        }
22    }
23  }
24}

1. Go back to the Google Cloud shell.
2. You can use the file editor (Open Editor button), navigate to cloud-for-marketing/marketing-analytics/activation/gmp-googleads-connector
3. Open config_api.json and edit the contents / paste your Google Ads Custom Audience configuration.

Then, run the following command to upload or update the config into Tentacles (the bash script will load or update the configuration into the Firestore database).

1bash deploy.sh update_api_config

The install script doesn't remove entries in the Firestore database, only adds or updates.

Now, Tentacles is all set to receive the JSON files containing the create and remove operations for Google Ads audiences.

4. Automate the Google Ads import files

The last step is to automate the generation of the newline delimited JSON files. The naming of these files should like like API[ACM]_config[audience_1_add]_<YYYYMMDD>.json, where the connector key (ACM) and config key (audience_1_add) should match the entries in config_api.json

The content of the newline delimited JSON files should look like;

1{ "hashed_email": "f660ab912ec121d1b1e928a0bb4bc61b15f5ad44d5efdc4e1c92a25e99b8e44a" }
2{ "hashed_email": "bceede42d53175d644efb79514d3967d3f9b92eb810b2c30bcddfae571b8b377" }
3{ "hashed_email": "8d1dc4e440a9965c3388c8714d3dd25ce99d855fd6828396f0fab07f57fba820" }

Most likely, you will have to execute the following tasks;

• Create a selection (audience) based on available data in your data warehouse (e.g. "all high-value customers")
• Calculate delta's / incremental tables, containing the create and remove audience operations
• When using BigQuery, you could create date sharded (partition) tables (audience_1_YYYMMMDD), holding all the the SHA256 hashed emails of that date for the specific audience.
• Based on these date sharded tables, you can calculate the differences between the most recent audience state and audience state of the previous date.
• With these differences/deltas you can create two intermediate tables, one containing the SHA256 hashed emails for the create operations (new users in audience), the other containing the delete operations (users need to be deleted).
• Export the 2 intermediate tables to JSON files (newline delimited), using the correct filename > API[ACM]_config[audience_1_add]_<YYYYMMDD>.json (can be exported directly from BigQuery)
• Move file to the Google Cloud bucket where the Tentacles Cloud Function is listening to (in the /outbound folder)

You could use the following SQL to calculate differences based on two date sharded tables (current date vs. previous date):

1#standardSQL
2SELECT
3  operator AS operator,
4  JSON_EXTRACT_SCALAR(record, '$.hashed_email') AS hashed_email
5FROM (
6  -- Get new / adjusted records compared to previous table -> CREATE operations
7  SELECT 
8    *,
9    'create' AS operator, 
10  FROM (
11    SELECT TO_JSON_STRING(a) AS record FROM `{table_name}_*` a WHERE _TABLE_SUFFIX = '{current_date}' -- New table
12    EXCEPT DISTINCT
13    SELECT TO_JSON_STRING(b) AS record FROM `{table_name}_*` b WHERE _TABLE_SUFFIX = '{prev_date}' -- Old table
14  )
15  UNION ALL
16  -- Get missing records in current table, compared to previous table -> REMOVE operations
17  SELECT
18    *,
19    'remove' AS operator, 
20  FROM (
21    SELECT TO_JSON_STRING(b) AS record FROM `{table_name}_*` b WHERE _TABLE_SUFFIX = '{prev_date}' -- Old table
22    EXCEPT DISTINCT
23    SELECT TO_JSON_STRING(a) AS record FROM `{table_name}_*` a WHERE _TABLE_SUFFIX = '{current_date}' -- New table
24  )
25)

There are a lot of solutions possible automating these tasks, but you could use one of the following setups;

1. Apache Airflow

Apache Airflow has all the connectors available to automate these tasks (create audience tables, create intermediate / delta tables) and export it in JSON format to the Tentacles Cloud bucket. Using Apache Airflow, you can create a scalable solution that can handle a lot of audiences.

2. Scheduled Queries + Cloud Scheduler + Cloud Functions

• Use BigQuery Scheduled Queries to generate the daily full audience tables and the delta tables (see example SQL above).
• Use Cloud Functions to run a function that starts the BigQuery export job to a JSON file in the Tentacles bucket. See this code.
• Let Cloud Scheduler run the Cloud Function after the scheduled queries ran.

Summary

Tentacles will enable you to activate your data directly out of your cloud environment pretty easily. It could save some money compared to expensive CDP tools / suites, but keep in mind

• It doesn't provide you with a nice user interface
• Only has some basic (error) logging
• Heavily Google focused, but customer connectors can added / developed easily

Some last tips;

• Google Ads Connector; when you using multiple identifiers (like email and telephone), you need to split up files according to the number of different identifiers used + add/ remove operations. Resulting in 4 files per audience when using hashed_email and hash_phone_number for example. Not that efficient, but you can adjust the Google Ads connector (nodeJs) without much effort, for example combining create and add operations in 1 file.
• You can debug Tentacles, using the logs provided in the Google Cloud Functions interface or the Cloud Logging / Stackdriver interface.

Useful links & documentation

• Tentacles GitHub repository
• Tentacles overview & architecture by Google
• Tentacles tutorial by Google