**Given the recent developments surrounding Google Analytics within the EU, it's not unrealistic to think that organizations are wondering what else is out there. Competitors in the analytics space are bashing Google Analytics every chance they get, but I would advise to look beyond that.**

Don't get me wrong, I think Google Analytics is an fantastic product and I hope they will sort it out. But it's always good to broaden your view, especially when talking about Google Analytics, which was / is a no-brainer for a lot of companies. 

Instead of giving you an exhaustive list with all the **alternative or complementary** analytics vendors and/or frameworks available, I will try to identify common analytics functionality and what **types of alternatives** are out there that enable you to build a data driven organization:

- An overview of common functionalities within analytics tools and frameworks.
- Relevant developments within the analytics and data space (not only on technical).
- Approaches & types of alternatives.
 
> I'm aware that the recent developments is potentially impacting more than Google Analytics alone, but I'm not diving into the legal aspect of things in this article.

## Google Analytics, a brief history

Yes, Google Analytics, 2005, Urchin and [all that](https://en.wikipedia.org/wiki/Urchin_(software)). Bottom line is, Google Analytics is the leading analytics platform for years. From my experience, Google Analytics has the following benefits: 

- Adoption & maturity
- Familiarity of the (reporting) interface
- Documentation & community
- Ease of implementation
- Reporting API
- Integration with other products. Not only GA integrating with other Google products but also out-of-the-box integration of GA in numerous tools (think of CMS systems, plugins etc).
- Advanced capabilities (like raw data exports to Google BigQuery)

It's not that other tools are lacking most of these capabilities, but making your organization data driven is mainly about people and process. Data analysts and technical marketers are scarce, so it really helps that there is a tool most users are already familiar with, even when they switch jobs (especially for the less tech savvy users).

## Common features

So let's dig into the (high-level) features of (web) analytics tools to give some context. In reality, there are countless features and every tool has it's own approach. But overall, you can argue that the main focus is to **measure and report on behavioral data, mostly related to web and app activity**. Common features:

| Functionality | Description |
| --- | --- |
| Data collection client | Think of Javascript libraries (e.g. gtag.js) and ready-to-use packages or SDKs that can be used directly within a PHP, NodeJS or other development environment. These clients will interact with the (collection) API of the analytics tool. Also closely related to Tag Management such as Tealium, Google Tag Manager or Segment: providing templates / out-of-the-box integrations for these data collection clients. The data collection client should also support customization in some form (e.g. custom events of properties). | 
| Basic reporting | A clear and easy-to-work-with reporting interface where you can get basic insights out-of-the-box. |
| Custom reporting | A custom reporting interface where business specific reports can be build & shared within your organization. |
| User management | Control user access to reporting data / dashboards |
| Integrations | Integrate with other products, think of data enrichment (e.g. auto tagging) and building and sharing audiences | 
| API & exports | Exporting data, both aggerated or raw event data. By providing a (reporting) API or real-time / periodical data dumps to (analytical) databases. |

![analytics-architecture.PNG](https://static.stacktonic.com/medium_analytics_architecture_d093116b6d.PNG)

## Developments within the market

As mentioned, becoming a data-driven organization is not only about the tool and its features. There are some relevant developments in recent years that have impact on how you should build your data-driven organization:

- Data analysts & technical marketers are scarce.
- The offering and ease of use of analytics related tools improved enormously (e.g. SaaS tools, analytical databases)
- Much easier (and cheaper) to work with large amounts of data (such as web/app clickstream data) within (cloud) analytical databases (Snowplow, BigQuery etc).
- Privacy & regulations (think of GDPR, cookie consent and the current buzz around Google Analytics / EU legislation)
- Move to "server-side" event / tag management
- "Self-service" BI tools on the rise. Related to the adoption of cloud analytical databases.

## Types of alternatives

These developments impact decision making but also increases the development and popularity of **alternative** or **complementary** tools and frameworks. So I'm not looking into "all-in-one" reporting suites only (as I would categorize Google Analytics). There are other solutions as well:

1. All-in-one reporting suites
2. Data collectors & data warehouse loaders
3. Event hubs / routers

To give you some examples, when talking alternative solutions:

- Switch to an alternative "reporting suite" vendor (duh!)
- Set up a data collector that collects and loads the data into your data warehouse. Use a BI tool to set up and share the reports within your organization. Modern analytical databases and BI tools (Superset, Looker, PowerBI, Tableau etc) are perfectly able to handle this types of data. But you will have to acquire some expertise to set up these flows and reports initially (+ onboarding / training of the end users).

Complementary solutions:

- Set up an event hub for data collection and route the events to Google Analytics or another analytical tool. 
- Use the raw data dumps next to the (native) reporting interface to execute more advanced use cases and/or reporting in a BI tool (or to prepare an organization for a life without the application / default reporting interface).

**The privacy aspect**

Privacy and compliance is becoming much more important (and rightfully so). Think of the location where the data is stored, user consent or what kind of data (e.g. PII, cookies etc) is send to the analytics servers. So make sure you know how a specific tool is dealing with these aspects and what the configuration options are. It could also be a wise idea to self-host a solution for example.

Almost every tool of framework is using a first-party cookie to measure the same user over different visits. But you could also decide to not use any cookies (or only after user consent). You will lose a lot insights, but it's still possible to measure effectiveness of your website for example, however not on user level.

Let's have a look at the different types of solutions.

> Note: I haven't tried all the alternatives in depth, so I'm giving a high-level description of the tools, instead of a full comparison. **Please let me know if you're missing something!**

## 1. All-in-one reporting suites

In these type of tools the user (reporting) interface plays a central role. Everything is built around the interface. How to create a measurement plan and set up data collection *the right way* is something else, but when you set up data collection in these tools, the default reports are populated immediately and you're good to go. 

| Vendor | Description | Pricing | Hosting |
| --- | --- |
| [Google Analytics](https://marketingplatform.google.com/about/analytics/) | Analytics offering within the Google Marketing Platform | Free & paid | Cloud |
| [Piwik Pro](https://piwik.pro/) | Analytics suite with a focus on user privacy and data security | Free & paid | Cloud |
| [Mixpanel](https://mixpanel.com/) | Focus on product analytics. | Free & paid | Cloud |
| [Amplitude](https://amplitude.com/) | Focus on product analytics. Part or a larger suite / other modules available. | Free & paid | Cloud |
| [Adobe Analytics](https://business.adobe.com/products/analytics/adobe-analytics.html) | Analytics product within the Adobe Experience Cloud. Corporate focus. | Paid | Cloud |
| [Matomo](https://matomo.org/) | Formerly Piwik. Free and open source analytics platform, focused on user privacy and data security. | Free & paid | Cloud / On-premise |
| [Heap.io](https://heap.io/) | Focus on auto capturing events and behavioral metrics | Free & paid | Cloud |

## 2. Data collectors & loaders

Secondly, this category of tools is focusing on the data collection part and storing that data into an (cloud) analytical database (e.g. Snowflake, Google BigQuery or AWS Redshift), without providing an reporting interface. 

Snowplow is popular in this category and around for some time. The tool provides an extensive library of data collection clients, such as a Javascript tracker, SDKs, packages for popular frameworks such as NodeJS. Main goal is to get the raw behavioral data to your (cloud) data warehouse in real-time.

**Note:** Equally important for the other categories as well, a good collection client / API, including a sound data structure / schema (protocol), handling of (first-party) cookies and customizability is not present in every tool. Good example of mature collection protocols: the [GA measurement protocol](https://developers.google.com/analytics/devguides/collection/protocol/ga4) or the [Snowplow Tracker Protocol](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/)

![analytics-architecture-datacollector.PNG](https://static.stacktonic.com/medium_analytics_architecture_datacollector_ce687a3341.PNG)

| Vendor | Description | Pricing | Type |
| --- | --- |
| [Snowplow](https://snowplowanalytics.com/) | Open source platform for (online) behavioral data collection, streaming data into (cloud analytical) databases. | Free & paid | Cloud / On-premise |

## 3. Event hubs / routers

This last category has overlap with the previous one, but ads an additional component: routing / distributing the collected events to a range of other systems (on the **server-side**).  

Almost every tool will provide you with an user interface and templates to configure those routes. Sometimes part of a larger suite (for example as a step before aggerating event data to user profiles and audiences).

Rules, filters and transformations can be applied to the (raw) data stream. Next to data warehouses as a destination, other types of data destinations are present as well. Think of analytics tools (category #1), marketing channels (e.g. Facebook) or marketing automation platforms. All without implementing additional Javascript tracking libraries, seen the events are forwarded on the server-side.

![analytics-architecture-eventhub.PNG](https://static.stacktonic.com/medium_analytics_architecture_eventhub_0c6cd9edf2.PNG)

| Vendor | Description | Pricing | Hosting | 
| --- | --- |
| [Tealium EventStream](https://tealium.com/products/tealium-eventstream/) | Collect & route events to external applications or the Tealium CDP.  | Paid | Cloud |
| [Google Tag Manager Server](https://developers.google.com/tag-platform/tag-manager/server-side) | Self hosted "server-side tag management" solution of Google. Distribute behavioral event streams to Google and non-Google destinations. | Free (only cloud costs) | On-premise |
| [Segment](https://segment.com/) | Collect and distribute data to other applications. Also has CDP capabilities.  | Free & paid | Cloud |
| [Rudderstack](https://www.rudderstack.com/) | Collect and distribute data to other applications. Focus on developers and has CDP capabilities. | Free & Paid | Cloud |
| [MetaRouter](https://www.metarouter.io/) | Server-side event router with a focus on compliance and security | Paid | Cloud / On-premise |
| [Jentis](https://www.jentis.com/) | Server-side tag management solution hosted in the EU | Paid | Cloud |
| [Jitsu](https://jitsu.com/) | "Open source Segment alternative" | Free & paid | Cloud / On-premise |

## Closing thoughts

Selecting an alternative or complementary data collection tool or stack should depend on your type of business and more important, your requirements. The list or requirements is growing, since not only the wishes of the business user are important. The privacy of the user tracked is equally important and it's not given that the tool that shouts the loudest regarding their privacy offering is the best option in your situation.

Start with your business objectives, requirements, the people that need to work with the tool and the expertise and resources available within your organization. Then pick a vendor.




Given the recent developments surrounding Google Analytics within the EU, it's not unrealistic to think that organizations are wondering what else is out there. Competitors in the analytics space are bashing Google Analytics every chance they get, but I would advise to look beyond that.

large_analytics-architecture-header.PNG

medium_analytics-architecture-header.PNG

small_analytics-architecture-header.PNG

thumbnail_analytics-architecture-header.PNG

analytics-architecture-header.PNG

Google Analytics alternatives - An overview

**Privacy related developments such as tracking prevention mechanisms impacts *web* measurement. But what about mobile apps? Next to advertising and ad attribution (e.g. Apple's App Tracking Transparency), it also impacts first-party app measurement, campaign and install tracking. This article will shed some light on app measurement in general and how to optimize first-party measurement of mobile apps and campaigns (both Android as iOS).**

- Impact of recent privacy related updates and changes (such as iOS 14.5)
- What user identifiers are used to measure cross-app (device) and in-app behavior.
- How to measure in-app activity & engagement.
- How to track campaigns when opening or installing an app.
- Technical solutions and architecture.

 > Note: This article does not express my opinion about companies like Google or Apple and their approach to privacy. I'll stick to the facts and consequences of decisions they made, but I certainly do recognize the underlying motives; an increased need and focus for privacy and transparency.

## The Advertising (Device) Identifiers (IDFA / GAID)

So, let's start with the user identifier that is (or was) used for cross-app, device tracking and received the most impact recently; the Advertising Identifier. The Advertising Identifier can be seen as the *third-party cookie* of the mobile app ecosystem. When enabled, the IDFA is able to track the same user (mobile device) over different apps. For example:

- Ad networks can track and (re)target users
- Attribution of ad impressions / clicks from an publishers app to conversions in your own app.
- Install tracking, by using the IDFA as an identifier before and after the redirect to an app store.

As mentioned, the IDFA can be used to track and target users cross-app and is an unique identifier for a device.

Both iOS (Apple) and Android (Google) use a variant of the advertising identifier;

| Platform | Identifier | Description | Remarks | 
| --- | --- | --- | --- |
| Android | GAID | Google Advertising ID. Identifies the unique device | User consent needed in most cases. |
| iOS | IDFA | ID For Advertisers. Identifies the unique device | User consent needed trough the AppTransparencyFramework and additionally your company specific consent. |

Recently, as of iOS version 14.5, Apple basically disabled the use of the IDFA by introducing the App Tracking Transparency (ATT) framework. This heavily impacted platforms like Facebook that depended on the IDFA for ad targeting and cross-app user tracking.

## iOS App Tracking Transparency (ATT)

Introduced in iOS 14.5, a user needs to give explicit permission though the iOS *[AppTransparencyFramework](https://developer.apple.com/app-store/user-privacy-and-data-use/)* (ATT) when certain types of (user) data is collected in an app and shared with third-party vendors. The framework basically is an iOS "native" consent popup which can't  be altered a lot (design or inner workings). You also need to document the data collected in your privacy details.

![apple-app-transparency-framework-popup.PNG](https://static.stacktonic.com/apple_app_transparency_framework_popup_cc7eb9a01e.PNG)

In their documentation, Apple is [mentioning situations](https://developer.apple.com/app-store/user-privacy-and-data-use/) (important!) where where permission through the *AppTransparencyFramework* is necessary. Mostly related to sharing data with third-parties that is used for ad targeting or to track users (cross-app and device). Not adhering to these rules can result in your app **being deleted from the iOS App Store**.

When talking about first-party measurement, the last situation they mention is important:

- *Placing a third-party SDK in your app that combines user data from your app with user data from other developers’ apps to target advertising or measure advertising efficiency, even if you don’t use the SDK for these purposes. For example, using an analytics SDK that repurposes the data it collects from your app to enable targeted advertising in other developers’ apps.*

Apple **allows** you to collect first-party data for analytics purposes without a user consenting to the ATT popup. (but, make sure you don't use this data for retargeting for example and also make sure you are compliant to local privacy rules). 

Without an Advertising-ID, it's perfectly possible to measure (first-party) in-app user engagement and campaign performance, and in some cases  install tracking.

## First-party measurement: The App Instance Identifier

To measure first-party app activity, you will need some sort of identifier that can identity activity of the same user over different app "sessions" (probably not the right term when talking about apps). In contrary to the advertiser-ID, the instance-ID doesn't identify the same user over different apps, so it can't be used to recognize the same user of different apps. You could compare this with a *first-party cookie* on web. In general:

- The instance identifier uniquely identifies an app installation on a specific device.
- Doesn't track a user over different apps (third-party context).
- In most cases, when app or device cache is reset or an uninstall occurs, the instance identifier is deleted.

As an app developer, one could create such an instance identifier yourself but many app measurement and (campaign) attribution tools can handle this for you. Some pretty random examples of app measurement tools and how they're handling the app instance-ID:

| Framework | Instance Identifier | Description | Docs |
| --- | --- | --- | --- |
| Firebase / Google Analytics 4 | Firebase installation ID (FID) | Previously FirebaseInstanceId. FID is reset on uninstalls, clearing device or app cache or after 270 days of inactivity. Used for the Google Analytics 4 client-ID / user_pseudo_id. | [docs](https://firebase.google.com/docs/projects/manage-installations#monitor-id-lifecycle) |
| AppsFlyer | AppsFlyer ID | Automatically created on every install | [docs](https://support.appsflyer.com/hc/en-us/articles/4408847686161-Device-identifiers) | 
| Adjust | Adjust Device Identifier | Created on install | [docs](https://help.adjust.com/en/article/gather-device-ids-android-sdk#adjust-device-identifier) |
| MixPanel | Distinct-ID | Reset on uninstalls or clearing app or device cache. | [docs](https://help.mixpanel.com/hc/en-us/articles/115004509426-Distinct-ID-Creation-JavaScript-iOS-Android-#ios) |

Using these tools, first-party measurement can be implemented (such as GA4 or Singular). Think of measuring *first-opens, screen views, events, purchases*.  Most of these tools will handle some default events (app opens, first opens / installs or uninstalls) and implement other functionality (like Google's Install Referrer API) out-of-the-box.

> When picking a first-party app measurement tool, make sure collection of Advertising Identifiers (see the first chapter) is disabled without user consent. Additionally, using iOS, when you want to leverage the IDFA, you need to enable to the Transparency Framework popup. 

In general, implementation of these tools is roughly the same, you have to implement an SDK and set up screen and event tracking within your app. Depending on the tool of choice, they will provide an (reporting) interface and in most cases provide the option to access the data through an API or dump the data in analytical database (think of Google BigQuery when talking about Firebase).

But what about **measuring campaign performance**? How did users find and install your app? And if they already installed your app, how did you reactivate those users? For this, deep links come in handy.

## Deep linking

It's convenient to open a specific screen within your app directly. For example when a user is searching for a specific product on the web, already has your app installed, clicks the product link and the app opens the specific product screen (instead of the home screen). For this, deep links can be used. When your app is already installed, this is commonly referred to as *direct deep linking*. 

Deep linking is closely related to campaign (and install) tracking, since you can add *campaign info* to the deep links such as the campaign source, name or ad content (think of the well-known *?utm_xx=* parameters), so campaign performance can be measured.

There is no industry standard regarding deep linking, so every platform handles it slightly different. Under the hood, iOS and Android are using the methods mentioned below (and by default only works when the app is already installed). 

- iOS: [Universal Links](https://developer.apple.com/ios/universal-links/) 
- Android: [Android App Links](https://developer.android.com/studio/write/app-link-indexing)

But what if:

1. A user clicking the deep link doesn't have your app installed? 
2. How to pass the deep link and campaign parameters to your app when a user is redirected to one of the app stores to install your app?
3. A user is on a desktop device?

In the first two cases, *deferred deep linking* can be used. Basically, the deep link is "deferred" after the app is installed (deep linking *with memory*). For this to work, most likely you will like need a tool or framework that is handling this for you (like Firebase, Singular or Branch).  Additionally (in most tools), you can customize flows that handle the different situations mentioned above. 

A simple flow could like like this:

![app-install-open-tracking-flow.PNG](https://static.stacktonic.com/app_install_open_tracking_flow_9522782fa2.PNG)

> Next to deep link flows , most vendors are offering related features like app measurement, monitoring, attribution, deep linking (flows) and/or error logging in one product suite.  

## Deferred deep linking / app-to-app / web-to-app flows

As mentioned, deep links can be used to track campaigns and traffic sources (by adding campaign information to the deep link URL so that these are passed to the app). One problem: what if the app isn't installed and the user gets redirected to one of the app stores? Can we still pass the deep links and campaign parameters?

Yes, but seen the recent developments (e.g. iOS 14.5) options are limited. 

Some of the most used mechanisms for *deferred deep linking* (and used by most tools in one form or another):

| Method | Type | Description |
| --- | --- | --- |
| IDFA / GAID | Deterministic | The IDFA / GAID is used to identify the same user between a (deep link) click and a first-open (install). This way the deep link and campaign parameters can be retrieved on app open. Limited used due to user consent necessary / iOS 14.5. | 
| Install Referrer API (Android) | Deterministic | Leveraging the Install Referrer API of the Play Store, campaign parameters used to visit the Play Store (before install) can be passed to the app on first-open.  |
| Pasteboard "hack" | Deterministic | Saving the deep link on click in the iOS pasteboard and reading the value when (first) opening the app. Not possible after recent iOS update. |
| Use the app store analytics reports | Deterministic | Leveraging the reporting interfaces of the app stores to get (aggregated) information about campaign effectiveness |
| SKAdNetwork (iOS) | Deterministic | (privacy-first) mechanism for aggregated install attribution statistics. Cannot be used for deep linking, is not on user level and is not real-time (24-48 hours). Also see [this article](https://www.singular.net/blog/what-is-skadnetwork/). |
| Fingerprinting | Probabilistic | Using identifiers like IP address to match the same user when clicking a deep link and opening the app after install. Not as accurate as the methods described above. Limited use duo to user consent needed in most cases. |

## Install measurement

So which methods are left? The App Store and Play store have their own (simplified) analytics / measurement interface. Both app stores accept campaign tracking (URL) parameters when referring users to the stores, so app installs and other metrics can be attributed to campaigns. 

** App / Play Store campaign parameters**

| Platform | Campaign tracking | Docs |
| --- | --- | --- |
| Android | `?utm_...=` parameters (like `?utm_source=` and `utm_medium=`, same as Google Analytics) can be used to label traffic to the Google Play Store and viewed in the Play Store reports. | [docs](https://developers.google.com/analytics/devguides/collection/android/v4/campaigns) |
| iOS | The `?pt=` (provider) and `?ct=` (campaign code) URL parameters can be used to label traffic to the Apple App store. Reports available within Apple App Analytics. | [docs](https://help.apple.com/app-store-connect/#/itcfa7936330) |

This is nice and will give you some insights, but can we retrieve these values within the app and therefore your own, first-party measurement system? In iOS, no. Android? Yes. By using the Playstore's *Install Referrer API* to request the install referrer (campaign parameters used when visiting the Play Store) on the app's first open.

![app-install-tracking.PNG](https://static.stacktonic.com/medium_app_install_tracking_1ad4f23aea.PNG)

To summarize the available options for install tracking;

- Heavily depending on the tool you use to create deep links (or a framework developed in-house)
- Within Android, the Install Referrer API can be used
- On Android, the GAID linking method can be used, but usage is limited due to user consent. This method in iOS is practically deprecated after iOS 14.5. 
- For both Android and iOS probabilistic methods (fingerprinting) can still be used, but make sure you know how this works exactly and have the necessary user consent.
- On iOS, the SKADNetwork can be leveraged for aggerated statistics (mostly for app-to-app flows)

## Summary & closing thoughts

Overall, it's still perfectly possible to perform app measurement, including campaign measurement. The use of cross-app identifiers (IDFA/GAID) is limited nowadays due to developments related to privacy and tracking prevention. 
This is a good thing, but it does impact attribution, deep linking and install tracking. So you have to rethink your setup (boot technically as your privacy setup) and be aware of the impact on your analytics.

Privacy related developments such as tracking prevention mechanisms impacts *web* measurement. But what about mobile apps? Next to advertising and ad attribution (e.g. Apple's App Tracking Transparency), it also impacts first-party app measurement, campaign and install tracking. This article will shed some light on app measurement in general and how to optimize first-party measurement of mobile apps and campaigns (both Android as iOS).

medium_app-measurement-tracking.PNG

small_app-measurement-tracking.PNG

thumbnail_app-measurement-tracking.PNG

app-measurement-tracking.PNG

First-party mobile app measurement & install tracking

Increased privacy concerns, legislation and browser updates affecting (third-) party cookies is resulting in a new era where users regain control over their data and organizations should rethink their customer and marketing strategy. This will impact a lot of area's such as data collection, advertising, first-party data strategy, legal and UX. 

Privacy, cookies & first-party data

**This guide shows you how to set up an independent "vendor-agnostic" Event Driven Data Layer (EDDL) and how to process events within this data layer. Example code is provided, so you can build your own data connectors. **

- Learn **how to set** up an independent Event Driven Data Layer.
- Learn how to **attach an event listener to process the events** in the data layer.
- Using the Google Data Layer Helper library.
- **[Example connectors](#example-1---log-events-to-the-console)**: Simple console logs and a Tealium iQ (Tag Management) connector.

## Why is this still relevant?

This article could easily been written 5 years ago. Still it's relevant today and even more important seen the recent focus on **quality first-party data** (which should have been a focus all along..). 

Anyway, a lot of articles are written on this topic (some mentioned later on), but I felt I could add some value with some practical examples and give it some context seen recent developments regarding tracking & first-party data. 

A well structured Event Driven Data Layer (EDDL) including mechanism that enables you to process and listen to events in the EDDL, can be extremely powerful when you're planning to create an abstraction layer between your website / app and TMS or other client-side data collection tool. Especially;

- When you want to take **full ownership of your data layer** and don't want to depend on vendor specific code and/or requirements.
- When **not** using GTM.
- Using data collection tools or tag management systems that **do not have an event driven data layer mechanism** in place.
- You want to create your own data collection library / script (for example to provide data to Google Tag Manager Server without using the `gtag.js` library or client-side GTM).

But what exactly is an EDDL?

## The Event Driven Data Layer (EDDL)

The term EDDL gained popularity through the efforts of Jim Gorden (Jimalytics). As mentioned, there are already a lot of good articles written about the workings and benefits of the EDDL. So I would recommend to check out the following articles first:

- [The Event Driven Data Layer](https://jimalytics.com/tag-management/the-event-driven-data-layer/) by Jimalytics.
- [Event Driven Data Layer guide](https://www.searchdiscovery.com/blog/event-driven-data-layer-guide-1/) by Search Discovery.

**Short recap: ** Many know the concept of an EDDL from Google Tag Manager (the `dataLayer`). Replace `digitalData` with `dataLayer` in the example below all should look familiar.  

The Event Driven Data Layer itself basically is just a plain Javascript array (`digitalData` in the example below), containing (event) objects.  Think of pageviews, clicks or purchases. The Javascript array is available at `window` scope.

Example:

```javascript
window.digitalData = window.digitalData || [];

// Pageview event
digitalData.push({
  "event": "page",
  "page_type": "product_detail",
  "product_id": "12345",
  "product_name": "Shiny red shoes"
  // ...
});

// Purchase event
digitalData.push({
  "event": "purchase",
  "transaction_id": "777",
  "product_id": "12345",
  "product_name": "Shiny red shoes"
  // ...
});
```

You can structure the events and properties any way you like, so that it nicely fits your type of business. Keep in mind, **consistency is key**!

When such a data layer is in place, one can write some code or use existing libraries to **process the events pushed into the data layer** array and **attach listeners** to it. The listeners can be used to build connectors or wrappers around vendor-specific tracking scripts, so you can map your data layer variables to vendor specific requirements (in one place).

Some common use cases:

- Connect event listeners to the data layer (e.g. Tag Management Systems / tracking script connectors)
- Connect data validators (e.g. JSON schema validation)
- Transform data to specific formats / business requirements before data is send to data collection or TMS tools.
- Switch listeners based on different platforms or environments, but using the same data layer. Think of parts of your website that are loaded in a Webview, for example in Hybrid apps (like Cordova)

![vendor-agnostic-data-layer.PNG](https://static.stacktonic.com/small_vendor_agnostic_data_layer_0c9f4e95b4.PNG)

## The benefits of using an EDDL

But what are the benefits of an EDDL compared to (vendor specific) data layer and collection setups?

- **No vendor specific knowledge** required **for developers** (e.g. vendor libraries), plain Javascript arrays and objects (except for building the connector itself).
- **No vendor lock-in** / vendor agnostic, swap out vendors (client-side TMS / tracking solutions) more easily.
- Data layer structure not tied to vendor specific requirements.
- Asynchronous
- Excellent fit for single page / event-driven applications.

## Vendor Agnostic what?

As you can see in the example above, besides (optional) event object structure, there isn't much vendor-specific logic involved. Since GTM uses the same concept, you could say that GTM already uses an "independent" data layer (besides the recommended event structure). 

But, Google Tag Manager isn't the only TMS used (yes, really). Think of Tealium, Adobe or Segment. But we also have data collection scripts that can be implemented without a TMS, such as the `gtag.js` library (and becomes relevant if you don't want to use client-side GTM to implement server-side GTM for example).

In many cases, these vendors require you to implement vender specific code. Some examples:

**Tealium iQ (TMS)**

A Tealium iQ pageview:

```javascript
utag.view({
    "tealium_event": "product_view",
    "page_type"    : "product",
    "product_id"   : ["12345"],
    "product_name" : ["Red shoes"]
});
```

**Adobe**

> Adobe recently (2020) launched it's own event driven data layer library, [called ACDL](https://github.com/adobe/adobe-client-data-layer), so bad example.

**gtag.js**

A gtag.js pageview:

```javascript
gtag("event'" "page_view", {
  "page_title": "<Page Title>",
  "page_location": "<Page Location>",
  "page_path": "<Page Path>",
  "send_to": "<GA_MEASUREMENT_ID>"
})
```

As you can see, there is a lot of vendor specific code involved. The point here is that you **don't want is to implement such vendor specific code all through your website or app**. So, creating an abstraction layer would be a wise approach. 

> I've implemented an event driven data layer for multiple clients over the last few years (DigitalDataQueue), and it really has proved its worth. For some clients, after the EDDL implementation, we've changed the TMS multiple times (something to do with the business being sold multiple times..). It saved a lot of people a lot of work and data quality was never compromised.

## Listen to and process events in the data layer

A good example of a data layer "listener" library is the [Data Layer Helper library](https://github.com/google/data-layer-helper) by Google (also used in GTM). We will use this library in the examples below.

> Back in 2018, together with my former colleague [Folkert Mudde](https://nl.linkedin.com/in/folkertmudde) we've rewritten added some features to the Google Data Layer Helper Library and named it [DigitalDataQueue (GitHub)](https://github.com/Oogst/digital-data-queue). It's a bit outdated now and Google updated their Data Layer Helper Library in the meantime, so I will use the Google library in the examples below.

The Data Layer Helper library will:

- Process events passed into the data layer (`digitalData.push({ .. })`)
- Activate listener functions (connectors) when new events are pushed into the data layer
- Process (historical) event already present in the datalayer before the library is activated. 
- Merges event data (object) into a data model (object)

**Order of operations**
![data-layer-helper-order-of-operations.PNG](https://static.stacktonic.com/data_layer_helper_order_of_operations_caf860beeb.PNG)

Firstly, check out the [GitHub repo]((https://github.com/google/data-layer-helper)) of the Data Layer Helper. The library is documented pretty well. 

You can add the library to you application easily;

``` bash
npm install data-layer-helper
// or
yarn install data-layer-helper
```

And import the package

```
import 'node_modules/data-layer-helper/dist/data-layer-helper';
```

Other option is to host the minified [`data-layer-helper.js`](https://github.com/google/data-layer-helper/blob/master/dist/data-layer-helper.js) version of the library somewhere and include it into you application.

Let's move on to the examples.

## Example 1 - Log events to the console

The first example will just log the events pushed in the data layer to the (browser) console. It nicely demonstrates the capabilities of the library.

```javascript
// Make sure to include the Data Layer Helper library first

window.digitalData = window.digitalData || [];

// Pageview event
digitalData.push({
  "event": "page",
  "page_type": "product_detail",
  "product_id": "12345",
  "product_name": "Shiny red shoes"
  // ...
});

const DigitalDataHelper = new DataLayerHelper(digitalData);

function listenerLogToConsole(model, event) {
  console.log(event);
}

const DigitalDataHelper = new DataLayerHelper(digitalData, {
  listener: listenerLogToConsole,
  listenToPast: true, // Process events already in the digitalData array before initializing the library.
});
```

Now, when you execute the code, you should see the pageview event in your browser console. 

## Example 2 - Tealium iQ connector

The seconds example show you how to create a wrapper function around Tealium iQ specific code, so its activated based on events within the data layer.

**Tip:** It's perfectly possible to include the data-layer-helper.js script AND the listener function in a `Preloader` extension within Tealium itself. Benefit is that you know for sure that Tealium is loaded before activating the listener (+ easy to edit the listener function within Tealium itself).

```javascript


<script type="text/javascript" src="data-layer-helper.js"></script>


<script type="text/javascript">
	// Declare datalayer
	window.digitalData = window.digitalData || [];
	
	// .. Populate datalayer
	// digitalData.push({ .. });
	
	// Tealium config overraide: disable default pageview `view()` on load 
	// To prevent duplicate pageviews, since the page event will trigger a pageview
	window.utag_cfg_ovrd = {
		noview: true
	};
	
	// Tealium Universal Base Tag
	(function(a,b,c,d) {
		a='//tags.tiqcdn.com/utag/ACCOUNT/PROFILE/ENVIRONMENT/utag.js';
		b=document;c='script';d=b.createElement(c);d.src=a;
		d.type='text/java'+c;d.async=true;
		a=b.getElementsByTagName(c)[0];a.parentNode.insertBefore(d,a)})();

	// Data layer listener function: Tealium connector
	// Tip: You can implement this code directly in the front-end OR add the code as a preloader extension within Tealium
	function listenerTealium(model, event) {
	   
		// Declare utag_data / UDO (Tealium datalayer)
		window.utag_data = window.utag_data || [];
	   
		// Tealium specific transformations
		var _event = event;
		var _model = model;
		
		// Mapping digital-data-queue events to Tealium
		if (typeof _event.event != 'undefined') {
			switch(_event.event) {
				case "page":
					window.utag_data = _event;
					window.utag.view(window.utag_data, function() { window.utag.DB("digital-data: Event: \"" + _event.event + "\" to utag.view()"); });
					break;
				default:
					// Using the model (instead of event) to populate event parameters, so page properties are available within the event.
					window.utag.link(_model, function() { window.utag.DB("digital-data: Event : \"" + _event.event + "\" to utag.link()"); });
			}
		} else {
			// No event property set
		}
	}

	// Init Data Layer Helper
	var DigitalDataHelper = new DataLayerHelper(digitalData, {
		listener: listenerTealium,
		listenToPast: true, // Process events already in the digitalData array before initializing the library.
	});
</script>
```

## Other interesting reads

The examples above showed you some of the possibilities using an EDDL / event listener. I'd love to hear your experiences and use cases.

Lastly, some interesting articles related to this topic not mentioned before:

- [How to trigger actions on a data layer](https://antonbies.com/event-driven-data-layer-how-can-you-trigger-actions-on-a-data-layer-push/) by Anton Bies
- [The Data layer explained](https://www.simoahava.com/analytics/data-layer/) by Simo Ahava (old but gold)

This guide will show you how to set up an independent (vendor-agnostic) Event Driven Data Layer (EDDL) and how to process events within this data layer. Example code is provided, so you can build your own data connectors. 

large_event-driven-data-layer.PNG

medium_event-driven-data-layer.PNG

small_event-driven-data-layer.PNG

thumbnail_event-driven-data-layer.PNG

event-driven-data-layer.PNG

The Independent Event-Driven Data Layer – A Practical Guide

**Running Google Tag Manager (GTM) Server in Google App Engine? Learn how to set up error alerts when 4xx and/or 5xx alerts exceed a certain threshold.**

When using GTM Server in production, it's important that the service runs stable and when it does not, you're notified straight away. This tutorial will show you how to:

- Set up alerts in when **5xx** and **4xx** errors exceed a certain threshold
- Using Google Cloud Monitoring
- Notifications can be send by  **email **,  **sms **,  **Slack ** or to  **other channels **.

## What is Google Cloud Monitoring?

Google Cloud Monitoring can be used to gain (performance) insights in the services you run within the Google cloud. Additionally, it enables you to create alerts based on all sorts of (performance) metrics. 

You can set up alerts by creating an **alerting policy**. This is basically done in the drag and drop interface of the GCP console OR you can upload a policy file using the *Monitoring API* (not all types of alerting policies and rules can be configured in the GCP interface). 

When an incident occurs, alerts can be send to different **notification channels** like: 

- Email / SMS
- Slack
- Webhooks
- Cloud Pub/Sub

## Create alerts for Google App Engine / GTM Server

Google Tag Manager Server can (and in a lot of cases will) handle a lot of traffic. The **ratio** of 4xx and 5xx errors can be a good indicator for your GTM server stability. For example, a lot of 5xx errors can point to server stability or scaling issues. 

Our goal was to set up a policy that will trigger when the ratio of 4xx and 5xx errors, compared to the total request exceed a certain threshold. However, we've tried to *drag and drop* the related metrics to create this policy in the interface, but we didn't succeed.

Luckily, I've came across [this example](https://cloud.google.com/monitoring/alerts/policies-in-json#json-ratio) in the Google documentation, explaining how to set up a *metric-ratio* policy, using the Monitoring API (and indeed, it's currently not possible to set this up by dragging and dropping stuff in the interface).

> So caution! When you want to update your policy, make sure you do it through the Monitoring API and not in the GCP interface. It seems to overwrite some of the settings, resulting in not triggering any errors.

I've adjusted to policy from the example a bit. The policy will:

-  Take 5xx and 4xx errors into account. 
-  Error threshold is set to 1% (`thresholdValue`). However, if you have a lot of traffic 1% of all responses are still a lot of errors, so adjust to your specific setup.

Save the code (policy) below to your local filesystem, in this tutorial we've named it `gtm_server_alerting_policy.json`. 

> Make sure you replace `<your-project-id>` with your GCP project-ID. 

Next, you can use the Cloud Shell to upload the policy file:

1. In the Google Cloud Console, open up the Cloud Shell (icon in the top right corner of the interface). The Cloud Shell is activated on the bottom of your screen.
2. Click the "Open Editor" button (on top of the the Shell window) and upload the `gtm_server_alerting_policy.json` file.
3. Click the "Open Terminal" button and use the following command to create the policy:

``` bash
gcloud alpha monitoring policies create --policy-from-file="gtm_server_alerting_policy.json"
```

4. Now the policy is created and is visible in the GCP interface (Monitoring > Alerting):

![google-tag-manager-server-error-alerting.PNG](https://static.stacktonic.com/google_tag_manager_server_error_alerting_9ed8be08c8.PNG)

5. Configure your notification channels (like email or Slack) in the GCP interface (also under Monitoring > Alerting).
6. You're all set!

**gtm_server_alerting_policy.json**
``` json
{
    "displayName": "HTTP error count exceeds 1 percent for GTM Server / App Engine",
    "combiner": "OR",
    "conditions": [
        {
            "displayName": "Ratio: HTTP 4xx and 5xx error-response counts / All HTTP response counts",
            "conditionThreshold": {
                 "filter": "metric.label.response_code>=\"400\" AND
                            metric.label.response_code<\"600\" AND
                            metric.type=\"appengine.googleapis.com/http/server/response_count\" AND
                            project=\"<your-project-id>\" AND
                            resource.type=\"gae_app\"",
                 "aggregations": [
                    {
                        "alignmentPeriod": "300s",
                        "crossSeriesReducer": "REDUCE_SUM",
                        "groupByFields": [
                          "project",
                          "resource.label.module_id",
                          "resource.label.version_id"
                        ],
                        "perSeriesAligner": "ALIGN_DELTA"
                    }
                ],
                "denominatorFilter": "metric.type=\"appengine.googleapis.com/http/server/response_count\" AND
                                      project=\"<your-project-id>\" AND
                                      resource.type=\"gae_app\"",
                "denominatorAggregations": [
                   {
                      "alignmentPeriod": "300s",
                      "crossSeriesReducer": "REDUCE_SUM",
                      "groupByFields": [
                        "project",
                        "resource.label.module_id",
                        "resource.label.version_id"
                       ],
                      "perSeriesAligner": "ALIGN_DELTA",
                    }
                ],
                "comparison": "COMPARISON_GT",
                "thresholdValue": 0.01,
                "duration": "0s",
                "trigger": {
                    "count": 1
                }
            }
        }
    ]
}
```


Running Google Tag Manager Server (GTM) in Google App Engine? Learn how to set  up error alerts when 4xx and/or 5xx alerts exceed a certain threshold.

medium_google-app-engine-4xx-5xx-error-alerts.png

small_google-app-engine-4xx-5xx-error-alerts.png

thumbnail_google-app-engine-4xx-5xx-error-alerts.png

google-app-engine-4xx-5xx-error-alerts.png

How to set up error alerts for Google Tag Manager Server on App Engine

**
Due to privacy related browser developments such as tracking prevention mechanisms (ITP / ETP), it's getting more difficult to correctly set persistent first-party cookies. Even if there is no data shared with any third-party. Having consistent first-party cookies is important to measure your website user behavior over multiple website visits. When using popular frameworks such as Next.js or Gatsby, your website is prerendered into static HTML files and can be served from an AWS S3 bucket and the CloudFront CDN. Lambda@Edge can be used to run server-side code when a page is requested. In this tutorial we will use a Lambda function to set a persistent first-party (HTTP) cookie in the page response.
**

- Using an **Lambda@Edge function** attached to the ** viewer-response event** within CloudFront, a **persistent first-party cookie** can be set by modifying response headers and setting an HTTP (server-side) cookie. 
- The cookie contains a **UUID**. For new users, a fresh UUID is created. For returning users the existing value is used to extend the cookie.
- Go directly [to the code](#the-nodejs-lamda-function)

> The UUID enables you to track users over different visits (on your first-party domain). Be aware that you need consent of your users, depending on how you use this UUID. So keep the interests of your users in mind and be transparent / ask for the right permissions when needed.

## Modify AWS CloudFront responses using Lamda@Edge

As mentioned, prerendering your website or application to static (HTML) files and serving them out of an AWS S3 bucket in combination with the CloudFront CDN is gaining popularity due to frameworks such as Next.js or Gatsby. 

One of the challenges is that you can't run any server-side code directly from your static files. Lambda@Edge solves this problem. It enables you to run code (Python or Node.js) on several events within Cloudfront, so you can modify requests and responses. 
Lamda function can be used to redirect users to A/B testing variants or country specific content. Modifying responses is also one of the possibilities, including creation of HTTP / server-side cookies. The Lamda@Edge functions are deployed on different locations in edge network, thus will execute as close to the user as possible.

Lamda@Edge functions can be execute on the following locations;

- Viewer Request
- Origin Request
- Origin Response
- **Viewer Response** > We will use this event to inspect the viewer request headers and set a cookie in the HTTP response. 

![first-party-cookie-http-response-lambda-at-edge.png](https://static.stacktonic.com/first_party_cookie_http_response_lambda_at_edge_8213d24a81.png)

*More information can be found in the [AWS documentation](https://docs.amazonaws.cn/en_us/lambda/latest/dg/lambda-edge.html)*

## Why use Lamda@Edge to set a server-side cookie?

Setting a first-party cookie containing an UUID user identifier directly in the response of the HTML page requested has several benefits:

- No dependency on client-side Javascript to make an additional request to a service that sets a HTTP cookie. 
- Using such a mechanism mentioned above could create dependency issues; when a service is expecting the first-party UUID cookie but the client-side cookie service has not returned a response yet.
- You can use the first-party cookie in server-side tag management solutions like Google Tag Manager Server, without the need of this tool to be responsible for creating the cookie (since it can be blocked)

## Deploying the Lamda function

*See [the AWS documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-edge-how-it-works-tutorial.html) for a more in depth tutorial.*

Steps:

We assume that you already have a website running in AWS Cloudfront.

1. Create a Lamda function in region `us-east-1`

> It's **important** that you create the Lamda function in the **us-east-1** (N. Virginia) region. When deploying the function to Lamda@Edge it will distribute the functions to all the different locations in the Edge network. Therefore debugging Lamda@Edge function can be difficult, because it generates logs in the region where the Lamda@Edge function is triggered (as close to the end user as possible). So when your located in Germany, most probably the logs can be found in region eu-central-1

2. Download the `index.js` and `package.json` from our [GitHub repository](https://github.com/stacktonic-com/aws-lamda-at-edge-set-uuid-cookie).
3. The Lamda function has some dependencies like the `uuid` and `cookie` packages. You will need to install the packages first. Run `npm run install` in the folder where you placed the package.json and index.js file
4. Create a ZIP file of `index.js`, `package.json` and the `/node_modules` folder (created after installing the packages).
5. Upload the ZIP file to AWS and deploy the Lamda Function
6. Add a trigger to the Lambda Function. Choose "CloudFront" and when configuring this trigger, select the "Viewer Response" event

![cloudfront-lambda-viewer-response.PNG](https://static.stacktonic.com/cloudfront_lambda_viewer_response_36e7c0a3e7.PNG)

- It will take some time (seconds / minutes) before the Lambda function is deployed to all Edge locations.
- When deployed succesfully, you can check if the cookie is set in your browser / developer console (also check that the UUID value remains consistent when navigating to different pages):

![first-party-http-cookie.PNG](https://static.stacktonic.com/first_party_http_cookie_2a251ee586.PNG)

**Some last tips**

- When deploying a new version of your function, **make sure** you also deploy the function to Lamda@Edge again (step 6). When deploying again, you can select "Use existing CloudFront trigger on this function" in the CloudFront trigger settings.
- As mentioned, you can debug the logs of a Lamda@Edge function only in the region where the function was invoked. 
- By default, the IAM role attached to your function will not properly create logs in other regions than us-east 1 (see [this thread](https://stackoverflow.com/questions/46503108/lambdaedge-not-logging-on-cloudfront-request)). 
- Solution is to adjust the IAM role of your Lamda function to;

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:*:*"
            ]
        }
    ]
}
```

## The Node.js Lamda function

The AWS Lamda code ([view full repository on GitHub](https://github.com/stacktonic-com/aws-lamda-at-edge-set-uuid-cookie)). The `package.json` is available on GitHub.

```javascript
// Lambda@Edge function, (re)writing first-party user UUIDs
// Author: Krisjan Oldekamp / Stacktonic.com
// Email: krisjan@stacktonic.com

'use strict';

const { v4: uuidv4 } = require('uuid');
const cookie = require('cookie');
const cookieName = 'FPD'; // Cookie name containing first-party identifier
const cookieLifetime = 730; // Cookie expiration of user UUID in days (2 years) 

exports.handler = async (event, context) => {
    
    const request = event.Records[0].cf.request;
    const response = event.Records[0].cf.response;

    try {
    
        const requestHeaders = request.headers;
        const responseHeaders = response.headers;

        // Generate random UUID.
        let uuid = uuidv4();
        let setCookie = false;

        if (requestHeaders.cookie) {
            // Get all cookie information from request headers
            let cookies = cookie.parse(requestHeaders.cookie[0].value);
            
            // Check if user UUID cookie is present and get existing UUID value from requestheaders.
            if (cookies[cookieName] !== undefined) {
                uuid = cookies[cookieName];
            } 
            
            // Check for presence of user UUID cookie. If not present, (re)write cookie
            if (cookies[cookieName] === undefined) {
                setCookie = true;
            }
           
        } else {
            setCookie = true;
        }

        // Set or extend cookies when neccesarry.
        if (setCookie === true) {
            let domainTld = (requestHeaders.host[0].value).split(/\./).slice(-2).join('.');

            // Build user cookie string with user UUID.
            let cookieStringUser = cookie.serialize(cookieName, String(uuid), {
                domain: '.' + domainTld,
                path: '/',
                secure: true,
                httpOnly: true,
                sameSite: 'lax',
                maxAge: 60 * 60 * 24 * cookieLifetime 
            });

            // Set HTTP cookies in response.
            responseHeaders['set-cookie'] = [{
                key: 'set-cookie',
                value: cookieStringUser
            }];
        }
        return response;
    } catch(err) {
        console.log(`Error handling response: ${err}`)
    }
    return response;
};
```


Hosting your website on AWS S3 / CloudFront? Deploy a Lambda@Edge function to set a persistent first-party cookie that can be used to measure website user behavior over different visits, without the use of any Javascript / client-side scripts. 

medium_edge.jpg

small_edge.jpg

thumbnail_edge.jpg

edge.jpg

How to set a persistent UUID cookie using CloudFront and Lamda@Edge

Cloud (data) platforms such as AWS, Google Cloud or Azure enable you to build a marketing data platform that doesn't only ingest and join data, but can also produce actionable insights and activate data in other tools and platforms. 

Building a cloud Marketing Data Platform

In many organizations, most data is already ingested and transformed in a data warehouse. So there is a lot of overlap in both data and functionality when implementing a completely stand-alone CDP tool next to an existing datawarehouse. We like the concept of a CDP that consists of multiple tools and services, Lytics is trying to solve this in their newest product feature (open beta). Would like to see more functionality like this from other vendors.

CDP vendor Lytics claims it will deeply integrate with your existing datawarehouse

Google Analytics 4 is more than just a measurement and reporting tool. The event data exports to Google BigQuery (available in the free version) and the GA4 API brings a ton of new possibilities. Think of advanced analysis, joining data with other sources, building prediction models or integrate with other systems.

Google Analytics 4 Advanced

Google confirmed that the removal of third-party cookies within the Google Chrome browser is delayed until late 2023. Google has accommodated several of it's new privacy related technologies, like FLoC (Federated Learning of Cohorts) within the so called "Privacy Sandbox". Unfortunately for Google, it seems that it's [not getting the support](https://www.theverge.com/2021/4/16/22387492/google-floc-ad-tech-privacy-browsers-brave-vivaldi-edge-mozilla-chrome-safari) needed.

This delay however, will give more time to to adapt to the "death of cookies", however we strongly advise to adapt now and not in late 2023.
