Scaling Marketing Data Pipelines

This blog is going away soon! :( Check out my new site where you can read the latest and subscribe for updates!

Lessons from Site Reliability Engineering: Part 2 #

If you’ve ever felt overwhelmed by the choices of vendors and tools available for data-driven marketing automation, you are far from alone. The need for operational, time-sensitive data pipelines is skyrocketing, mostly driven by the demand for fast response times expected by most customers.

When you break the problems/solutions of data processing down to it’s components, there are three phases of evolution that data pipelines go through: Entry-Level, basic functionality provided by many platforms; Mid-Level, usually custom scripted and implemented in-house; and Big Data, which turns marketing automation standards on their heads in order to innovate.

Operational / Time-Sensitive vs. Reporting #

Before we dive in, it’s important to understand the two types of data pipelines typical to marketing operations. (However, the paradigm is not exclusive to marketing data).

While reporting pipelines are essential to long-term marketing optimization, our focus here will be on operational pipelines. These are defined as any time-sensitive data flow on which day-to-day business depends.

The example I most commonly use is a “Contact Sales” web form: when a visitor wants to purchase your products or services, they fill out this form, which then hits a data intake system. Then, multiple processes fire off to create an appropriate lead in a CRM system, route to the appropriate teams, send a confirmation email to the visitor, and possibly drop in to a related cross-sell/up-sell nurture flow.

The key difference here is the downstream impact of pipeline issues. If a dashboard user notices an issue with the business logic traced back to an ETL, it can easily be fixed in the next scheduled release. If a “Contact Sales” submission falls through the cracks, it results in immediate lost revenue for the business. For these pipelines, it’s essential to have a reliable development, testing, and release process, as well as monitoring to ensure data keeps moving.

Let’s look at a few examples of reporting and operational data pipelines:

Reporting:

Operational and Time-Sensitive:

Now we can dive in to the progression of operational data pipeline tools.

Entry-Level #

Basic data flow tools are natively available in most marketing automation platforms (MAPs), usually as a drag-and-drop interface with a standard set of data selections:

Depending on product and configuration, they offer anywhere from near-real time batch(<2min) to macro-batch (>15min) processing times. All processing takes place “behind the scenes” within the MAP.

Pros:

Where scale and reliability become issues:

Mid-Size #

The next step is often an in-situ transformation - data is exported (on a scheduled basis) via an API, a scripted process applies transformations (or integrates with other systems, such as a CRM), and at the end of processing the data is returned (in either a modified or enriched state) to the marketing platform.

Processing can occur at nearly any range, from minutely to daily, usually with the limitation of streaming/real-time processing being excluded.

Common tools include:

Pros:

Where scale and reliability become issues:

Big Data #

For groups wanting to be exceedingly innovative, the next logical step is using a big data platform. In this case, the big data platform becomes the focal point of the data, with the MAP being an integration on the periphery.

Pros:

Where does Marketing Automation live? #

Transitioning to big data comes with a fundamental shift in marketing automation architecture. In the Entry-Level and Mid-Size solutions above, the MAP is at the center of the data, acting in it’s own as a data management platform (DMP).

Big data requires infrastructure and flexibility that most MAPs can’t keep up with, so we keep the MAP doing what it’s good at: usually, this is sending and tracking emails. The end result is the high-level architecture shown below:

Target big data architecture with marketing automation

This design has many benefits, which I hope to dive into in future articles.

Where are the microservices? #

Microservice APIs are all the rage these days - especially in context of moving away from monolithic applications (i.e., marketing automation platforms). While I plan to address microservices in relation to data engineering and marketing automation later, I’ll give a few high-level points here.

Microservices are helpful for:

But are less useful for:

Conclusion #

The landscape of data pipelines is rapidly changing, and marketing automation experts must keep up to meet business demands. Operational pipelines are time-sensitive data transformations and integrations which keep the business moving day-to-day.

At the Entry-Level, most Marketing Automation Platforms provide the necessary tools for basic business processes. Companies who grow more complex usually require pipeline development outside of their MAP. Those seeking to use data in a game-changing way transition completely outside the MAP to a big data platform.

Feel free to connect with me!

 
0
Kudos
 
0
Kudos

Now read this

Spark Summit 2017 - Day 2 Takeaways

This blog is going away soon! :( Check out my new site where you can read the latest and subscribe for updates! Day 2 was a bustle of activity at Spark Summit 2017. I started the day catching up with a bit of work at Workshop Cafe in the... Continue →