How Marketing Automation Works
Dec 10, 2018, updated Apr 3, 2019 | 44 minute read
Dec 10, 2018, updated Apr 3, 2019 | 44 minute read
Today’s world, thanks to new technologies we get to use everyday, changes a lot. Computing power increases, new solutions emerge and various systems we use are getting more and more complicated.
Marketing is no different. Modern marketing tools make teams work faster and more efficient in reaching their audiences. Internet and computers enable reaching huge amount of people in a scale never experienced before.
“How marketing automation works” is a technical description of marketing automation systems and solutions which greatly improve effectiveness of marketing efforts on the Internet.
Start with introduction or jump straight to technicals.
Word “marketing” appeared in texts in sixteenth century and described buying and selling wares on the market. We don’t know exactly when humans began marketing, but lot of sources point us to ancient Greece and Rome, where in 1500 B.C. products were transported in amphoras. Those amphoras had various shapes and incrusted markings, which helped differentiate origins, quality and authenticity of products.
Arrival of trading ships was announced by port “screamers”, shouting around the crowd where these ships are coming from. Oftentimes screamers performed with musicians, creating distinctive commercials. In Pompei, archeologists found commercials on building walls dated to 100 BC. They were promoting political candidates, theatre plays, sport competitions or taverns.
Marketing and commercials in middle ages were not very different from those in ancient times, but some “brands” survived today - for example barber’s pole.
In 1450, thanks to Gutenberg printing invention, marketing experienced huge growth. Printing machines were producing leaflets and brochures for massive campaigns and not much later newspapers offered their pages for advertising. Pensylvannia Gazette in 1730 published ads for slaves:
May 3, 1733. [Advertisement] There is to be sold a very likely Negro Woman aged about Thirty Years who has lived in this City, from her Childhood, and can wash and iron very well, cook Victuals, sew, spin on the Linen Wheel, milk Cows, and do all Sorts of House-work very well. She has a Boy of about Two Years old, which is to go with her. The Price as reasonable as you can agree. And also another very likely Boy aged about Six Years, who is Son of the above said Woman. He will be sold with his Mother, or by himself, as the Buyer pleases. Inquire of the Printer.
In 1836, La Presse daily newspaper from Paris for the first time in history placed ads on its pages which reduced the price, expanded reach and in consequence increased revenue. Soon, all other titles took this path.
In 1920, more and more popular radio stations broadcasting 24/7 started to play commercials. In 1940s first TV commercials appeared, and only ten years later telemarketers were pissing off people all around the world.
Marketing activites up to the end of XX century were difficult to measure. Only datapoints available to marketers were newspapers circulation and estimated number of listeners and viewers. Marketing in mass media was expensive and the only way to reduce the cost was to publish ads in smaller publications like themed magazines or local radio stations.
Measuring effects itself was difficult, too. Running marketing campaign at given time would increase sales later, but reports were not always precise - sales were dependent on many more things than only advertising, like fashion or economic situation.
Biggest headache were multichannel campaigns where it’s impossible to pinpoint sales increase to particular medium. After running campaigns in TV, radio and newspapers, advertisers have seen cumulative sales increase without any option to separate effectiveness of each channel. This situation persisted until the beginning XXI century, where Internet advertising enabled very detailed measurement of ads performance.
Internet changed marketing forever. In 1994 HotWired.com (Wired.com today) was the first to sell advertising space on website - this event is considered as the beginning of online marketing. Twenty years later, in 2014, marketing spending for Internet advertising reached 135 billion USD and it is estimated that in 2019, it will be higher than spending on TV advertising.
First Internet ad
Internet advertising was a great success for two key reasons. First, ability to almost immediately send marketing communication without geographic or time limitations. Secondly, much bigger ROI compared to traditional marketing methods - modern adtech networks let you target your ads very precisely only to people interested in your offer.
Worldwide Internet access enabled small companies to go global - today, company website is like a storefront available for everyone in the world. Lots of companies exist only on the Internet.
Advanced analytics systems gather data to show you the most relevant ads possible. They follow and spy on you every second you’re using Internet, creating a profile of your activities and interests.
Apart from that, analytics let advertisers precisely measure effects of their campaigns. Website owner today knows which ad you have clicked and if you have bought the product or maybe you’ve just left after 3 seconds. This is invaluable information about trends, consumers and market demand.
Nowadays, analytics tools are advanced, multipurpose, huge pieces of complex software providing you all sort of metrics and reports. Google Analytics is the most popular one - it’s installed on over 50% of websites around the world. List of information gathered by Google Analytics is impressive:
All of these together can answer important business questions, ex.:
Having answers for these questions we get huge advantage over competition. Biggest companies invest a lot into understanding customers and their demands to optimize offer and maximize revenue.
Despite all of the possibilities above, analytics tools still have huge limitations. The first and foremost is no information about who is visiting your website. You don’t know your visitors and you have no way to get in touch with them.
Since marketing is getting more and more complicated, it is nothing new to run campaigns in multiple channels at once. Also, now people visit your website from multiple devices. Imagine a person visiting your website from phone and later, at home, from desktop computer. Analytics has no idea who is sitting behind those devices and it has no tools to recognize the same user behind different devices. For analytics, these two visits are two separate visitors.
Similar errors in reports are common and can result in bad decisions made by marketers. Let’s take an example: customer visits product website during dinner from his smartphone, and buys the product later in the evening using his desktop computer. Marketer can assume that ads targeting mobile users are not effective (no purchases) and takes it down, but it’s actually a key step on customer journey leading to purchase.
Mentioned earlier, another limitation is no way to get in touch with visitor. Analytics is just passive observer, noting down what it sees. What if you could dynamically modify website content to show different groups of users information they are most interested in?
Constantly increasing number of marketing activities in companies makes it impossible for smaller teams to manage their work effectively using only human work. But that’s what computers are for - we can automate all repetitive stuff which not only makes marketer’s work easier, but often is faster and more comfortable for customers.
Advanced marketing automation systems, based on defined rules, user activity and information from external sources (ex. sales CRM) can personalize messages and content for every person, bringing more value without real-time human to human interaction. These systems are taking workload off people, taking care of simple things like onboarding emails or managing access to downloadable materials.
They also provide more information about users themselves, recognizing the same users logging in from different devices, which makes traffic analysis and revenue sources tracking more precise.
The simplest explanation how automation works would be “if X, then Y”. These rules can contain singular “ifs” (ex. if contact name equals “Patryk”, send email with subject “Hello, Patryk”) or more complicated set of multiple rules. Table should help:
| Rule | Possible result |
|---|---|
| Contact was sent an email | Yes / No |
| Contact opened email | Yes / No |
| Contact clicked link in email | Yes / No |
| Contact property | Is equal / Contains / Is not equal / Does not contain / Is bigger than / Is less than / Begins with / Ends with / Is known / Is not known / Was ever / Was never / Ever contained / Never contained |
| Contact visited URL | Yes / No |
| Contact visited URL (regex) | Yes / No |
| Contact belongs to segment | Yes / No |
| Contact did X at Y | Precisely at / Before / After / Between / More than / Less than |
| Contact filled out a form | Yes / No |
| URL where something happened | Is equal / Is not equal / Contains / Does not contain / Begins with / Ends with |
| Something happened X times | Number |
| Contact did something | Yes / No |
Rules above can be chained freely, creating infinite number of combinations. For every contact meeting requirements of combined rules we can define a set of actions to perform:
Let’s see how all of these above can create real life examples:
No 1:
If all requirements below are met:
Perform actions:
No 2:
If:
Perform actions:
No 3:
If all requirements below are met:
Or, if all requirements below are met:
Perform actions:
As you can see, this can be a lot of help in everyday work, reducing need for bigger marketing operations teams.
Of course, marketing automation is not all sunshine and roses. First of all, it’s complicated, time consuming to set up and fairly expensive. Amount of options is overwhelming for users and requires months of trainings and practice to effectively work with automation. This makes the team less effective in the first months after implementing marketing automation systems.
Marketing automation is usually sold in SaaS (Software as a Service) model with monthly payments. Depending on number of contacts in database, cost of the system can reach thousands of dollars per month, without upper limit. There are self-hosted and open source solutions like Mautic, but they often require some technical staff to maintain it and beefy machines.
Automation can be insidious, too. Marketers often dive into it too much, launching countless of workflows, sequences and emails to poor visitors. It’s not unusual for new contacts to receive dozens of spam-like promotional emails right after signing up, sometimes daily for months. It achieves nothing but user irritation, doing disservice for everyone involved.
Measuring productivity is not easy. How do you know if new marketing automation is influencing revenue positively? Attributing revenue is one of the hardest thing in the industry and without good integrations between marketing and sales systems it’s even more difficult. If you lack in IT department, it might be even impossible.
Despite their shortcomings, abilities of marketing automation systems let businesses achieve very high return on marketing investments. They are usually pretty complicated though, both for end users and developers working on them. Most marketing automation systems can be divided into three modules: analytics, segmentation and actions.
Analytics module has two main tasks: recognize user and track his activities. There are multiple implementations of web analytics available, ex. Google Analytics mentioned earlier. I will show you how analytics module works using Ahoy.js library created by Andrew Kane and Google Analytics documentation. Ahoy.js works completely on client side (user browser).
Let’s start with launching Ahoy. Include it on a page and call the most basic tracking function:
<script src="ahoy.js"></script>
ahoy.trackAll();
Cookies are small chunks of data saved in browser which remember various information for visited websites, like e-commerce cart content or logged in user. Analytics module uses cookies to identify users (visitors) with unique ID number. This number, or rather string, is generated during the first visit and attached to user device and activities (page views, clicks etc.) in subsequent visits. Cookies simply recognize who you are on website. Ahoy.js uses generateId() function for this:
function generateId() {
return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, function(c) {
let r = Math.random()*16|0, v = c == 'x' ? r : (r&0x3|0x8);
return v.toString(16);
});
}
generateId()is based on RFC412220 and, despite not guaranting 100% uniqueness, probability of collision is so small that we can ignore it.
It’s important to know the difference between user (visitor) and visit. Visitor is a device (computer, phone) with saved unique ID number valid for two years since last visit, while visit is group of all user activites during certain time period. This time period is called session and it may last 30 minutes (in Google Analytics) or more - Ahoy.js’ session lasts 4 hours.
Ahoy.js contains createVisit() function, which saves new and recognizes returning visitor with cookies named ahoy_visit and ahoy_visitor. First, it checks values of those cookies - if both exist, it means that user is known and his session (visit) is still active (current time is within 4 hours since first activity).
If ahoy_visit does not exist, it means session lasted 4 hours and its cookie expired. Then, Ahoy.js creates it again with current time as beginning of visit.
If both ahoy_visit and ahoy_visitor cookies don’t exist, it means that user has not been recognized - it’s a new visitor. New cookies are being created using setCookie():
export default {
set: function (name, value, ttl, domain) {
let expires = "";
let cookieDomain = "";
if (ttl) {
let date = new Date();
date.setTime(date.getTime() + (ttl * 60 * 1000));
expires = "; expires=" + date.toGMTString();
}
if (domain) {
cookieDomain = "; domain=" + domain;
}
document.cookie = name + "=" + escape(value) + expires + cookieDomain + "; path=/";
}
}
After reading or creating cookies, Ahoy gets more information: current URL, window size (resolution) and referrer. This is being sent as JSON POST request to /ahoy/visits/ using sendRequest(), where backend server (which I will introduce later on) is taking care of computing the info and saving it in database.
Payload sent by Ahoy can look like in the following example:
{
visit_token: "ec0f3a70-4489-417f-bf9b-3142ccf9b738",
visitor_token: "2aaabcc2-b838-4d90-a237-df33ab8e0780",
platform: "Web",
landing_page: "https://www.example.com",
screen_width: 1280,
screen_height: 720
}
At the beginning, we invoked function trackAll(). It does nothing by itself, but rather makes use of more detailed helper functions: trackView(), trackClicks(), trackSubmits() and trackChanges(), which generate so called properties - information about user activities (events):
trackView() saves visited pages
properties:
http://example.com/welcometrackView() can contain other custom properties like blogpost category
trackClicks() tracks links and buttons clicks
trackSubmits() tracks form submits
trackChanges() tracks changes in input, textarea and select fields
All of the above invoke general track() function with two arguments: name and properties. track() creates event object with following information:
ahoy.track = function (name, properties) {
var event = {
id: generateId(),
name: name,
properties: properties || {},
time: (new Date()).getTime() / 1000.0
};
}
name can be $view, $click, $submit or $change. Event object is now ready to be saved. It’s being sent to /ahoy/events/ with Web API (navigator.sendBeacon ) or, if it’s not available, as asynchronic AJAX POST request.
While analytics module functions on client side using Javascript, segmentation is implemented on backend side (on server). Segmentation module can be written in every programming language, but here I will use Ruby, Ruby on Rails and official Ahoy backend gem for Ruby - ahoy_matey, which provides me some nice scaffolding to save and manage tracking data.
Ruby on Rails app working as a proof of concept segmentation server is available here: marketing-automation-rails. Following examples will be based on this app.
Segmentation module is responsible for saving registered user activity to database and, given previously saved data, adding them to segments based on their activity.
Let’s assume a brand new visitor goes to our website and client library sends two requests to our server:
/ahoy/visits, containing visit ID, visitor ID and additional information described earlier/ahoy/events, tracked activity, in this case data from trackView() function{
"events" => [
{
"id" => "e85b59eb-8266-4fad-bbed-9f10eb03b4ab",
"name" => "$view",
"properties" => {
"url" => "http://localhost:3000/",
"title" => "Automation",
"page" => "/"
},
"time" => 1502043123.768
}
],
"visit_token"=>"4ba9c74b-c0d3-4e2b-a4ff-8a86b07e7a80",
"visitor_token"=>"e02c51a0-5dfb-4483-a162-f8f6aa704243",
"event" => {}
}
Received data is being saved to database in visits table in columns:
Data for each event is being saved in events table in columns:
Notice that events table has no columns for each property, ex. page URL or title. Instead, properties column contains all information saved as JSON:
{
"url":"http://example.com:/",
"title":"Example site",
"page":"/"
}
This structure is needed for system to be flexible. Imagine more sophisticated tracking with hundreds of properties - if every one would need a dedicated database column, everything would easily go out of hands, slowing down database (and subsequently rest od the app) to a crawl. Modern databases have no problem with working on JSON strings.
user_id is also saved. It’s an unique ID of currently logged in user (if your tracking distinguishes them), which enables tying all activities to given person and therefore gives me ability to add particular user to segment later.
Segments are group of users who meet defined criteria. Segments are automatic - users are added to segments once they meet criteria and removed from them when they don’t meet them anymore. Example criteria for segment might be “user visited /contact page”. or “user email contains ‘@gmail.com’”
Let’s see how segments are implemented in marketing-automation-rails:
To create a segment, you need a name for it (segment.name) and some criteria who should belong to this segment (segment.filters). These criteria (filters) are bunch of conditions in two-dimensional tables. Every rule describes properties of event which will qualify user as meeting this rule.
In our case rule is a Ruby Hash containing three keys: name, match and properties. This is how it looks for rule “user viewed /services page”:
filters = [
[
{
name: "$view",
match: "=",
properties: {
page: "/services"
}
}
]
]
name defines which type of event should we look for and takes values from the list:
match describes how are we going to compare the values (this is important):
= (exact match),~ (contains),^ (begins with),$ (ends with),> (is greater than - only for numbers),< (is less than - only for numbers),!= (is not exact),!~ (does not contain),!^ (does not start with),!$ (does not end with)properties defines one property to be searched for among events.
This single rule above contains enough information to create a segment of users, who visited /services page. Exact match throws out all pageviews with additional characters in URL, like URL params. Users who visited /services?source=banner won’t be added to this segment.
Defining segments based only on one rule often is not enough, that’s why you can merge conditions. Let’s say we want to create a segment of users who visited /services page and another page with title containing offer OR visited page with URL param source=banner.
I defined two groups of criteria, and user meeting any of them will be qualified to join my new segment. For the first group we create two conditions:
{
name: "$view",
match: "=",
properties: {
page: "/services"
}
},
{
name: "$view",
match: "~",
properties: {
title: "offer"
}
}
Second group is defined by single rule:
{
name: "$view",
match: "~",
properties: {
url: "source=banner"
}
}
All conditions merged into filters table look like this:
filters = [
[
{
name: "$view",
match: "=",
properties: {
page: "/services"
}
},
{
name: "$view",
match: "~",
properties: {
title: "offer"
}
}
],
[
{
name: "$view",
match: "~",
properties: {
url: "source=banner"
}
}
]
]
Filters table contains two another tables describing criteria, which contain conditions. To pass criteria, user needs to meet every rule inside it. To pass filter (and qualify for segment), user needs to meet only one of all criteria.
It might seem complicated, so I prepared a small graphical diagram:
After defining conditions and criteria for segment, we can now analyze previous activity of users and add them to segments they qualify for. This process is called segment building i consists of several steps.
At first, server reads the conditions and generates lists of users meeting criteria for joining segments. The code which does this is in BuildSegment class. The class has several functions which called one after one build list of users meeting criteria for single conditions, whole filters and add or remove users from segments.
To begin building a segment, we have to call Segment.build for selected segment, then server reads filters and conditions, then BuildSegment::FindUsersToAdd builds database query based on saved conditions. Query building is managed in query function:
match rule we look for and if its negative match (contains !)@conditions_hashAhoy::Event class and functions created specifically for segmentation which extend original Ahoy code. find_users functions query database for events meeting conditions.After getting lists of users for every rule in given filter, users_passing_all_rules function creates another list of users - this time only those, who are present on every other list containing user meeting conditions. This is how we get users, which activity meets criteria for the whole filter (and therefore criteria for joining segment) - users_passing_filter.
users_meeting_requirements_for(segment) function builds final list of users for given filter.
Last step is to finally add those users to segment - and saving this fact in database. add_users adds new users for segment and remove_users removes users which do not meet criteria anymore.
In current marketing automation systems actions module have various names, but two most used are workflow or campaign. In marketing-automation-rails I have used workflow. In my example app actions module has only one function - send email messages to segments of users.
To find users belonging to given segment, workflows use the same set of filters as segmentation module, but it can additionaly look for users in segments:
filters = [
[
{
name: "$segment",
match: "=",
properties: {
id: "1",
}
}
]
]
Workflows also contain set of actions to be made for every user which joins this workflow:
actions = [
{
name: "$email",
action: "send",
id: 1,
delay: 300
},
{
name: "$email",
action: "send",
id: 2,
delay: 7200
},
{
name: "$email",
action: "send",
id: 3,
delay: 259200
}
]
Actions above will send user three emails with IDs 1,2 and 3. First email will be sent 300 seconds (five minutes) after joining workflow, next one after 7200 seconds (two hours) and last one after three days.
For automatically triggered actions to be launched with proper and precise delay, queue of actions should be created immediately after user joins segment.
In Ruby on Rails framework we can use very nice Active Record Callbacks, which enable us to call functions right after updating segment (which is right after adding users to segment). Using after_update callback we can easily call set of functions to schedule emails.
Every time server builds segment, callback after_update calls build_workflows function which checks if segment update qualified any new user for workflows, which subsequently defines set of actions to be launched for this user. Class responsible for this is called EnrollUserInWorkflow and it launches ActionMailer, which in turn is responsible for delivering messages to user in given time.
Email sending is first and foremost function of every marketing automation system, but more advanced commercial products have lots of interesting possibilities. Workflows can add users to other workflows and segments, creating a chain reaction which gives huge possibilities for very precise user paths based on modular stages of actions.
For different segments we could show popups, notifications or even modify content in real-time using JavaScript. To show user in segment a message, we could for example use similar code:
if (current_user.segments.includes(1)) {
showPopupWindow();
}
By observing and analyzing user activity, we can also apply so called lead scoring, which will qualify users with points for every action. Users with activity suggesting buy intention can be automatically recognized and added to segment and workflows.
D. Twede, Commercial Amphoras: The Earliest Consumer Packages?, Journal of Macromarketing, Vol. 22 No. 1, 2002, p. 98-108
Gary Grossman, Rob Blumenstein, Sean P. Geary, Sell & Spin: A History of Advertising, 1999
William Andrews, At the Sign of the Barber’s Pole: A Study in Hirsute History, 1904, p. 11
S. Bennett, From Print To Social Media – The History Of Marketing, 2012,
La Presse, December 15th, 1836, p. 4,
Pensylvannia Gazette, Selections 1730-1743
Ryan Singel, Oct. 27, 1994: Web Gives Birth to Banner Ads, 2010
PwC, Global entertainment and media outlook 2015–2019, 2015
W3Techs, Usage of traffic analysis tools for websites
Hubspot.com, How to choose your workflow actions
Mozilla Web Docs, HTTP Cookies
