Top Requirements for Achieving Continuous Data Integration

February 6, 2018

“Data” has become more than just a filler word, and, instead, an engraved term in the minds of recruitment teams as they launch searches for the next data-driven CMO or Digital Marketer at their respective organizations. With a growing urge to iterate marketing campaigns in real-time, marketers are relying upon larger tool stacks and an endless set of dashboards full of performance metrics to monitor any sudden changes. This is no easy feat.

You may ask yourself, “How can a marketer do this and possibly succeed?” The answer is not simple, but it is possible. Sleep? Not so much.

Over my tenure at Treasure Data, I’ve managed all integration related efforts. From getting campaign metadata into Treasure Data from Salesforce Marketing Cloud, to how to export a list of users into the Google DoubleClick Data platform to activate a campaign. I’ve heard it all. The knack for getting data into and out of a centralized data hub is ever-pressing across our standard product persona of a digital marketer.

While this remains true and essential in their day to day, there is a key use case being forgotten: continuous data integration.

As digital marketers are eager to get data in and out as quickly and seamlessly as possible, they continue to make a distinction between input and output integrations. “What systems does Treasure Data integrate with?”, “Are they for data coming in or out?” Common questions during routine sales call eavesdropping that I often stumble upon.

This also remains a common misconception and product development principle applied at many companies seeking to tackle the “integration issue.”

The truth is, there is no business added value in simply viewing data as a single flow or direction.

To best explain this concept, I will use a real life example. Say you are a Data Analyst tasked with analyzing the data living within your company’s marketing automation system. You are asked to bring into the Treasure Data Customer Data Platform (CDP) all lead activity data. Your marketing team is then going to use this data to create an audience of leads that have been stale for over 90 days. Their hope is to find hidden gems to target via their next campaign. However, this will lead to the same outcome…stale leads.

In an ideal world, you should be tasked with building an audience of stale leads, while also capturing back all campaign performance results to ensure the data is actionable.

This is continuous data integration, or viewing an integration as more than just an in or out, but a bi-directional movement of data that has a direct business impact. By applying this concept to your marketing efforts, you’d understand that your task is far greater than just moving data from one system to another. It is to add value by acting upon data in real-time to attain the highest marketing performance results.

This goal sounds great on paper, but not easy to implement without the proper tools. While I’d hate to turn this blog into a sales pitch, I will simply highlight some of the key functionalities your system of choice should have.

A secure authentication method, preferable OAuth. With access to data comes the responsibility to protect that data from outside attacks. Having an OAuth flow in place allows you to connect your systems without much of a worry of the data being compromised. Take it a step further and review your vendor’s certifications, SOC 2 as an example.

Flexible schema ingestion. Getting data in and out of a system may prove to be a lot of work if much of the data preparation has to happen outside of that system. You’d want to ensure that most of your time is spent on analyzing and acting upon the results, and not preparing the data. Most notably, a schema-flexible system allows you to save time as you continue to add other tools onto your technology stack.

Incremental loading. As much as you’d like to avoid doing data preparation to ingest your data into the right schema, you would also enjoy not having to dedupe your data daily. With incremental loading, you can simply do an initial historical data dump, and then schedule all future transfers to load based on your last update. Take Salesforce CRM as an example, with incremental loading you don’t have to bring in all millions of contact records into Treasure Data daily, instead, only ingest the last 100 records added to your CRM since your last upload.

Mode flexibility. This may sound highly technical, but it is not. This simply means whether you can append, replace, or update your data as it lives in a table. While this may be a nice to have for some, I recommend this as a must have. You’d hate to not be able to update existing contact records in bulk after cleansing your CRM or marketing automation system.

While I am sure there are plenty of functionalities you may be considering for your Customer Data Platform, keep these in mind as essentials.

Most importantly, understand the value of seeing an integration continuously versus simply getting data in and out of a system. The latter can lead to poor results and just a day-to-day activity. The former can lead to significant ROI as you iterate in real-time. Plus, wouldn’t it be great if they said your name during the next round of promotions?

Recommended reading: