‘We don’t have a Martech problem, we have a change problem’ Scott Brinker
Today I had the pleasure to hang out with MinneAnalytics, the country’s largest community-based analytics group in their free Boston event. Thanks to my wife, who enrolled and alerted me, and organizer Dan Atkins from Optum, who put me on the late list! Below my own top 5 insights about unicorns and data scientists:
Scott Brinker from Hubspot never fails to impress on Martech; the management of technology in marketing. The barrier holding us back is not technology, it’s management: how to operate marketing with this new technology? This is especially tough as the Marketing technology landscape continues to expand instead of consolidate: from about 150 Martech in 2011 to 5000 in 2017 (28% just last year). On the plus side, what $100K bought you 10 years ago, now buys you so much more. Faced with this diversity, marketing managers typically fall for the one-stop shopping temptation from a large provider, but then realize they still need to make it work, reconfigure their marketing team, motivate and train. While technological change is accelerating, organizational change is at bet linear. Just improving this a bit in an agile organization will help you outrun your competitors. Beyond hiring data scientists, it is key to focus on improving your people’s skills to leverage these tools and to give them the freedom to use them in agile development (only 20% of tech initiative solutions are currently delivered in under 6 months). For instance, Microsoft uses the ‘infinite loop’ of the customer journey as the organizing force in their Martech Stack – submitted to the ‘Stackie Awards’:
Talking about Microsoft, Francesca Lazzeri gave a detailed account of how they build deploy AI models, at scale in particular Deep Learning Deployment. From descriptive to predictive and prescriptive, AI drives business value from demand forecasting to HR analytics. As to the former, Microsoft helped a retailer predict market and customer trends, optimize its marketing allocation and adapt predictive tools. As to the latter, they used 5 years of project transaction data to optimize staff assignments to new incoming projects. To a question about a counterfactual though (what happens if we increase price?) she responded deep learning is great to compare scenarios, but answering ‘what-if’ questions (still) required statistical tools such as regressions.
As the gold standard of multi-equation regressions in economics and business, Vector Auto- Regressive (VAR) models are the ones to beat for neural networks. Jeffrey Yau from Alliance Bernstein explained first why standard feed-forward neural networks (eg. Random Forests) don’t work with time series data: information flows only forward, inputs are assumed independent, and there is no ‘device’ to keep past information. In contrast, Recurrent neural networks (RNN), often applied to language processing, have a hidden layer including the past hidden state. Specifically , the Long Short Term Memory (LSTM) architecture (Hochreiter and Schmidhuber 1997, Greff et all. 2016- see above pic) could outperform in forecasting, provided you scale the series right, define the right network and run many options – ‘painful’ is how Jeffrey described it. In contrast, VAR models were fast, easily implemented in Python (the VARMAX routine), and achieved out-of-sample forecasting error (RMSE) of below 4% for beer sales and 0.22% for the Customer Confidence Index – even with stationary assumptions (which I believe should be relaxed given below pic: difference the evolving series and estimate the VAR model!).
How to visualize your wonderful analytics results? Mark Schindler from GroupVisual.io evolved from architect to dashboard visualizer, keeping the user and her purpose front and center in abstracting, narrating and interacting data. Inspiring books include Jonathan Feinberg’s ‘Modern art at the border of art and brain’ and Steven Pinker: ‘The stuff of Thought’, which defines human intelligence as the combination of concepts and metaphorical abstraction. For instance, time is abstract, so we need to make it concrete by e.g. using special metaphors such as ‘Fall is ahead of us’ and our brain region related to ‘space’ lights up. Abstraction is wonderfully illustrated in the first episode of the ‘Abstract’ Netlfix series with Chris Neiman, and Gapminder’s visualization of how countries improved their wealth and citizens’ life expectancy: https://www.gapminder.org/tools/#$state$time$value=1808;;&chart-type=bubbles
What could possibly go wrong implementing your data science model? was the question answered by dr. Ingo Mierswa, who moved his German company RapidMiner to Boston 5 years ago. Of the 105 million ‘information workers’ identified by Tableau, only a fraction are data scientists, and even less are the ‘unicorns’ who can solve any business problem with a PhD in stats and data science experience. Possible solutions include 1) finding ‘citizen data scientists’ or 2) make existing data scientists more productive. Automated machine learning appears to achieve both by offering ‘Data Science in a Box’ but this easily leads to horrible results – as Ingo illustrated in his own customer churn project where a $120 M projected gain turned into a $ 15 M loss! The reasons included imprecise validation, resulting in a claimed 96% accuracy versus an actual 87% accuracy in the hold out, and the selection of a feature (recent customer complaint calls) that was not timely available in the implementation. To avoid this, Ingo favors opening the black box so a business problem owner or citizen data scientist can debug it. His software highlights variables that should not be used (eg a customer’s phone number) in red, and in yellow variable that are suspiciously close to the target variable, such as recent customer complaint calls.
Thanks for a great conference, and see y’all next year back in Boston!