How AI Happens

An Ensemble Approach to Optimization with George Corugedo

Episode Summary

CTO George Corugedo explains how the relationship between physics and math is a model for the relationship between business questions and artificial intelligence, as well as Redpoint Global's ensemble approach to optimization.

Episode Notes

CTO George Corugedo explains how the  relationship between physics and math is a model for the relationship between business questions and artificial intelligence, as well as Redpoint Global's ensemble approach to optimization.

Episode Transcription

0:00:00.0 George Corugedo: That evolutionary programming approach frequently out-performs all of these other common algorithms. However, if we handed that to somebody without the context of those familiar algorithms, they wouldn't necessarily know if they could trust it.


0:00:24.0 Speaker 2: Welcome to How AI Happens, a podcast where experts explain their work at the cutting edge of artificial intelligence. You'll hear from AI researchers, data scientists, and machine learning engineers as they get technical about the most exciting developments in their field and the challenges they're facing along the way. I'm your host, Rob Stevenson, and we're about to learn how AI happens.

0:00:54.4 S2: On today's episode of How AI Happens, our show might be better titled before AI happens, because we're going to take a look at how we determine the questions we want our AI to answer and the importance of making sure those questions are relevant to the people you want to be using your technology. First, we're going to begin with how the language of artificial intelligence shapes the work, not the language of Python or R, I'm speaking of course about math, because before you have exciting new applications, before you have an algorithm, you need linear algebra, calculus, gradient descent, maybe even sprinkle in game theory for good measure. My guest today, George Corugedo is a chief technology officer, a polyglot, and first and foremost, a mathematician.

0:01:45.8 GC: I studied mathematics all through school largely because I wasn't really sure what I wanted to do, and mathematics seemed to be applicable to everything that I could potentially wanna do. So I studied it, ended up with a master's degree in it and figured I'd be a professor, and instead I ended up working at Accenture, did consulting for numerous years, working across numerous industries, learning a lot how to apply the capabilities that I developed in studying math, which really turned out to be very tight logical thinking and ability to solve complex problems, and brought that into the technology world. And at one point we decided to start a software company. And so in 2006, we started RedPoint. The intent of RedPoint, was to build the ultimate marketing platform based on the years we spent at Accenture trying to figure out either how to compose one or how to deliver one.

0:02:56.5 GC: The other part of it is that I actually taught physics for many years as well. And physics is like an entire discipline of solving nothing but word problems in mathematics. So the combination of the two give you a tremendous capability for solving difficult multi-dimensional problems in virtually any setting, because it's all about digging through what that word problem presents to you as information, plucking out the relevant facts out of that and then putting them together in such a way in a formula or some other way that then helps you yield an answer.

0:03:44.0 S2: In the same way physics poses a question only math can answer, George's approach encourages users to pose questions that RedPoint's technology will answer, an approach he believes will remove some of the more common roadblocks we find when attempting to scale and distribute AI.

0:04:00.5 GC: At RedPoint, what we've done is we have approached the problem of delivering AI technology into the hands of our users in a way that reduces the specific skill and experience that is typically associated with utilising these kinds of tools. Historically, when you were gonna work in statistics, which is what AI used to be called, and you were gonna be a statistician, you had to bring a lot of experience. When you took statistics courses, there were always these rules of thumb that you had to remember, and that very much was the skill, the judgment that a statistician or these days a data scientist might bring to a problem.

0:04:53.7 GC: What we've tried to do at RedPoint is reduce the amount of that necessary and let the machine and the horse power in the machine take over that part of the process and allow the users to really just bring the business knowledge and the business question to the tool then translate it into a problem that the tool can solve with lots of horsepower and memory and a bit of optimisation and yield an answer that can be immediately applied for the business' benefit. So that's really what we've tried to do is simplify the application and the use of AI and defeat the roadblocks that some of the modern approaches of AI have created in scaling out the use of AI across a business.

0:05:52.8 S2: What are some of those roadblocks?

0:05:55.1 GC: Well, what has happened since the days of big data and the popularity of Hadoop and data lakes is that analysts and data scientists got their revenge, right? They were always struggling with data warehouses because data warehouses were highly summarised data for reporting purposes. Well, if you're a statistician or a data scientist or work in AI, you realise quickly that summarised data isn't gonna do you any good in developing predictive models. You need the granular behavioral data. So what data lakes enabled data scientist to do was to gather up and collect raw data from the original feeds and throw it into a sandbox that they had control over.

0:06:44.0 GC: Now, that's a wonderful achievement for them in the context of a data warehousing world, but what that requires of them is a tremendous level of skill, first of all, to sort and process all that data into a form that then can be fed into an algorithm. And then further, once you get it into that algorithm, many of the algorithms they use require coding or they use coding methods to deploy those algorithms, and so the entire process of developing a model now becomes a highly skill-based code-driven exercise that is very difficult to scale across an enterprise because these resources are very hard to find and expensive, and so it's very difficult to take AI and push it into every corner of your enterprise that way.

0:07:50.5 S2: Okay, so we know some of the challenges, we know George's math focused approach, time to get our hands dirty.

0:07:57.3 GC: What we did is we used a statistical process, which is algorithmic optimisation, to solve this problem. So one of the key steps that any statistician has to do is to determine what's the best algorithm for a particular problem. That usually requires some understanding of the business problem, of the dataset and its sparsity and its richness or its breadth, and then what type of outcome are you looking for? Are you looking for a yes/no, are you looking for a categorical, are you looking for a continuous variable or a sequential type of output? All of this then plays into what algorithm I'm gonna use.

0:08:42.9 GC: What we do instead is we look at those outcomes, we ask the user to determine what kind of outcome they want, but then we throw half a dozen or more algorithms at that problem and then build maybe 100 or 200 models in the space of a few hours, using those algorithms and permutating different elements in the algorithm so that we have lots to choose from. So you may ask, "Well then, how do we choose the right algorithm?" And that's where the optimisation comes in, because what it does is it sifts through all of those algorithms and basically puts them through an actual simulation process, which is very CPU-intensive, but CPUs are cheap, so we can do that, and it runs and runs and runs those simulations until it finds the very best model.

0:09:47.3 GC: Now, best, of course, is really determined by the business context, so whatever KPI best describes or measures the success of that business problem is what you use as your fitness function within the algorithmic optimisation, and then at the end of the process, you build 100 or 200 models, the algorithmic optimisation has simulated all those models, calculating them against your KPI, what comes out the other end is a very robust model that has been pressure-tested against lots of different approaches and variables and combinations of independent variables, and you can count on it being very, very effective out in the field.

0:10:44.8 S2: What are some questions you can ask to make sure that you are assessing something on the right kind of KPIs?

0:10:51.2 GC: Picking the right KPI can be a little tricky. Sometimes it can be tricky because what you wanna measure isn't so necessarily straightforward. The first thing you have to determine or figure out when you're trying to solve a business question is whether it's a well-defined question. Well-defined in mathematics means that it has a completely unambiguous answer like a yes or a no, no middle ground, or it's specifically one of maybe five mutually exclusive categories for instance, then when you have that, now you're trying to figure out one measure that represents the outcome you're looking for. So in situations like the website, for example, if you're measuring click-through, well, that's a simple number. You can just measure that off the website and see what's going to click through.

0:11:49.0 GC: Now let's look at something a little more complicated like, say, ROI. ROI is something that often is exactly what you wanna optimise, but ROI now is a combination of two other calculations, which is cost and price, and you take those and subtract those and, in a simplistic way, and you've got ROI. There are other calculations where you might have interactive variables, where, for example, say you're trying to sell tickets to the opera, you figure, "Well, it might be related to income, but it's also related to education, so now what I have to do is actually create a hybrid variable that mixes those two dimensions or captures them in such a way that I can optimise for that combined variable." So there is some art that goes into determining these KPIs, but if you build them correctly, they can be very effective when used in conjunction with an optimisation tool.

0:13:00.1 GC: One of the things we always talk about is the trinity of data, insight and action. Data is the source material, it's the raw material that you have to work with. Insight is drawing out the ore from that material, or drawing out the gold from the raw material, and then action is where you actually monetise that insight and create value with it. So those three things always work together. You've gotta have good data, then you have to have good effective and timely insights, and then you've gotta put it to work to make money. And one of the things that you find in our industry a lot is that there are a lot of companies and a lot of vendors, and the first thing they'll start reciting is the resume of their various statisticians and data scientists and how brilliant they are, but at the end of the day, the best model is the one that makes you the most money. It's just that simple. So the best thing to do is to test lots of models, test a lot, figure out what works and plug it in everywhere.

0:14:13.2 GC: So a lot of it happens in the definition of the KPI in that simulation engine, right? Because what happens is when a model gets built, that model is then fed some inputs to then test the outcome against that KPI. So that process has to happen very rapidly. And it's not exactly a parallel process, it's more of a champion challenger process, because what happens is the first few models will... Like when... So say your KPI is way up here, and that first model is pretty weak, but it's the first model, so that sets the benchmark pretty low. Well, every time you get a better model, that ratchets up a little bit more towards the ultimate goal of the KPI. Each time you get a little more, a little more, a little more accurate, that ratchets up continually until you get either as close as possible to that KPI or you run out of time, or you run out of machine horsepower. But the idea is to take that champion challenger approach where with each successive iteration you either get improvement or it's discarded, improvement or it's discarded, and each generation goes through that in the simulator, and then you end up with something that has actually succeeded in preference to all of these other models that have failed.

0:15:54.4 S2: At this point, I had a very basic curiosity about the ensemble approach to generating the model outcomes. Where do these algorithms all come from? Is the RedPoint team writing all of them? Are they copying and pasting them from GitHub? And is there another reason perhaps besides survival of the fittest why it's so important to run them all in tandem?

0:16:15.7 GC: We use a combination of both proprietary algorithms and commonly used algorithms that are out, public domain types of algorithms, things like k-means and different types of decision trees, partial least squares and other regression types. All these different algorithms are out there and common techniques that people use. And we have included them because those are techniques that people feel comfortable with and can understand or have a certain level of familiarity with what kind of outcomes they get. So we have included those well-loved techniques in the system, in addition to our own proprietary algorithms that are really neural net based evolutionary programming. And what we find is that that evolutionary programming approach both is very fast at converging on an ultimate solution, and it frequently out-performs all of these other kind of common algorithms. However, if we handed that to somebody without the context of those familiar algorithms, they wouldn't necessarily know if they could trust it. So those more familiar algorithms create that benchmark that lets them understand and evaluate the outcomes we get and then use whichever one, frankly, they would like to.

0:18:00.6 GC: They can always choose to use any one of the algorithms that are built. Obviously, you always wanna use one of the better ones, but there might be a reason why the first one or the second one might be impractical. Actually, let me, if I may, give you an example. Our data scientist who is very bright as well as everybody else is, he actually worked with one of the pharmaceutical companies, and he was given a problem to determine a molecule that fit into a particular receptor in the body somewhere, and he purely did it without any particular knowledge of chemistry or biology. It was purely on the shape of the molecule and the form of the receptor and some of the basic constraints of molecules. So he coded all that into the algorithm and he came up with five molecules that the pharmaceutical company tested. Well, the very first molecule that fit perfectly would kill you, so they didn't wanna use that one, but the second one actually became a hugely popular drug that sold around the world. So there are certain constraints and real world considerations that have to be brought into looking at these model outcomes.

0:19:23.4 GC: What we do as part of the overall evolutionary programming approach is that we do subtle permutations to all of these formulas to get better outcomes. Now, that is an approach generally falls under the category of genetic algorithms where much like genes you get random changes in those genes and some succeed and some don't. In this particular approach, what we use is evolutionary programming, which actually has a feed-forward and feedback type of interaction that allows you to get to an answer quicker. So we use the open source algorithms, we use our algorithms, we do all these little permutations and variations. We get lots and lots of models coming out of the other end, and then the optimiser runs through them all and then figures out which one is the best.

0:20:27.2 S2: This notion of asking the right question kept coming up, whether the question comes from physics or from your end users, like in RedPoint's case, but I kept wondering how much any AI expert needs to be able to discern between a merely fascinating mathematical question and one that can end up working in the product sense. In other words, what is the responsibility of any AI practitioner to have a deeper understanding of business needs, in addition, of course, to their robust technical know-how?

0:20:57.9 GC: When we're working or looking for talent in this arena, what we're looking for is not necessarily the greatest statistician ever to walk the face of the earth. Frankly, someone like that would probably be in a university developing new techniques and doing other kinds of research. What we're really looking for is someone who can walk in with a statistical toolset as a cost or a price of entry, but then understand how to be the midwife that can understand enough of the business problem to convert it into that well-defined question that can be tied to a KPI that then you can measure clearly whether that model succeeded or not. And that combined skill is often hard to find. And my bias is I find that physicists are some of the best thinkers in that respect because they know how to take those word problems, bring it all together, understand enough of the context and then actually apply a technique to solving it.

0:22:14.7 S2: Before I let George go, I wanted to take him out of the simulators, the optimisers and the evolutionary programming, and just hear what excites him most about the field of AI.

0:22:25.5 GC: What I think is really exciting about AI in the world in general and as I see it play a role in our software, is that AI can do a lot of the grunt work and the non-value add work that people have to do every day. Much as with statisticians and how much data processing they had to do just to be able to get to that one piece of the algorithm where their skill really comes to bear, similarly, marketers often spend their days working on a lot of stuff that's logistical and really not marketing strategy, same with accountants and lots of people. They do a lot of work that isn't the highest value work they can do, and I think writ large, AI can help a lot of that grunt work be done and release people from having to do that and elevate their focus to more strategic, higher value activities. And certainly I see that specifically in marketing, and we're focusing on the next generation of our software platform.

0:23:43.5 GC: We're already focused on how we're going to turn our data management into a pure data quality service, and what we're thinking about for the next act after that is how do we use AI to release marketers from all of that logistic work so that the machine is taking care of all of that and their entire focus is on feeding the beast and thinking about strategy. How do they want to set up the relationship between their brand and their customers, and how do they want to define success in those various segments of customers or approaches to the customers? So that I think is really exciting, and obviously it leads to lots of interesting gadgets and self-driving cars and all those things. It will be a lot of fun as well.

0:24:46.5 S2: Lastly, is physics applied math, or is math applied physics? Who's on top here?

0:24:52.3 GC: Well, that was always a running joke with my uncle because he said, he would always claim physicists solve the hardest problems, and I would say, "Yeah, but when you get stuck, you always need an applied mathematician to bail you out." So it kind of depends. It's all a very rarefied environment, and the best is really when those two work together where a problem is brought in that's really pushing the boundaries of physics, and maybe new math has to be developed just to be able to handle that problem, and that's really where some really exciting things happen, and it's a lot of fun for those folks that wanna keep a foot in the academic world, as you mentioned earlier.

0:25:39.8 S2: So it sounds like your answer is math. Math is the truth?

0:25:42.0 GC: Absolutely, absolutely.


0:25:52.2 S2: Next time on How AI Happens.

0:25:55.1 Speaker 3: So gone are the days where our workforce would come in and be like, "Oh, is there a dog in this picture?" or something like that. They're gonna do way more sophisticated work, doing everything from training machine learning models and driving very sophisticated training data sets to evaluating, quality sharing, training data sets that have been produced by automation, all the way to identifying what's missing? Do we have a representative and complete data set?


0:26:23.1 S2: How AI Happens is brought to you by Sama. Sama provides accurate data for ambitious AI, specialising in image, video and sensor data annotation and validation for machine learning algorithms in industries such as transportation, retail, E-commerce, media, MedTech, robotics and agriculture. For more information, head to