Arria CTO Neil Burnett explains how advances in NLG will develop trust and more widespread adoption in AI Technologies.
Arria is a Natural Language Generation company that replicates the human process of expertly analyzing and communicating data insights. We caught up with their CTO, Neil Burnett, to learn more about how Arria's technology goes beyond the standard rules-based NLP approach, as well as how the technology develops and grows once it's placed in the hands of the consumer. Neil explains the huge opportunity within NLG, and how solving for seamless language based communication between humans and machines will result in increased trust and widespread adoption in AI/ML technologies.
EPISODE 25
[INTRODUCTION]
"NB: There's the element of trust. If I’ve got a machine telling me to do something, telling me to act on something, telling me something is happening, how do I trust it? How do I believe it? The advantage of language, as the mechanism here, is for communicating, is that you can explain why."
[00:00:16] RS: Welcome to How AI Happens, a podcast where experts explain their work at the cutting edge of artificial intelligence. You'll hear from AI researchers, data scientists, and machine learning engineers as they get technical about the most exciting developments in their field and the challenges they're facing along the way. I'm your host, Rob Stevenson, and we're about to learn how AI happens.
Today on How AI Happens, we're jumping back into natural language generation. There's a growing belief that nailing NLG and NLP will unlock mass adoption of AI technologies. And it's not hard to see why. When people can interact with machines, as well as, or perhaps even better than, they interact with humans, it will result in trust, better technology and crucially democratize access to the kind of insight that historically needed, at the very least, computer literacy, and often, much more advanced coding chops.
Here to explain some of the use cases, opportunity and hurdles of bringing advance NLG to market is the Chief Technology Officer over at Arria, Neil Burnett.
[INTERVIEW]
[00:01:32] NB: I did my undergraduate in computing, first of all, with nothing related to AI, nothing related to language technologies. Just a very general computing degree. And when I was looking for my next step, I kind of felt like I wasn't fully ready for industry. I wasn't really clear on what I wanted to do. So I picked up the opportunity to do a master's post-grad at the University of Aberdeen, which is actually where I met some of the people that are founders in the company now. I started working in this field of NLG, natural language generation.
So I graduated from there. Forgot all about it for a few years while I went and worked for a company. And what I started doing was actually building a social media site from ground-up, like a Java-based application that was a social media site, which is kind of before we had any of these sort of frameworks for doing that. Like, you get something off the shelf these days or extend something.
Kind of evolved there through that company where I was a developer, to begin with, as I say. Then I became sort of like a manager, a manager of a very, very small team. And we were doing some interesting things. But what we found was that competitors who had more money than us, that were better backed than us, were able to replicate all of our key functionality and just kind of wiped us out.
So I was looking for an opportunity, something to move on to. Something I could be at for a long time. Something I could grow in. And most of all, really interesting, tech. And I saw the name of Professor Ehud Reiter, who actually oversaw my master's project. And he was advertising to come work at his company called Data2Text. So I called him up and said I’d be good to talk about this. Did an interview. Took about three months, I think, to get back to me. So they must have gone through plenty of other candidates. Rejecting them first before they came back to me.
But they brought me, and I was a general-purpose, fairly junior developer, and was doing that for a few years. Worked on a couple of the initial applications we had as the company was called Data2Text. It evolved. It was acquired by an organization called Arria. And for me, it was just a case of moving up in the development ranks to sort of leading teams and covering as many projects as we possibly could.
I was working closely with the CTO at that time of the company called Dr. Robert Dale. He took me to Australia for six months to open an office there. And over the time that we were kind of working together, there was enough drinks and evening socials and things for me to rant to him about everything that I thought we should be doing differently. And came back to the UK and was appointed as the Head of Development. And did that for a couple years. And through various changes and through progressions, moved up to CTO, and been doing that for several years now actually. And kind of over that time, we've grown from 50 people to around about 200. And day to day, I kind of focus a bit more on the core technology and how it kind of works and the nuts and bolts. And we've got other very capable people who are overseeing other detailed aspects for other sort of products as well.
[00:04:33] RS: I have to say, I’ve definitely, towards the end of a happy hour, told my boss what I really thought about a company. And it never resulted in me getting promoted. So you must have approached that with a tad more diplomacy intact than me. But this is also an interesting part of your journey, how you've kind of – As you said, junior developer. Now you are kind of in the C-level. I really want to get into the technology. But I am curious just about your career growth. Because for folks out there who are working in AI, they are going to be forced with this decision to continue as an individual contributor working and developing these tools, or to grow into leading teams, managing teams, etc. So how did you kind of, one, chart that growth and like succeeding being able to have this Chief Technology Officer position? And two, how did you think about whether that was the role you really wanted?
[00:05:26] NB: It's sort of a natural progression to pick up more and more responsibility. That's a path for a lot of people who just want to move up through the ranks and become more and more senior, because with that goes recognition, control and salary. But it's not for everybody. There's often just getting more deeply involved in the technology. And we need people like that in the organization. It's essential to have people who just really want to know absolutely everything about technology and get incredibly deep into it.
For me, it's more than just the technology that that's kind of a key aspect to it. The three items that you normally focus on would be around people, process and technology. And on the people's side, I think we get a lot of thoughts there about management or working with people. And the idea that a manager can't really define the boundaries of the capabilities of the team, right? A manager or somebody who's a leader the team needs to kind of appreciate that other perspectives are required and they won't always fit with your own. And in the same way that a CTO can't set the bounds of the technology, right?
So for me, my journey has been appreciating through the people I’ve worked with who've all been capable, intelligent, articulate people and just being noticed for that, right? And being noticed for not being the one who's fighting to be the most seen. That appreciates the strength of the organization is the strength of the team. And enjoying working with people, right? Enjoying working with people. Enjoying evolving processes as well as just enjoying working with technology. Staying relevant in the field. Understanding some degree of depth. So sort of covering all those things, it's been better for me to be moving up through management levels rather than just focusing on the technology.
[00:07:11] RS: Well, having said that, let's focus on the technology. Shall we? I guess, first, can you share a little bit about the company and what the technology is and the use cases?
[00:07:19] NB: Yeah, sure. Yeah. So in the simplest form, Arria's technology is about turning data into text. But the text is really to allow communication that could be well-understood by readers, by people that are being communicated with, right? Beyond just generating text for tech's sake, right? We allow people to gain insights on their data, to communicate with the data, to understand the insights through clear language and explanation. It brings benefits to organization who use that technology in the form of cost saving, time saving, as well as through consistent understanding of critical insights, critical concepts, and that consistency that machine-based approach can bring.
With Arria, we've worked on a fairly broad set of use cases and a broad set of verticals. In the early days, we worked on weather, weather reporting, where we work with an organization who had so much data that they couldn't manually generate enough weather forecasts. So a machine can do that in a very consistent way that showers always means the same thing, that a chilly day means the same thing. That's weather in Scotland, where I’m based, right?
We worked in an oil and gas, where we were dealing with very, very large data sets and too much data for a human to process properly. So they were making decisions without knowing the full information. And we could support decision-making as a key aspect. You can interact with our technology in terms of generating reports and automating reports. But you can also interact with our technology in the form of Q&A. So you can ask your question in a very natural human form and get a natural response back. And revolving our technology into the style of being able to create reports from Q&A, as well as building up from sections.
[00:09:05] RS: There's this belief that more data is always better. And perhaps with infinite processing resources, infinite computational resources, annotating resources, that's the case. But who has those resources? And so it strikes me that an important process here is you mentioned you quickly had more data than any human could hope to process. This need to prioritize and then develop a narrative from that prioritized data, is that the case? And if so, how do you go from honing in on only the relevant data and then turning that into a meaningful story to the user?
[00:09:43] NB: You might think that defining a set of rules across massive sets of data might be very difficult, very laborious, and often end up missing edge cases. But kind of that rules-based approach is kind of required for quality. And then being able to sort of introduce pieces of technology that bring more machine-based approaches to kind of augment, right?
I mean, there's a lot of buzz around end-to-end natural language technologies. And rightly so, because it's fascinating. It's a developing field. And when you're left with massive sets of data, just seeing what the machine can do and have a bit of a look at it and see what it can generate, it's a fascinating thing to do. But if you want to derive meaningful insight, then you cannot just let the machine go and find things out for you and just come back with what it is, because it's often inaccurate. And it's often not quite missing the right important cases.
So our approach with Arria is a combination of rules-based approach with augmentations, right? But you always need that consistency of having a rules-based approach in there to ensure quality and prevent hallucination, when you use that word hallucination. We build these rules. We work with subject matter experts to refine and define those rules to make sure that we're finding the right insights, that we're prioritizing the right insights. But then we're augmenting that with edits and suggestions that can be machine-generated that might not find their way into the final report. They're just sort of helping the author build a case.
[00:11:18] RS: Could you give an example of one of those edits?
[00:11:21] NB: So we can fix the language. That's sort of one of those sorts of edits, right? We're automatically fixing the language afterwards. If the author slightly gets it a little bit wrong, we can make those changes for them sort of on the fly. But we can also flag up facts that haven't been seen before. We can suggest alternate ways of expressing a point to make it more salient or more in keeping with the domain you're working with. Like a very generic description of a trend is good. But when you're talking within the finance domain, do the words and the phrases that you use are important in conveying the meaning? So being able to look across pre-written reports, pre-written narratives created by your organization, we can look and say, "Well, these words are maybe a little bit better in this sort of context."
[00:12:08] RS: Can you explain hallucination for me? I’ve never heard that term before.
[00:12:11] NB: Yeah, hallucination. It's not just what happens when you sit down with your CTO and have a few drinks and imagine things are coming at you across the bar.
[00:12:19] RS: When you take your CTO to Lollapalooza, yeah.
[00:12:23] NB: No. But quite literally, seeing things that aren't there. But from the machine perspective, it's a term that's coming up in the field quite a lot. It's started to be recognized. And it's a great term, right? Like the idea of hallucinating and seeing something that isn't there.
I’ve been trying some of the more end-to-end approaches for language generation and language processing, even things like summarizing. So I took a text that Arria generated, a report that Arria generated. Probably only a couple of paragraphs, sales report talking about the key countries that were driving sales and the key regions that were driving sales. Pretty simple report in the grand scheme of things. The summarization, the idea was could we compress those two paragraphs to the most salient points a couple of sentences?
And what the machine did was it started talking about sales in New Zealand. And New Zealand wasn't in the original text. And because the end-to-end doesn't even look at the data whatsoever, it was basically inventing New Zealand. It knew New Zealand was a country. And you were talking about countries. But it was stating as a fact that sales were increasing in New Zealand, when sales are not even performed in New Zealand. We're not selling anything in New Zealand, right? So that's just a complete hallucination. It's coming up with a fact and speaking as if it's fact, when it's not really. It's just something that's there, right?
[00:13:44] RS: How does that happen? How do those cheeky kiwis make their way into that report?
[00:13:47] NB: It's interesting, isn't it? It must just be the frequency of which certain countries are put together. Like maybe our report was talking about Australia, or had Australia in there as a key driver of sales. And then they know that New Zealand goes along with Australia frequently and put those together. It's a really interesting thing, because you only ever see the result. You don't sort of see what's going on behind the scenes. You just kind of have to guess. But educated guess.
[00:14:16] RS: Are there any like attention mechanisms in place or ways to sort of figure out how hallucination might occur? I’m curious like how you would go about mitigating them once you had experienced it.
[00:14:27] NB: It's difficult. If it's sort of this black box of text-in, text-out and a whole lot of machine learning in the middle, it can be very difficult to control. I guess you would need to put some checks and balances on the output, which have been a bit more NLP to kind of parse it. I mean, it's really about preventing these hallucinations through control. Restricting the insights to be true with the data, which is what Arria is really doing, right? That's why the rules-based approach is required.
And we've spoken to people as we've been recruiting. We've been interviewing people who've worked for very large organizations working in the field. And they're finding the same thing, right? They're having to implement a rules-based approach even on sort of some of these commercial devices that everybody has in their living room, right? It's just the way things are with the state of the tech at the moment.
[00:15:19] RS: The state of the tech at the moment is another thing I wanted to kind of speak with you about just because there are all these applications, like you said, you're luck in fintech. There are ways that we engage with it as consumers every day without even realizing it. What are some of the barriers to adoption would you say? What is sort of holding language technologies back from being more widespread? Is it a state of just consumer understanding? Is it a state where the technology is and where the products are?
[00:15:49] NB: Yeah, I think this is a really good question actually, because we've been on a bit of a journey as a company. And the joke is they could never spell NLG in the early days, let alone, nobody did. But the world's become a lot more educated at this point. And there's a fairly good understanding of what it is. But it's still kind of on the upward curve of adoption, because people have got various different reasons for not trusting it, not wanting to work with it.
I think in our experience, there's the element of trust. If I’ve got a machine telling me to do something, telling me to act on something, tell me something is happening, how do I know it's a fact? How do I trust it? How do I believe it? And the advantage of language, as the mechanism here, is for communicating, is that you can explain why. You can explain your justification. This is happening because of this. Or we've seen this in the data, which backs up our point. And we've looked at ways of augmenting the text with click and go to a graph and things like that to sort of support our point. But you can be persuasive and you can tell people. But people really need to sort of adopt that first use case to really see that, see the trust. They really want to be able to see how the internals of the system works, which you could do with our technology. And then have that trust that the rules that are in place are going to give you the right results.
One of the things that we see people commenting a lot is they want to use their own voice. They want to use their own company's voice and their own company's way of saying things. They want to be able to edit what's going on in the technology. They want to be able to edit the insights that are being generated, the way they're expressed. And that one is totally alleviated by better products. I mean, allowing users to be able to control users of different skill levels to be able to control. Because I’m a developer, I’ve been doing it for 10 years, I can build an NLG system. But this company that's just picked up our technology loves what we can do out of the box, but wants to make modifications to the way things are phrased or the order in which things are reported upon, the way things are drilled down. We need to provide them with tooling and better products to have it work and speak the way they want it to work.
And the naturalness and the persuasiveness of language is important for that. Language is kind of what people are most comfortable with, and it's how they communicate with each other. I mean, sure, there's body language and things too. But if you can get language right, there's almost this – It almost feels like you're connecting at the point of being told something you're building trust because you're communicating with language. If I view a graph, then it leaves to my own interpretation to some extent so that it's fine, right? I can see a graph. I can take my picture away from it. I can tell my story. But going that extra mile with language technologies of being able to tell people in detail. They want that extra level of robustness, of trust in it.
One other thing that we found, it was an interesting one. We came up with a – We had an Alexa skill, and it allowed you to – I mentioned we could do Q&A with the data with our technology. So with the voice interface. And we had a customer. And we said, "Here's an Alexa. Put it on your desk table." And then when you go into meetings, you can start talking to it and it can back you up in a meeting, in a sales meeting, or on a board meeting. And the feedback was that we don't want to put an Alexa on the desk in the meeting room, because anybody can walk into our unsecured meeting rooms and find out facts they didn't want to know. So there's always something that can turn up. But at the end of the day, I think we, as commercial organizations, can be listening to what the world is telling us and how they'll feel more comfortable adopting. How they'll be able to handle more use cases? And listen to that, and create better products that are going to address those concerns. That's when this technology is going to become much more ubiquitous.
[00:20:00] RS: Especially, with language the way people speak, we understand innately these rules one. One example is why can you have a great green hill and not a green great hill, right? It's like it's just an order of words. It doesn't mean anything, but it does. Like, if I said, "Oh, that's a green great hill behind you," you'd be like, "Rob talks weird."
And so there's this need with language to – And it ties in with what you're saying about trust, to meet people where they are with how they communicate and how they expect to be communicated with. Think of the way one speaks to a Siri or an Alexa versus how you search in Google, right? Like when I’m searching in Google, I don't type, "Can you please show me the directions to this gas station?" right? I just go, "Gas station near me," right? But like if I was going to speak to a virtual assistant – Or sorry. An AI assistant. I would. I would pose it as an actual human question.
And so this level of comfortability, it feels to me, as you were saying, one of the last pieces where people can speak and engage with this technology the way they engage with everything else in their life. And then it doesn't become interruptive. Then it just becomes natural language, right?
[00:21:11] NB: Absolutely. I make sure my kids say please and thank you to the Google assistant that's in the living room as they're asking for music and things to dance around, to use language as human, right? We're used to interacting with each other in a certain way. When we're interacting with technology in a certain way, we also then expect it to interact back to us in a certain way. And it's not bullet points. And it's not factlets that float around in a dashboard detached from the rest of the information. It's well-backed up articulate arguments or points well-made in good flowing language that just allows you to feel in your flow, right? Feel in your flow that you're communicating and communicating with in a way that just works for the way the human mind has evolved to work, to be interacting with language.
[00:21:59] RS: Yes. And it's this interesting flipping sort of, where, originally, computers just having any sort of output was groundbreaking. And then you had to know how to – Humans had to know how to speak to computers, right? You had to know a coding language, or you had to know Boolean search, or you had to know some advanced way to interact with this machine. But the idea is, now, we want machines to learn how to interact with us, right? So how much of that has been part of generating the technology of – What is the gap between how machines process information versus how humans process information and finding the meeting point?
[00:22:35] NB: It comes down to to sort of a methodology or a process. The nuts and bolts level in the system, it comes down to that. Because you listen and you understand, and then you analyze. And humans do that very, very quickly, not on large sets of data, but on past experience. But then you're sort of determining what your opinion is or what you're going to say. And then you organize it into the best way of communicating. And then you speak it out articulately.
And the technology is to replicate each of those levels in a way that is not as vague and susceptible to headaches and tiredness like humans are. It's interesting, because you have to think about the way humans process information and articulate their point and replicate that in technology, because all those steps that you're performing without even necessarily knowing it are the steps that have to be in the technology, too, to make those points articulately.
[00:23:36] RS: Yes, absolutely. Well, Neil, we are creeping up on optimal podcast length here. Before I let you go, I’m gonna put it on you to kind of tie a nice bow on this episode. I’m curious when you think about just the field writ large, whether it's natural language processing, or any sort of application of this technology, what truly excites you? What is kind of an inspiring outlet for this tech that just makes you excited to be a part of the industry?
[00:24:04] NB: So I’ll give you an answer, one, about what interests me in sort of the language field. And then I’ll give you another one about just my own pet interests. So within the language technology field, I mean, I was really interested by some of the work we started doing in sports just because it feels like something that has mass adoption and it has something that is very achievable with technology to be reporting on summaries, and key facts, and highly personalized interaction for people on a one-to-one between you and what's going on in your team.
But overall, for technology, I really feel like somebody, and I’m sure it's going to be us, is going to crack it so that most cases that you would want to interact with a computer in a database way, in a way based on data, is that somebody's going to crack it and handle most of those cases. It'll be in a brain. It'll be in an insight engine that people can use and interact with. And so our ambitions and the ambitions of the field are to handle more and more of human interactions with computer and to make language get to the point that people are beyond dashboards and people are beyond looking at pictures and they're just working with language. So it's kind of what we exist to do, but it's just a really interesting thing in the field that we could change the way people interact with computers, people interact with technology, people interact with data.
And then that all said, you mentioned self-driving cars. I was playing with the AWS DeepRacer, and I don't want Formula 1 or NASCAR to become fully autonomous, because you need those humans in there. But just the idea that you can figure out, or AI can figure out the fastest way around the track. Just like it was an interesting thing that they did to put that out there as a way of helping people understand and see how AI can work.
[00:26:02] RS: I’m just now realizing, we're going to have a moment with AI and Formula 1 or NASCAR that we had with, like, Garry Kasparov in Deep Blue, right? Like there'll be a point where machines are so good at driving that they won't be allowed to compete because it's is not interesting. Chess computers are better than the best chess player, like every single time. So then it's just, okay, we admit that, that like computers are better, but the competitions we still watch are people. It'll be the same, right? Like a Formula 1 self-driven car will smoke everybody. And then like, "Okay, congrats. You guys did it." But no, this is a human’s game. Thank you very much.
[00:26:39] NB: And it won't be interesting anymore, and people would stop watching. Exactly. They're already trying to take all the little things that help the human out of the mix, right?
[OUTRO]
[00:26:54] RS: How AI Happens is brought to you by Sama. Sama provides accurate data for ambitious AI. Specializing in image, video and sensor data annotation and validation for machine learning algorithms in industries such as transportation, retail, e-commerce, media, medtech, robotics and agriculture. For more information, head to sama.com.
[END]