Today, we are joined by Ram Venkatesh, the founder of Sema4.ai – an enterprise AI agent platform that enables businesses to build, operate, and scale AI agents. Ram shares the risks of training LLMs from agent data, and the contextual work training protocol for agents. We also unpack the requirements of a large language model when you’re not responsible for training it, the various modalities and how they can be improved, the threat that agents pose to SaaS, and Ram’s vision of the future of AI.
Key Points From This Episode:
Quotes:
“I’ve spent the last 30 years in data. So, if there’s a database out there, whether it’s relational or object or XML or JSON, I’ve done something unspeakable to it at some point.” — @ramvzz [0:01:46]
“As people are getting more experienced with how they could apply GenAI to solve their problems, then they’re realizing that they do need to organize their data and that data is really important.” — @ramvzz [0:18:58]
“Following the technology and where it can go, there’s a lot of fun to be had with that.” — @ramvzz [0:23:29]
“Now that we can see how software development itself is evolving, I think that 12-year-old me would’ve built so many more cooler things than I did with all the tech that’s out here now.” — @ramvzz [0:29:14]
Links Mentioned in Today’s Episode:
Ram Venkatesh: And you're able to talk to the code, you're able to kind of shape what you want it to do at such a higher level of abstraction. And we've been waiting so long for this, we don't know what it was. But now that we can see how software development itself is evolving, I think that 12 year old me would have built so many more cooler things than we did. All the tech that's out here now.
Rob Stevenson: Welcome to How AI Happens, a podcast where experts explain their work at the cutting edge of artificial intelligence. You'll hear from AI researchers, data scientists and machine learning engineers as they get technical about the most exciting developments in their field and the challenges they're facing along the way. I'm your host Rob Stevenson and we'about to learn how AI Happens. Welcome back one and all to How AI Happens. Rob Stevenson here at the helm and I have another great guest lined up for you today. Hes not big on titles so Im going to humiliate him at the top of the show here and just call out that he is the co founder and chief technology officer over at Sema4.AI Ram Venkatesh. Welcome to the podcast. How are you today?
Ram Venkatesh: Good, thank you. How are you?
Rob Stevenson: I am really great. We are rapidly approaching the holiday season here and happy to push to next year if you prefer, you know, time of the year and I've gotten a bunch of those already but you stuck with me here Rom and we're getting the episode in before 2025 so I appreciate your diligence.
Ram Venkatesh: Fantastic. Looking forward to this. I really can't tell what happened to the year between July 4th and now so it's been a blur.
Rob Stevenson: Yeah, I was thinking the same thing. It helps for me not to look at calendars these days I finding but you are doing some really interesting work with generative and sort of identifying the context for work and that's part of why I was excited to speak with you. We'll get there but lay we'll, we'll earn the insight a little bit. We'll build up to the good stuff. Before we get into that Rom, would you mind just sharing a little bit about your background and kind of how you wound up in this current position why you u set out to to found somemaphore.
Ram Venkatesh: Sure. Happy to. You know I think probably spent the last 30 years in data so if there's a database out there, whether it's relational or object or XML or JSON, I've done something unspeakable to it at some Point in the past, most of the last decade has been with big data. Before that, just been a nerd, very focused on all things programming. Started programming when I was 12 years old and never stopped.
Rob Stevenson: How about that? I love an origin story that involves you actually having been technical working on it your whole life. Because there's a couple different classes, classes of entrepreneur, and one of them is usually a savvy business person who can understand a market very easily and identify a solution. And then there's also the entrepreneur who struggles intimately with a problem and then decides that the best way to attack it is to maybe just be the solution itself and found a company. It strikes me that you're probably in that ladder camp.
Ram Venkatesh: Yeah, I was going to say the first one sounds like such a nice, pleasurable way to get there, but for me it's always been, I think the journey and the struggle is what is interesting about this.
Rob Stevenson: So what was the problem that you were experiencing that led you to found the company?
Ram Venkatesh: Look, the most recent one, I think this was almost like a fairly straight line from what we were doing in the past. So my most recent before this was, I was SCTO at cldera. We're probably managing some of the largest data estates on the planet. And we realized that, gosh, people just have too much of too many different kinds of data all through the enterprise. And While the last 30 years have been really good for folks like me to sell them data, infrastructure of various kinds to collect and shape and manage and refine data they really want to get to, customers are looking for, how do they do something with all of this data? Right. How do they actually get to outcomes and seeing some business value that's tangible from the data. And here's the catch. Without having a lot of people being added just because your data is growing. So the opportunity we saw was data continues to grow, but the things that people can do with the data to get real value out of it is still gated by the number of people that you can have applied to that problem. And with Gen AI, we saw an opportunity to actually unlock that. And that was pretty exciting. And so we said, okay, this company has to exist. And, we got started.
Rob Stevenson: Yeah, for a while there, it was just when big data was the trendy tech approach, more data was better. It was like this arms race for who had the most data points or what have you. And it feels like we're a little bit past that now. Like, data is more of a commodity. Data is out there to be had and it's less like how do we get our hands on enough data and more about can you actually do something with it? Would you agree that we're sort of according back to a more efficient version of this?
Ram Venkatesh: Yeah, I think it's nuanced. On the one hand I agree that people are like enough already with collecting data, let's do something with it. And the other piece is the people, the analysts in these companies, they've been quietly doing that already, except they do that in Excel or Google Sheets or various ways. And I think now these two worldDs are meeting where the way people do the work and the way the data infrastructure is supposed to help with the work, I think that's finally coming together. So instead of two silos or two wors, the ideal here is to have like one way of using the data to make decisions inside a company.
Rob Stevenson: Sure, yeah. So it sounds like when you were explaining your experience at Clouderra that when you were sort of considering how large scale data is handled at large companies, it's equal parts like data hygiene and that the data exists in lots of different places and there's no kind of throughputs it. Was that your experience?
Ram Venkatesh: Yeah. it's almost like I've heard CFOs in particular describe it. Like some of it is data literacy. Like they just have to figure out what is the right data that I really need to understand to understand how my company is functioning. And there's so much of it and we've all experienced this, especially if it's like a consequential decision. If somebody shows you a graph that shows the data, the first question you ask is where did the data come from? Right. And then the second question might be, was the analysis done? Right. So this is all I think questioning, being able to be confident of the quality and the hygiene and the lineage of the data. It's a non trivial problem for most companies.
Rob Stevenson: There was a running joke at a startup I worked at. Whenever someone would have put a graph up on a screen in a meeting, we would say please ignore the incorrect data. When you get into like looker or something and you can run a query and you can get a result. But again, how can you really trust the output unless you know underneath it is actually clean and ordered. So I'm sure everyone within the reach of our voices experience that to some degree. So then what is the solution that a semaphore is sort of suggesting for folks.
Ram Venkatesh: Yeah. Even before you get to is the data. Right. The question you will get asked Is is the query right? And if somebody goes like oh here's the dashboard and you right click and show the SQL, nobody in the room is going to understand what that report is actually showing you. Right. So for us the central mission at Semaphore is how do you make agents like enterprise agents that can interrogate your data sources and come up with good insights into your data that you can then do something with. But the cool part is all of that introgation that questioning is in English. So if you want to know how is this agent coming up with this insight, you ask it and it replies back and it's not tecxt peak. You can actually look at the question and go yep, that sounds reasonable. And that's where I would look for the data. So I think there is a level of transparency to this entire process that this agentic way enables, which is very exciting.
Rob Stevenson: I see. So then the agentic approach here is that you can ask a clarifying question and the attention mechanism in this case is natural language.
Ram Venkatesh: Yeah, if you started the sort of, you start the top right. Agents are thesis is agents have to actually do the work. They are not like chatbots or assistants or copilots which help you with the work. They just have to do the work themselves. Whether it's like processing an invoice or handling a ah, remittance notice or awesome cool things like that. Right. So if the agent has to do the work, you have to specify the work to the agent. I believe we are at the stage where agents are great at following plans. They're not so good at creating plans. And so to follow the plan we're saying that itself is natural language. So that then drives all of the reasoning and all of the data interrogation that's done by the agent. It's all in natural language.
Rob Stevenson: Right. Without it being able to do the work of its own accord, you don't have an agent. Right. Like that is in the name itself. What is agency? You know, exactly. And so to give a copilot or to give an LLM agency in order so that it might do the work, what is necessary for agency? Is it about initiative, is it about it not being told to do it or not ask clarifying questions or what defines agency in your view?
Ram Venkatesh: Ye, I think the central point is like this notion of, I like to think of this as constrained autonomy where if you tell the agent like hey, go process this invoice. And the agent says okay, I'm going to get going on it. And this invoice was from this supplier I'm going to verify, is this supplier somebody we know? Do we do business with? Did we actually. Did they deliver the service that they were supposed to? Did they charge us what they said they were charges. Did we like the work? So it's going to go do all of these sort of interrogations of the business and then decide, okay, that all looks good. So I'm going to pay the invoice right now. Everything that I'm telling you, you could write a program to do it in a very, very fixed way. What's interesting about agents is that you can now do this in a flexible way and still have the deterministic repeatable execution that you're looking.
Rob Stevenson: I am curious if you can speak a little bit more about that last bit. The difference between writing a program to execute if then on for example, processing invoice versus an agent being able to reason with the process.
Ram Venkatesh: Correct. So the top level framing for me is agents are programs, but not all programs are agents. Right? Programs. Typically if you write the program down by hand, it's great at doing the same task in the same way very well. If anything changes, you need a new program. Correct. And if the program does not follow the script, that's a bug. So that's kind of how your conventional program works. And then people kind of try to put like a, low code experience on it, which is a shorthand for. You're going to drag and drop components that produce a program for you which will have that same rigid and repeatable reasoning that comes with that kind of a program. On the other hand, if you take a natural language runbook, an LLM, and then have that be the reasoning engine, that reasoning engine is going to be capable of flexibility. So making minor changes, for example, you're going to change the English, the agent's going to pick it up and it's going to keep running. Now one might say, oh, that's like changing a program, except you didn't have to actually walk it. Let's say you are over in finance, you didn't have to walk this program metaphorically over to it and had them make the change. You can do that yourself as the person who's doing the work. And then the agent just gets better. So that's the kind of sort of responsiveness and flexibility that an agent can express. And an agent can also decide for some tasks it's going to improve over time. It's going to take five steps to do something and then realize that, oh, I don't need to do Step four, if I can actually have this validated input that I'm starting with. So the agent is going to change the order of execution for itself based on the context in which it's operating. So these are two fundamental ways. How I see agents are very different from deterministic conventional programs certainly.
Rob Stevenson: Now in the case where the agent understands the context for work and can perform the work, it feels like that is a more nuanced way of processing data. Is that more like the equivalent of paired programming? Can an agent be trained on an entire process rather than bits and pieces? What is the training for contextual work look like?
Ram Venkatesh: Yeah, part of it is also if you start with what is contextual work in an enterprise context? A lot of it is connecting the dots between different things. The things could be if you have lots of different data, it could be connecting the dots across different data sources. You could be asking questions across MongoDB and Snowflake and a few legal documents to understand what the return policy for a customer is and then handle how that particular return transaction is going to go. So that is just interrogating a bunch of different data systems. Or it could be interrogating a bunch of different line of business systems. You could say, oh, I'm going to talk to my CRM system, I'm going to get information about this customer, then I'm going to go look at my erp. somewhere in there there's probably like a servicenow, like trouble ticket management system that you may have to encounter. People are really good at this at navigating information flows across the different systems that make up a typical small or medium or even large enterprise. So that's the. When we talk about context, we're describing these data sources, these documents, this back end line of business systems. These are the kinds of areas where an agent needs to be aware of them. We will never train the LLM on any of this information because the minute you look at that information, sometimes without a date, for example, what is in your shopping cart right now, there's no reason, I think nothing good can come of training an LLM on that data. So this is where an agent has to work with context rather than train on context.
Rob Stevenson: Interesting. So the LLM and the agent would be separate. The agent is better viewed as an app that sources the LLM such that it can have generative natural language expression.
Ram Venkatesh: Yeah, I think this matteraphor kind of works is the LLM, is predominantly used for reasoning, it's for plan verification, is to understand the language. So the agent Orchestration loop is sort of how does the agent go about its work? The LLM is an integral part of how the work gets planned, how the work gets sequenced, but how the work gets done. We are not relying on the LLM to understand all of the data in the enterprise. The agent says the LLM kicks out the question. The agent can go out to the context layer to say, I would like to answer this question. So that responsibility is with the context layer because then you can enforce all the boring but important things like access control and privacy and things of that nature can be done in a layer that's very decoupled from the LLM.
Rob Stevenson: Is there a risk of training the LLM on this data or is it merely not necessary?
Ram Venkatesh: I think that training the LLM on this data is risky because fundamentally you're creating a statistically comm minledd representation of the data. Imagine if you took your salary database and you trained an LLM on it, or trained any kind of machine learning model on it for that matter. After that, if you to, you cannot do two things. One is if you were to go and ask what is this person's salary? You're going to get back a statistical answer for the salary for a person who looks like the person you're asking about, but not the same ident, not the exact one, right? That information is, is thrown away. This is why the LLM is not a database. And then the other piece is that now you have information that can be used by the LLM in ways that you did not intend for it to be used by, because once it's in the LLM weights, right, it's really hard for you to understand when did this information get used and when did it not? So now you can't, for example, impose access control that says people should only be allowed access to their own salary and to their salary of their reports. You cannot say that anymore if all of that data is comm minled inside the LLM. This is the reason why I believe it's truly thatm is awesome. It's a massive unlock for reasoning, but it's a very poor way to think about enterprise data as a repository for that data.
Rob Stevenson: It sounds like there's sort of only risks in terms of how the data would be used. And then also you're sort of short of it being the data being misused. It could be misrepresented, right? It could just result in output that is just makes no sense. So you're gumming up the work of your LLM unnecessarily here. And that's like the best case scenario is you only do that.
Ram Venkatesh: Yeah. I'm thinking, like, what, what good can come of that?
Rob Stevenson: Right.
Ram Venkatesh: It's almost. You might get imprecise answers, you might get accidentally embarrassing answers and wrong ones. So, yeah. And this, it's interesting. Some of the first experiments that customers have done with, even building things like knowledge bases, they realized that they built knowledge bases. You might do that by, like crawling an S3 bucket and indexing everything in it. It turns out some of those documents are like, from 5 years old. They're just wrong. Previously they were like way back in the bucket that nobody knew about them. Now you have a chatbot. The first thing you ask you, it's going to give you a response from something that was outdated five years ago that might have sensitive information in it and that might be wrong. So then people had realized that they need a way to actually make sure that their knowledge bases are curated and accurate.
Rob Stevenson: The compulsion to train or to dump one zone data into the LLM, is this mere convention? Is this just like the developmental process that people have become used to? It's like, oh, we trained an LLM on blank so that we could query it and that the output would be relevant to our business. That is just the compulsion. And then decoupling it is just like the next generation of trying to use these tools more efficiently.
Ram Venkatesh: Correct. I think it's almost like LLM vanity. Right. Last year, if you had talked to somebody at a large company and you ask them like, are you guys adopting Gen AI? And they say, yeah, so what are you doing with it? We're buying H100s. I'm like, okay, that's a start. But what are you doing with H100s? Right. The answer has got to be, we're going to be training. So what are you training for? We're going to be training LLMs on all of our internal data. And I'm thinking, oh, boy, I know how the rest of this story is going to end. But I think that as people are getting more experienced with how they could apply Gen AI to solve real problems, then they're realizing that they do need to organize their data, and their data is really important to. Don't get me wrong. I think that I feel data is one of the main things that can help LLMs be really accurate or a solution built around LLMs to be really accurate. And now this is the way for people to get there rather than train the LLM.
Rob Stevenson: So assuming then that you were not interested in training the LLM. what then are the requirements of an LLM? It strikes me that they would maybe go down.
Ram Venkatesh: I think that language is hard, you know, the nuance with which people apply language to do their job. We see that it's not, it's not just about linguistic nuance, but it's also domain specific applicability. I see that LLMs are starting to get better at recognizing plan or plan steps or plan priorities and constraints. So I think that just focusing on the middle L, the language part, but like really how we use language to do whatever it is that we need to get done on a daily basis, I think that L LLM is improving on them. That's going to just continue to make the overall solutions more robust. I think modality is going to continue to get more and more nuanced. So better vision, better audio, being able to be more responsive in these other modalities other than text. I think that, so the LLM as an interface, as a logical interface to what you're trying to get done, I think that is so much more powerful. And if that continues to improve the rate that it has even in the last 12 to 18 months, I think that'll be awesome.
Rob Stevenson: Yeah, certainly I'm gonna exhibit some real self control and not go down the modalities rabbit hole with you right now.
Ram Venkatesh: I did throw it out there for that reason. Exactly. Yeah.
Rob Stevenson: I mean should we. We can do anything we want, Rob.
Ram Venkatesh: So when we started looking at financial work, right. So things like unstructured documents like invoices would keep coming up and I was thinking, gosh, isn't this a solved problem? You know, there's so many companies that focus just on parsing PDF documents and you can use one of those and you'll be done. And customers tell us that the accuracy rates are just horrendous, less than 50%. So it's worse than a coin task to go with some of these ways of detecting documents. And just in the last year since GPT4.0 came out, it's been amazing what we've been able to do with, with the vision capabilities of these models along with human in the loop and prompting and all of that. But it's just gotten easier to understand documents. Right. So that's a good example I feel where modality improvement has for us it's been a measurably good thing.
Rob Stevenson: What is the additional modality there that is helping processing vision vision of just like, oh, training on the images of a PDF rather than the Text.
Ram Venkatesh: Correct. Of seeing the PDF for what it is. Like, here are some tables, here are some visual details. This is how these visual elements relate to each other and then starting to understand that. So a lot of the trick with processing something like an invoice might be somebody who's skilled in the art, like the person who's been dealing with that kind of document for a significant amount of time. They will tell you, this is how I understand this invoice. Now I can tell that to the LLM and turn around and say, now look at the document. And I mean it. I'm saying now look at the document. I'm not saying go parse all the text and please try to make sure that you don't throw anything on the floor. So that's been really nice to see.
Rob Stevenson: Okay, yeah, that makes sense. Look at this as opposed to say process 100% of this text. You know, this is even related to the context for work. Right. There are more modalities here, sensory inputs, than just processing text. And that's how human beings learn in all these new ways. And that feels like such an important unlock, is that the machines right now are being taught with mostly text. And text is just such a. It's not, I wouldn't say a small way that we learn, but it's one of many ways that we learn. And it doesn't feel like we've really unlocked the other ways. We're just getting there with video and audio. Forget about touch, forget about taste, forget about smell.
Ram Venkatesh: Exactly, exactly. Right now some of those I think, you know, are more applicable in a, in a consumer context or non work context. But I agree just as following the technology and kind of where it can go, it's a lot of fun to be had with that.
Rob Stevenson: Yeah, certainly we did actually have a guy on this show who was training machines to smell. And basically it's like a think of it like a smoke detector that can detect o. Yeah. Tens of thousands of chemicals as opposed to, you know, just a couple and then beep. I'll eat my words right there. It's not that there's no progress in the other modalities, but early days for sure. Now I wanted to ask, just this ability to make a request of an agent and then for it to go out and do this kind of work. Because traditionally that is done by an effective worker who's using a lot of different tools, a lot of different technologies, SaaS mostly. And a lot of SAS is like reporting there are applications, of course, but it's just like, okay, here's a report on blank, here's whatever, I'm going to regurgitate your data back to you in a way that is meaningful. And the idea to be able to give feedback to an agent and say, no, actually I want this, that feels fantastically disruptive to the SAS model because the alternative would be like getting on the quarterly call with your customer successor up from the software company and being like, man, I really wish that it could do this. And they would say, oh yeah, that's a good note. Maybe'll we'll put that in our little feedback document and if enough other people in addition to you, or if you're in a big enough account, maybe we'll build it for you. And with this M model, with this agent model, you are kind of in a position to build it yourself. Do you see that as like, am I ont to something there? Is this a part of the disruption?
Ram Venkatesh: I think this is definitely part of the disruption in and it might be broader than, than SaaS, right? In that if you think of sort of how people deploy software right, in a company today, SaaS makes that motion very convenient, right? So you can just, instead of you hosting a bunch of infrastructure, you're going to say, I'm just going to use the SaaS service x, y or Z. And they also have all of these other systems deployied. But you are right that whether it's SAs or non SaaS, you have exactly two mechanisms to make it do what you want. One is wait for the vendor to fix it. And the second one is train somebody on some super arcane toolt that's specific to that vendor. So you might typically have to go to their conference and their training class to learn how to quoteote program that particular piece of software that's deployed in your company to do what you want. And if you think of how humans approach our problem, usually when somebody joins a company, they won't tell you that I only work with SAP 4.7, whereas applications do. So if you try to put all these things together, it's always been really hard to do it in software. But people are more giving and more able to work with flexible ways to like move across systems. Even though they might be like an older version or something that you're not, you can learn how to use the product and that same person can be effective in that role. So I think if you think of like the agentic enterprise, I would say now you have a layer that kind of has some of those flexible capabilities. I want to Be careful. Like, it's not like, oh, these agents can do everything that a human can do. No way. But what they can do is navigate six different versions of your ERP and they can navigate this other SaaS application in ways that are more flexible than, than what you could do in the past. So I think the main disruption there is instead of having your business user talk to the SaaS application directly, now they're going to interact with an agent which is going to talk to the SaaS application. And that disintermedtion is, that's very exciting for me because there'a lot of consequences that can come from that.
Rob Stevenson: So the agent would be speaking to whom on your behalf?
Ram Venkatesh: To all of the systems that are out there that you've deployed, be it SAS or non SaaS.
Rob Stevenson: Okay, I see.
Ram Venkatesh: So whenever you have the business user talking to something else, so then the system that they previously were talking to now is sort of one level removed from where it was in the architecture. So now it can evolve at a slower pace. You're not tied to the tier point. You can do things in this intermediate layer with the agent that you don't have to wait for anybody to unlock that for you. And best of all, you don't have to have these, these two systems don't have to work with each other. You can work with them and they may be completely agnostic. If there are two SaaS applications that you're using, for example, now you can start to work across them in meaningful and interesting ways for your company.
Rob Stevenson: Okay, I see that makes sense. Rom. Well, here we are creeping on optimal podcast lengthier. And before I let you go, I just wanted to ask you to kind of reflect on the space a little bit. Like when you think of that 12 year old nerd who was printing hello World on a Commodore 64. Yeah, what about the space gets you as excited as that? The tech obsessed boy, like what is the part that makes you feel, didn't feel really excited and optimistic about our space?
Ram Venkatesh: I think that the way we write code actually has not changed for like 29 years or whatever.
Rob Stevenson: Right.
Ram Venkatesh: for a very long time I think. Like these days I find that writing code is again, it's enjoyable. If you're in any of these new environments for writing code with a really good AI companion, like a cursor AI or something like that, it's just so fluid. You're only thinking about the problem that you want to solve. You're able to look at the code and you're able to talk to the code, you're able to kind of shape what you want it to do at such a higher level of abstraction. And we've been waiting so long for this, we didn't know what it was right. But now that we can see how software development itself is evolving, I think that 12 year old me would have built so many more cooler things than we did. All this tech that's out here now, so I'm personally very excited by that.
Rob Stevenson: Yeah, me too. It's well put Rom. Ram, this has been really fun chatting with you man. Thanks for coming on the podcast and sharing what DrBO2 over there at Semaph4. I've really enjoyed learning from you today, so thanks for being here.
Ram Venkatesh: Absolutely. My pleasure.
Rob Stevenson: How AI Happens is brought to you by Sama Samas. Agile Data labeling and model evaluation solutions help enterprise companies maximize the return on investment for generative AI LLM, and computer vision models across retail, finance, automotive and many other industries. For more information head to sa.com.u.