Rizel describes how she helped others to understand how GitHub Copilot should be used, her duties and ongoing projects at Block, how the conversation around AI copilots has evolved in the last 18 months, and the importance of having a refined system for objection handling.
Today we are joined by Developer Advocate at Block, Rizel Scarlett, who is here to explain how to bridge the gap between the technical and non-technical aspects of a business. We also learn about AI hallucinations and how Rizel and Block approach this particular pain point, the burdens of responsibility of AI users, why it’s important to make AI tools accessible to all, and the ins and outs of G{Code} House – a learning community for Indigenous and women of color in tech. To end, Rizel explains what needs to be done to break down barriers to entry for the G{Code} population in tech, and she describes the ideal relationship between a developer advocate and the technical arm of a business.
Key Points From This Episode:
Quotes:
“Every company is embedding AI into their product someway somehow, so it’s being more embraced.” — @blackgirlbytes [0:11:37]
“I always respect someone that’s like, ‘I don’t know, but this is the closest I can get to it.’” — @blackgirlbytes [0:15:25]
“With AI tools, when you’re more specific, the results are more refined.” — @blackgirlbytes [0:16:29]
Links Mentioned in Today’s Episode:
>> Rob Stevenson: Welcome to How AI Happens, a podcast where experts explain their work at the cutting edge of artificial intelligence. You'll hear from AI researchers, data scientists, and machine learning engineers as they get technical about, the most exciting developments in their field and the challenges they're facing along the way. I'm your host, Rob Stevenson, and we're about to learn How AI Happens. Okay, we are back here with another classic installment of How AI Happens. I'm your host, Rob Stevenson, and I've been looking forward to this conversation that I'm about to have today because our guest has had a ton of interesting roles in our space and has been working on some projects that I'm guessing the folks out there listening have used themselves. Our guest has been a developer advocate at GitHub where she worked on generative AI tools before they were cool tools like GitHub Copilot. She serves as the advisor and board member over at G Code House. Currently she is a staff developer advocate at Block Rizel Scarlett, welcome to the podcast. How are you today?
>> Rizel Scarlett: I'm good. Thanks for having me. I'm excited to speak to you as well.
>> Rob Stevenson: I'm thrilled to have you. You are, broadcasting in. I just to paint a picture of the folks that are only listening, you've got the soft purple euphoria lighting in your office today. You've got the whole set up looking great, sounding great. So I'm thrilled you're here. And did I do justice to your curriculum vi there when I was kind of stumbling through your experience? Anything I left out there, you did amazing.
>> Rizel Scarlett: I think that pretty much covers the roles that I have right now.
>> Rob Stevenson: Cool. Well, you have this technical background, but you have served as this developer advocate at a couple different places now. And the developer advocate role is. I'm seeing it come up more and more. I assume folks that there listening are familiar with it, but I would just love to hear it directly from you. How do you kind of characterize your remit? What does that role mean to you?
>> Rizel Scarlett: Yeah, I would say when a company hires a developer advocate, it could be for many different reasons. There's different types of roles that a developer advocate can kind of play, but the whole idea is basically connecting the business with the developers. Right. So sometimes the business has developed this developer tool and they want other developers to be able to use it, but they're not really sure how to market it to them in a way that's like authentic because the developers are really like, sensitive, I guess. So, like, oh, this is the fastest, most performant, like best tool ever. Like, developers are like, okay, sure. So it helps to have someone who's technical or has been the developer before to kind of show how to use that tool in a way that's not like sleazy, like you're really trying to push it on them. So my role might look like writing blog posts about myself leveraging the tool or speaking at a conference, or just interacting with other developers and using that tool. And then from there other developers or potential users can say, hey, I think I like to use that. Or they might say, that might not work for my use case.
>> Rob Stevenson: Okay, yeah, that makes sense. Thanks for walking me through that. Technical arms of the business are frequently misunderstood by the non technical parts of the business. And it kind of comes up a lot on this show, this idea of how you position your team and yourself and your skill set to the rest of the org who are going to be making requests of you, particularly in AI. AI is this magic wand that can be waved over problems and then like, okay, tech side of the business, you go solve this. And being able to provide this sort of feedback boundary, what have you is an increasingly important skill set for folks to have. And in your case, you've been brought in deliberately because of that skill set, maybe to sort of serve as a bridge from the more technical parts of the business to the non technical parts. Is that a fair, is that's not an oversimplification, is it?
>> Rizel Scarlett: No, I think that's pretty fair. Yeah.
>> Rob Stevenson: Got it. Cool. So, yeah, I think most people will probably not have the luxury of a developer advocate to kind of go between them and the other sides of the business, but at a block that does exist. So what do you think are the kind of fundamental ways that non technical parts of the business misunderstand the tech side, which I guess would just amount to your main focus?
>> Rizel Scarlett: I think I could use GitHub as an example. I think when it first came out, the marketing team took one approach. They're like, oh, look, it helps you to code developer productivity and stuff like that. So when it was first rolled out, developers didn't really understand how it will work. They just installed it and they're like, it's not coding for me, like you all said. And then that I think was like, maybe a gap. Because the marketing people might not have realized, like, oh, we should tell them that you have to type something first, you have to prompt it with a comment and stuff like that. So I think when my team came in, we kind of helped to kind of show how that works, give them Tips for prompting it better. And same with any developer advocacy stuff here. I think right now it's not really for block the AI tool. It is not really super focused or the business side is not really like super focused on being the voice for it, but the engineers are. And I still think there's a little bit of a gap where the engineers are like, whoa, you can make this really super complicated thing. And then other users are like, but what is it? So I've been able to maybe break it down to like a lower level. And I'm like, you can like even this really simple example of making a to do app, and people are like, oh, okay, I get it. I don't know if that made sense. I'm kind of rambling. But yeah, feel free to like, help me expand on that a bit.
>> Rob Stevenson: Yeah, no, that makes sense. You are sort of like, because the value of whatever the tool is is so obvious and taken for granted by the people who make it because they've been living with it for months of development, right? And then they have to meet people where they are like, as a developer, you are on step 12 and everyone else that this tool is for is maybe on step one, Right? Enter razl.
>> Rizel Scarlett: Exactly. Like, I basically have to gauge, like, has this been promoted or shown to developers in a way that's too simple or in a way that is too complicated and people don't know where to start? So I can be like, okay, I need to make this for big. Like, make this clear for beginners or make this clear for more advanced folk.
>> Rob Stevenson: Yeah, yeah, that makes sense. So when it came to Copilot, marketing sort of is over promised or just mis promised or. Or just like, I don't know, it sounds like that there was just this gap in between what was advertised versus what the actual use case was. So what was your process for coming in and actually showing how Copilot ought to be used?
>> Rizel Scarlett: Yeah, totally. I can go into that. Right? So to give context at least, this is like the perspective that I had. I did not just to make sure, like, clarified everybody. I didn't create GitHub Copilot or really have a hand in it. The engineers did an amazing job at developing it. And I joined GitHub as a junior developer advocate around the time that GitHub Copilot was in like beta or technical preview. And there was all these like, really mixed reviews on social media where people were like, I don't really like this. Or some people were like, I really like it. And I was like, this is interesting. So really, my team didn't tell me, like, go look into it or anything. I was just like, I want to know the truth. Like, is it something that's bad or is it something that's good? Because I would go on Twitter or like, join a Twitter space, and people would be like, it's actually stealing your code. And I'm like, I don't know if I want toa work somewhere where it's stealing code. So I was like, let me try to, like, play around with it. So after a few weeks of playing around with it, I was like, oh, okay. I kind of see how to approach GitHub Copilot. And what I had been doing at this time was like, writing blog posts. And I just started writing blog posts about, like, oh, here's how I was able to do XYZ withithub copilot. Or I just discovered this really cool feature. Withithub copilot. I was able to, like, translate a code snippet from JavaScript to Python or whatever else. And I think people started to kind of pick that up inside of GitHub and outside of GitHub. And I think that helped people to kind of approach GitHub in better ways. Not saying I was the only person doing that, but I do think I helped to push forward the understanding of how GitHub Copilot was working and how people could leverage it.
>> Rob Stevenson: Okay, yeah, that makes sense. And again, like, it is just a different skillt than the folks developing the tools tend to have. The developers don't have the marketing chops, the marketers don't have the technical chops. Right. So whereas that feels like is what creates the need for developer advocacy.
>> Rizel Scarlett: Yes, exactly. There's so many different types of developer advocates, but I think what you just described is like, where I thrive.
>> Rob Stevenson: So what is the advocacy in your current role? What are the kind of the projects where you're like, okay, I, need to go be that in between.
>> Rizel Scarlett: Yeah. Recently, well, I joined Block, and they had different business units. One was called tbd and that was focused a little bit more on, like, decentralized identity. I was just curious about it. Not necessarily AI. I was like, let's see how that works. And that was really interesting, was open source as well. But we've kind of pivoted away. Like, Block has kind of pivoted away from that business unit. So now I've been focused on some of our AI tools. One of them is called Goose. It's an AI agent that, enables developers to kind of It's a productivity tool. It's basically an AI developer agent that will help you to code. You give it one prompt and it's basically pretty semi autonomous where it'll just go through everything for you. Whereas maybe a tool like GitHub, Copilot or Clau or something like that, you have to give it one prompt at a time and it does it. But you can tell Goose, like, hey, I want to build out this entire app. And it'll just be like, okay, I'm gonna install this, do this, do that, like step once through 10 or whatever and it'll do it for you. So that's what I've been focused on right now.
>> Rob Stevenson: Gotcha. So, you have. I mean, it's only been like a year and change between those two projects, but that's kind of an eternity in generative time. And so because it's part of your role to kind of monitor the conversation and how people are understanding or misunderstanding, more accurately, how to a lot to be used. I'm curious how you've seen the conversation shift when it comes to copilot tools and generative tools. What's been the change over the last like, you know, 18 months or so?
>> Rizel Scarlett: Yeah, I think the change was. At first there was more fe, maybe confusion. How is it really working to Now I'm seeing more developers maybe relying on a lot of these tools to use it, but there's still a lot of questions that people have that, I'd love to figure out the answers to. Like sometimes people are curious about. Like AI hallucinations. That's a big frustration point for developers. Or. Yeah, I think AI hallucinations is like a top one. That's like the top of my mind. But yeah, I think right now we've gone from like, what is this weird tool? Like, how is it working? Is it taking my code? Like, how was it trained to. Now everyone's like, oh, I wanna try this tool out. Every company is like embedding AI into their product some way somehow. So it's being more embraced.
>> Rob Stevenson: To use like sales parlance, there's a little bit of objection handling it sounds like that you need to do where you were saying in the early days, GitHub copilot stealing your code. And so then you just went off to investigate whether that was or was not the case. Now it's hallucinations. There are these critiques, some accurate, some not. Either way, the organization has to address them. Right. If you want people to use your product.
>> Rizel Scarlett: Yeah, yeah, I definitely think that, like, I Think it hurts a, company when they are like, oh, I don't want to say anything about it. Like I don't want us to get in trouble. I'm like, just explain_ain, how the tool works because developers like to have that visibility and understanding before they dive into using a tool.
>> Rob Stevenson: So currently that is hallucinations, as you said, what is like the fear and how are you planning to address this to make people a little more trusting of block perhaps?
>> Rizel Scarlett: Yeah, I think the fear around hallucinations is like it'sometimes an AI tool sounds so confident in like how you can approach a problem and then you've invested so many hours going down like a rabbit hole where you're like trying use a method and you realize it doesn't work. And I think the way I've been approaching it so far is like explaining to people like you don't want to rely only on the AI tool. Like continue to use your own, like brain, I guess, and your own research. I mean, to be honest, it sounds mean, but yeah, like continue to use your own reasoning to be like, wait, does this make sense? And then also like think about the fact that like Most of these AI tools, these LLMs, they don't know anything. They're just using like predictive reasoning. Right? It's just like math and matching things and they just spit out whatever is like, oh, this is a good match. So they don't actually know anything. So think about that and kind of go back and forth with the AI tool and be like, why did you, like, what's the context for suggesting this method and all that to kind of help you reason better with the hallucinations.
>> Rob Stevenson: Yeah, that's an interesting call out, right? That it is doing this predictive thing. It's regurgitating a little bit, but it's not reasoning. And we've spoken a little bit about the recent research paper that Apple put out, sort of like testing the reasoning ability of all these LLMs. And for my money, it was a pretty big javelin through the heart of short term AI hype. Right? And the answer is just like, look, it's not thinking on its own. And that's important to remember when using the tool. And with hallucinations in particular, the problem feels like a generative tool is never gonna say I don't know. Right. It's not going to be like, no results found. It's gonna say something.
>> Rizel Scarlett: Yeah.
>> Rob Stevenson: Does that feel like an important hurdle? Like shouldn't it say I don't know at some point?
>> Rizel Scarlett: Yeah, I think So I think that would be helpful because, like, you just said, it's super overconfident. Or sometimes it even. I don't know if you've ever experienced where you tell it something and then you tell it, no, that's not true. And they're like, yes, sorry, you're always right.
>> Rob Stevenson: Yeah. Just immediately backpedals.
>> Rizel Scarlett: Yeah. I'm like, okay, so I would love to see some AI tools go in that direction where it's okay to say I don't know, but I understand that it's not necessarily, like, programmed that way.
>> Rob Stevenson: Yeah. You as a user, you feel like it has to give you an answer. And if it said, I don't know, then would that undercut its utility? Like, maybe the belief of a developer might be like, oh, it's better to give a slightly wrong answer than no answer at all, which feels like a gross incentive.
>> Rizel Scarlett: O, that's a good question. I don't know for other people in particular, but I always respect someone that's like, o, I don't know. But like, this is the closest I can get to it. So even if.
>> Rob Stevenson: Or let me find the answer for you. Yeah, I've respect that a lot. Like, hey, I don't know, have that in front of me. Let me get back to you. I would rather you do that than just make something up. But. But the compulsion is always to make something up.
>> Rizel Scarlett: Yeah. It's kind of like how. How real humans act sometimes.
>> Rob Stevenson: Exactly. Yeah. So then when it comes to assuaging, the concerns about hallucination, what is the approach for, for you and Block? Is there a way to kind of show off? Hey, here's how we're addressing it. here, here's how we're trying to mitigate it. How would you go about trying to make people feel better?
>> Rizel Scarlett: Yeah. While we don't have any specific publications out yet about this, we do have different tools built in place of the Goose tool or the Goose AI developer agent. So one thing we have is called Goose Hints, where it's just like this text file and you give it like, extra context of where exactly you're trying to go. And I find that with AI tools, when you're more specific, like, the results are like, more refined. Right. So we have Goose Hints. We have another thing called Plans, where Goose, because it's semi autonomous, it will essentially go and Talk to an LLM like OpenAI'GPT4 or whatever result said she wanted to make a web app that does this or create unit tests that do blah, blah, blah. Can you Help me create a plan for this. But if you wanted you could create like as the user, you can create your own custom plan. That way the LLM doesn't just hallucinate its own thing and you're like, I already know what I want it to do. I'm just maybe too lazy to manually do it or want to save some time. So that can also help the process of refining the output as well. So there's a lot of different things in place to make sure that the AI tool goose hallucinates less.
>> Rob Stevenson: Okay, yeah, those are good examples and it kind of goes back to what you said earlier about like hey, use your zero and brain a little bit. Like in the example of the semi autonomous part where it can kind of provide what sounds like an attention mechanism to be like here's my plan for going to generate this response. Is this right? How much responsibility can we expect users to take?
>> Rizel Scarlett: Yeah, I think it depends on the purpose of the AI tool. Right? Like I think Microsoft had their Microsoft ignite thing or conference and they announced like all these different AI agents that they had. Those didn't seem, or when day they announced it, they didn't seem geared toward developers. So they may have more things in place to make sure like hey, there's these guard rails and you don't need to customize it as much. But for a developer, we know that a developer might be a little bit more skilled and I think that you can basically not skilled but they may feel more comfortable going in and editing things and that kind of gives them the ability to just say hey, I want it to go this specific way. I really think it's just like what type of agent is this? Is this like a customer agent or is this like a developer tool agent? And like the way I think of it too, it is s kind of like you have people who know how to use a computer, but then you have people that like to customize their own like their computer, they like to build it on their own. I'm not that type of person. But like the goose tool that's geared towards those type of people that are like I want toa customize this on my own. So it's not really like oh, some tools are like putting the responsibility on the user. It's like they're giving them that optionality if they wanted to.
>> Rob Stevenson: This is a challenge because the compulsion from folks I'm speaking to is like how do we get these tools into the hands of everyone? How do we democratize access to these? And yet the tools themselves, unless they're in the hands of someone really, really skilled, might not be that reliable. So I don't know where we come down there if it really is a matter of stakes or like more guardrails for less skilled people. But I mean that's also, it's not like a skilled person. That means they also are going to use something morally either. So it's not, it's not as cut and dried as like less guardrails for a better trained person.
>> Rizel Scarlett: I mean maybe it comes down to education as well. Like maybe we need to be educating non developers as well on like how do we really leverage this tool? Cause that did make me think of like my friend, she's a professor and she sees her students using Chat GBT for social work essays and stuff like that. And I use those tools for blog posts, but I'm not gonna like just copy and paste it and just trust everything it said. I still wanted to have my voice. So maybe it's like we need to find a way to kind of explain to other folks who are not really understanding how AI works. Like okay, this is what might happen with your home loan application or be aware of this when you're writing an essay. I don't know. That's the only thing that I could think of like that would be like low hanging fruit.
>> Rob Stevenson: Yeah. And that was the case when computers came out, you know, and when chat rooms and the Internet started to go into people's hands. It's like people need to understand what they're dealing with. That was a part of my education was learning how to, you know, when it came to developing websites, when it came to using the Internet, when it came to using Wikipedia, like using chat GPT, it's the same for, to write an essay for example is the same as using Wikipedia to you wouldn't just copy paste the text. So it's just the next generation of plagiarism. And so this is. Teachers need to be more nimble, I suppose, but it's the same problem.
>> Rizel Scarlett: Yeah, that's true. It's a difficult problem. And yeah, you just reminded me how much my teachers used to be like don't use Wikipedia constantly.
>> Rob Stevenson: Yeah, I had a, I had a professor say that and she like brought up Wikipedia and then she's like, see how easy it is to just put in the wrong information? And she just like edited the article live there in class.
>> Rizel Scarlett: Oh, that's a good example.
>> Rob Stevenson: Yeah, except I was the jackass who raised my hand six minutes later to point out that her edit had been undone.
>> Rizel Scarlett: You were re like, well, they fixed.
>> Rob Stevenson: It, they fixed it right away. I didn't look at it in the six minute window that you edited it. So I'm fine. But yeah, anyway, this is, it's a point well taken that this is a matter of education. I tend to agree with you there. And now result. Before I let you go, I wanted to make sure we speak about your work with the G Code House because it feels really important for companies who wish to invest in the next generation of talent and have more representative talent in their own organizations to partner with companies like this. And so I would love it if you could just explain a little bit about first what G Code House is and then also your work with them.
>> Rizel Scarlett: Gotcha. Yeah. G Code House is basically a program that exposes one of color and non binary people of color to code and we make sure that they have other things in place for them like food or housing or whatever so that they don't have to worry about those external things. Cause sometimes that's a huge barrier. When we notice that there's not that much diversity in tech, it's often because we don't really have the privilege to focus on that. And then in terms of like companies thinking about bringing in or hiring this diverse talent, it's beneficial for many reasons. I know we're talking about AI right now. It's beneficial for the reasons of getting better results from AI. A lot of times AI is really focused on one specific demographic because that's the training data that's been out there. So it's good to get folks from all backgrounds involved.
>> Rob Stevenson: Gotcha. So what would you say when it comes to folks from these backgrounds participating in tech, in addition to having basic needs met, which it sounds like Geo House is invested in, what is standing between these folks and having a career like yours and getting access to tech?
>> Rizel Scarlett: Oh, interesting. I think like one of the things like I mentioned was just, I guess, general privilege. I also took a non traditional route where I went to a, coding boot camp that paid me. And it's not like one of those ones that asked me to pay them back or anything. They literally just paid me. And that made it so much easier because oftentime you're like, yo, I need to be in a stable job, I want to take this class, but I can't. And then I think other things that can be in the way are like, I mean the teams often are just mostly white male, so it can feel like you don't belong. It can be discouraging and oftentimes women of color, even just women in general, they end up leaving. So I think it's the lack of having a stable path into tech and then the lack of having a way to feel comfortable andn to stay that like your career may be shortened and you just end up being junior and you go to something else. So yeah, I think that's the only thing that is between them and getting this AI career. I guess.
>> Rob Stevenson: Yeah. You know, I like how you explained in your background you were paid to take this boot camp in a way functions almost as like a scholarship for you to receive this education. I never considered this idea to like I always figured a boot camp is like an alternative to a four year degree in a lot of cases and I've seen in the last decade plus of my career companies shifting to be like, yeah, a boot camp is perfectly good training. Like we will accept that we want those folks on our team. But even compared to a four year degree, paying for a boot camp and spending the time to go to a boot camp, that still does require privilege. Right. That still requires the ability to make the investment and the ability to spend the time. So. So it's like closer but it's not perfect in terms of engaging I think with the type of communities that it was intended to. So this is just a shout out for the folks listening because I feel like they are going to be in positions to treat with organizations like G Code House.
>> Rizel Scarlett: Yeah.
>> Rob Stevenson: And to keep that in mind that you might believe that you are being more inclusive by treating with a hacker house, by treating with a bootc Camp, for example, keep this in mind that who can still afford the time and the investment to go to it? Is that still the kind of person that would make your organization more representative? It's a reflection I want people to make. Which is why I wanted you to share about G Code coau. So thank you for walking me through a little bit about that background.
>> Rizel Scarlett: Thank you. And that's a really good reflection.
>> Rob Stevenson: And before I let you go Rol, I wanted to ask you like as a developer advocate, what do you think is the ideal relationship between you and the technical arms of the business for whom you translate?
>> Rizel Scarlett: Yeah, I would say if they understand the role of deverel. Sometimes people think like as a developer advocate or whatever, I'm doing everything. Like I think it's a developer advocacy is a whole company effort. Like sometimes it's just one person that's on the developer advocacy team or three or whatever. It's a small group so, like, I can create a strategy, like content strategy with you. I can think about, like, what relationships we can create. But I feel great when the technical team is like, okay, here's the part that will play in this. We'll maybe keep good communication with you or we'll continue to do blog posts as well, or whatever it is where it's not only like solely on the developer advocacy team to get it out there.
>> Rob Stevenson: Okay, that's great advice. Yeah. I wanted people to be able to, like, in the event they find themselves to someone like you, how do they make the most out of it? So. So that is helpful. Rol, this has been really fun chatting with you today. Thank you for coming on the show and sharing your experience. this has been a delight. So at this point I would just say thanks for being here.
>> Rizel Scarlett: Thank you for having me. This was great.
>> Rob Stevenson: How AI Happens is brought to you by Sama Sama'agile data labeling and model evaluation solutions help enterprise companies maximize the return on investment for generative AI, LLM and computer vision models across retail, finance, automotive and many other industries. For more information, head to sa.com.