Orchestrate all the Things - The state of AI in 2022 - Evolution, Distribution, Safety. Featuring Nathan Benaich, Air Street Capital General Partner, and lan Hogarth, Plural co-founder

Episode Date: October 12, 2022

Shining light on AI research and applications in a conversation with the curators of the most comprehensive report on all things AI. If you think AI is moving at a breakneck speed and it's almost impossible to keep up, you are not alone. Even if being on top of all things AI is part of your job, it's getting increasingly hard to do that. Nathan Benaich and Ian Hogarth know this all too well, yet somehow they manage. Benaich and Hogarth have solid backgrounds in AI as well as tons of experience and involvement in research, community- and market-driven initiatives. AI is their both their job and their passion and being on top of all things AI comes with the territory. Nathan Benaich is the General Partner of Air Street Capital, a venture capital firm investing in Al-first technology and life science companies. lan Hogarth is a co-founder at Plural, an investment platform for experienced founders to help the most ambitious European startups. Since 2018 Benaich and Hogarth have been publishing their yearly State of AI report, aiming to summarize and share their knowledge with the world. This ever-growing and evolving work covers all the latest and greatest across industry, research and politics. Over time, new sections are added, with this year featuring AI Safety for the first time. Traditionally, Benaich and Hogarth have also been venturing on predictions, with remarkable success. Equally traditionally, we have been connecting with them to discuss their findings every year upon release of the report. This year was no exception, so buckle up and let the ride begin. Article published on VentureBeat.

Welcome to the Orchestrate All the Things podcast. I'm George Amadiotis and we'll be connecting the dots together. Shining light on AI research and applications in a conversation with the curators of the most comprehensive report on all things AI. If you think AI is moving at a breakneck speed and it's almost impossible to keep up, you're not alone. Even if being on top of all things AI is part of your job, it's getting increasingly hard to do that. Nathan Benahy and Ian Hogarth know this all too well, yet somehow they manage. Benahy and Hogarth have solid backgrounds in AI, as well as tons of experience and involvement in research, community, and market-driven initiatives. AI is both their job and their passion, and being

on top of all things AI comes with the territory. Nathan Benach is the general partner of First Street Capital, a venture capital firm investing in AI-first technology and life science companies. Ian Hogarth is a co-founder of Plural, an investment platform for experienced founders to help the most ambitious European startups. Since 2018, Benach and Hogarth have been publishing their yearly State of AI report, aiming to summarize and share their knowledge with the world. This ever-growing and evolving work covers all the latest and greatest across industry, research and politics. Over time, new sections are added, with this year featuring AI safety for the first time. Traditionally, Benach and Hogarth have also been venturing on predictions

with remarkable success. Equally traditionally, we have been connecting with them to discuss their findings every year upon release of the report. This year was no exception, so buckle up, let the ride begin. I hope you will enjoy the podcast. If you like my work, you can follow Link Data Orchestration on Twitter, LinkedIn, and Facebook. I guess, well, congratulations are in order again. I think every year you manage to make the report a little bit better. I think you've had a little bit of help as well this year. So I saw there were two other people credited in the report as well. So good job, everyone. We're definitely looking to expand that.

So very interested folks out there, let us know. Yeah. So I thought I would start actually in a little bit of reverse order compared to how you've chosen to destruct the report. And the item that picked my attention the most is AI safety. The reason being that, well, it's a new entry both for the report,

but also relatively a new topic overall. So I guess a good place to start would be to ask you, what do you think actually is AI safety and why do you think it's important enough that you chose to include it this time? Maybe I should take that one, Nathan? Sounds good. AI safety, I think, is a sort of umbrella term that captures sort of the general goal of making powerful AI systems aligned with human preferences and values. And so there's the sort of the nearer term challenges. So for example,

taking a computer vision system used by law enforcement and trying to understand where it exhibits bias, you know, and for example, in previous years, we've had major sections on facial recognition, use by, you know, law enforcement and use by these systems that has triggered wrongful arrests. So that's one end of AI safety. And the other end of AI safety is what I think is more typically referred to as AI alignment,

which is ensuring that an extremely powerful and super intelligent AI system doesn't ever sort of go rogue and start treating humanity badly in aggregate. The section that we have is very much biased towards that end of safety. And that is because we think that the topic in general is just not receiving enough attention. You have exponential growth in, you know, capabilities of these models, but still a very, very small number of people working on making sure that if they continue to get orders of magnitude more powerful and start displaying, you know, unsettling emergent properties, there are still only a couple of hundred people worldwide working on how we actually make sure that a system that is as intelligent as us treats us well. Okay, interesting. So it sounds like I'm trying to interpret what you just said here, so you can correct me if I'm jumping to conclusions.

But it sounds like the reason basically why you think it's so important is because, to put it in a very populistic way, let's say you think that we're sort of rushing towards AGI and there's no guardrails in place. And this is something that should receive more attention just in case things go wrong. Yes, and I think that's exactly right from my perspective. And I think you're seeing exponential gain in capabilities, exponential use of compute, exponential data being fed into these models.

And yet we have no idea how to solve the alignment problem yet it's still a unsolved technical problem where there are no clear solutions yet and so that's what alarms me and i think that the thing that is probably the most alarming about all of it is that the feedback loops now are so violent you know you have huge kind of wealth creation happening in AI. So there's more and more money flowing into making these models more powerful. There's more geopolitical awareness of the significance of this. And so there's, you know, competitive dynamics between countries accelerating this stuff. For example, the stuff we've covered in previous year on military use of AI.

And there's more social prestige. You know, you get kudos for working at deep mind or open ai and so there's a lot of these powerful feedback loops that are that are kicking in and making these systems more sort of increasing capabilities at a greater rate and we don't have the same feedback loops starting to kick in on Okay. So then I guess the question is, right, I think the way you're portraying the situation is pretty much accurate. So the question is then, what do you think could possibly be done to basically make the deep minds and Googles and metas of the world pay more attention to this issue? I think it's very interesting to think about the comparison with gain-of-function work on biological systems. So if you have a biological system like a virus and you want to do work to increase the sort of gain of, you know, to increase the capabilities

of that system in some way, like, for example, you know, doing gain of function research on a coronavirus, you are not allowed to do that work at a random biotech startup. You have to do that in a facility that is a, you know, biosafety lab that is regulated according to national and international law to ensure that something bad doesn't happen. And yet, if you want to do gain of function research on a computer virus in the way that researchers at Microsoft did last year, where they used machine learning to evolve malware, or if you want to make a system like GPT-3, you know, more powerful, make it into something like GPT-4, there is no regulatory oversight of that. And so I think that the most likely way that we sort of start to put guardrails

around this is, right, some kind of regulation that mirrors the work that we've done on biosafety. Okay, interesting. Well, since you mentioned the world regulation, which I also share the conviction that it's, if not 100% of the solution, at least it's part of the solution. So there's a couple of topics to explore around that. So you mentioned, for example, things like GPT-3 and other large language models. So I think a lot of people would argue that the degree of potential harm that can be caused by those models is very much dependent on where they are applied. It's, you know, having a model like that serve as a substrate for a chatbot is very, very different from having it serve for having it repurposed, for example, to do things like drug discovery or be applied in other fields. And I'm mentioning that because I have something specific in mind, the

EU AI Act, which actually sort of tries to make that distinction. So it makes a, it defines certain categories of AI applications, and it treats them differently, depending on, well, the potential they have for harm, or, you know, how they can affect people. Do you think that the approach makes sense? I don't think that the distinction is actually as straightforward as kind of they're a good and bad application. I think the nature of this being so general purpose is that, you know, intelligence can always be used for sort of good and bad things. So we have a slide, which is slide

44 in the report, which kind of gives examples of, you know, how drug discovery, this kind of flagship AI for good application can be used, you know, for misuse, for example, you know, designing toxic molecules, chemical warfare agents, you know, increased toxicity um i think you know in general anything where you're creating extremely high extremely high degree of intelligence can be can be sort of repurposed in some way in the same way that deep mind's work on applying deep reinforcement learning to um to starcraft was then repurposed for use by the US military for flying, you know, autonomous dogfighter kind of jets. So I don't really think there is a kind of a bright line between kind of good and bad

kind of application categories. I think that just like greater and higher levels of risk within a category. But, you know, if you're like the best, if you're the best human chemist in the world, you can use that skill set for good and, you know, good and evil. And the same is true of kind of almost any other category of AI. All right. So then what could be a practical approach? Would it have to be like a case-to-case examination of applications of AI?

Or would you just say, well, any AI system that is capable above a certain threshold should be regulated? I like the threshold argument, because I think it starts to get into the core of this, which is like the capability, the sort of, you you know the intellectual capability of the system so for example you could say something like any model that in some ways demonstrates more transferable intelligence than a prior model has to be has to you know has to undergo this kind of safety and alignment audit um that would be a that would be the sort of thing you could do um in the limit though like we just have to solve alignment like alignment is like

really the only kind of the only get out of jail free card here because um until we solve that like we are we are basically like you know far more likely to have a negative outcome as a species than a positive one well i have to say this is a very interesting topic and one that i've only like personally started to uh to scratch the surface of very recently so i saw some uh research work that some people did were where they were basically trying to evaluate the ethical, the moral compass, let's say, of large language models. And it's a very hard thing to do, admittedly. It's very hard to even begin how to think about, you know, an evaluation for this type of thing. And it's even harder probably to actually do that. So by all intents, I think that, yes, this is an area that needs more attention and obviously needs a lot more work and a lot more funding to make any kind of progress.

Yeah. One thing we should probably pivot to telling you to talking about is just like the institutional innovation that's happening in the space. I think, Nathan, it might be worth you talking through the distributed research collective stuff, because, you know, we've talked about it a little bit in terms of centralized actors and how one would regulate them. But there's actually this real explosion in decentralized solutions, which might in some ways offer more resilience and better opportunities for kind of, you know, coordination around these problems. Yeah, indeed. That was a point that also stood out for me in the report. And I think you're right. It ties into that conversation pretty neatly. So Nathan, if you want to elaborate on that, feel free. Yeah, the main point here is that the last couple of years,

the central dogma in machine learning has been that of centralization, which is to say that the entities that will profit and advance and be defensible in the era of machine learning are those that can acquire the most resources, whether that's money, talent, compute, and data. That's probably been true over the last couple of years so you know the most powerful labs are those that are owned by google and microsoft and those that are partnered with them but what we saw in the last 12 months or so and is increasingly happening even just this week um is that there is the emergence of these distributed research collectives, which are kind of broadly defined as either like not even companies

or either, you know, Discord servers that emerged last year, such as Luther or nonprofit institutions or startups that are fundamentally open source. And they've really emerged as another pole to do AI research and specifically kind of work that focuses on diffusing and distributing inventions in centralized labs to the masses incredibly quickly. And that's been illustrated by these text-to-image models. And we have a number of slides in the report that present the sort of lineage of Cambrian explosion

of first you have a closed source model and then within a matter of a year, you see the first open source model. And nowadays that timeline has really compressed and it goes across everything from language to images to biology. And so that presents like a real contrast to how like machinery development was done before. And, and, you know, it,

it accesses I think a broader community of people that otherwise wouldn't participate because, you know, to get jobs in some of these big tech companies need to have a PhD, you need to be, you know, extremely technically literate, et cetera. And these open source collectives care a lot less about that and they care more about the value of each person's contribution and the contributions can be different. And so there's that contrast with what happened in the past and then there's the contrast of the substrate that's enabling all the progress in this field, which is the compute infrastructure.

Despite there having been a lot of investment and willingness amongst the community to try and dislodge NVIDIA as the giant in the space that powers everybody, it doesn't seem like that much headwind, that much has been done or achieved to dislodge them. Even though that's what we hear in the community talking to engineers and researchers and people building companies, it's always been very hard to put numbers on that feeling. And so this year, we proactively mined academic literature and just open source AI literature for papers that mentioned the use of a specific hardware platform that was used to train the models that they reported in their results. And we enumerated those papers. And that's where we find that the chasm between the sum of papers that mentioned using any form of NVIDIA hardware

and either papers using TPUs or hardware generated or created by the top five sort of semiconductor companies is sometimes, you know, 100, 150 X. And that hasn't really like changed all that much in the last few years. So despite there being like the sort of busting up of centralized ownership of models and at the software layer that

hasn't really happened at the hardware layer yeah um just a couple of comments on what you said and starting from the whole nvidia versus the rest of the world thing uh i think it's sort of common knowledge for everyone that that has has any kind of relationship to the field that, well, NVIDIA is dominating and has been dominating actually for the last, I don't know, 10 years maybe or so. The real question is, how far can they go? Because, well, a big part of the reason why they're dominating is they had a head start basically from everyone else. And so their architecture is rather, it has its own philosophy and therefore its own limitations, let's say. They keep pushing the limits of what they can do based on this architecture. But just to mention the argument from the other side of the spectrum is like, well, if you're building something from the ground up,

then you don't have those limitations. You don't have to be backwards compatible. On the other hand, you don't have this huge ecosystem and these very well-developed software stack. So do you think there will be at some point a tipping point where things will sort of equal out? I would like that to happen. I think the industry would like that too, but the data doesn't suggest that because the incumbent behaves like a startup.

It's also incredibly innovative with how it's making use of machine learning to advance its machine learning hardware and has been reported to use RL to design the H100, which itself is reported to be like 10x better, roughly speaking, than the A100. The developer ecosystem is massive and the tooling is well understood.

So I think the question is always like, how much better can a new design be? And how much better does it have to be if its software is less well understood and the learning curve is higher and you're also fighting with like a massive install base that many companies already have and so I think it's just like a really hard uphill battle that you know has been has been fought for at least you years or so. And we would have thought that the chasm would be narrower than where it is if the future would actually look more distributed than pure NVIDIA.

Just to add some anecdotal evidence to what you just started, which I think, by the way, is a great initiative. And it would be interesting to keep doing that year after year. So we have like year over year statistics to compare to see if the chasm is indeed closing in some way. So just to add some anecdotal evidence from that, I've been talking to some people from a startup recently called Run AI. And what they do is basically they offer this sort of virtual layer that sits on top of AI hardware and tries to abstract the details for other people that do work on top of it.

And I asked them this exact same question. So because of the fact that you are in that position and you work with all this different hardware, how do you see adoption playing out out there? And their answer was that, well, basically nine out of ten or maybe 9.5 out of ten of our clients are still using Nvidia. There's some adoption of new hardware, but not that much, really. So it goes to back up what you also found.

And just to return to your previous point about how research and progress in general in machine learning is going a little bit distributed. I mean, yes, you're right. That's definitely happening. And the timelines that you mentioned are quite impressive, to be honest. It's definitely been a speed-up in how proprietary models are being, well, not open source themselves, but how we see the equivalent, let's say, of those models become open source. The big question I have around that is, well, how practical that really is, well, both for the people who do the job of open sourcing and for the people who may potentially want to use those open source models. And I mean, in terms of having an open source model out there is one thing, but being able to

use it is another thing. So you need things like documentation and training and all of those things. And I'm pretty sure not many of the people who would potentially like to use those models are actually able to. So that's one part of the equation. The other part is, what do the people who participate in those open source efforts actually get out of that? Do you see like a viable business model growing out of these efforts? Yeah, I think there's, I think there's actually quite a big opportunity to do something quite

innovative on the latter point of like, how do you incentivize the community to build on your project? Especially as you know, we've been discussing in other settings with Ian, like this question of if you're training massive models that have access to like all of humanity's information, then who should own those models? And if everybody's contributing towards engineering those systems, then should there be some form of like shared ownership? So I guess it's an open question that hasn't like particularly been like a solution to that hasn't been implemented

yet. But I feel like there's a tension that will arise at some point because the economic value of creating powerful AI systems is pretty massive. Okay. Yeah, it's definitely an open question. And speaking about things around ownership, really, I'd like to quickly jump to a slightly different topic, but there is some similarity. missing from this year's report is diffusion and the models that have been generated around that using a diffusion which in turn have spurred a whole series of conversations including the one about well are do this is what these models generate really art and if it is art then who's the artist is it the model is it the one who prompts the model? Who should own it?

Should people be allowed to make money out of it? And so on and so forth. What's your take on that? I mean, again, quite an evolving topic. And I mean, the probably most extreme version of this recently was that Atlantic author who was basically forced to take down the image that he generated of mid-journey in an article because of these ownership questions. Perhaps like another version of this is that it'll prompt more formal partnerships between AI companies and generators of these models and corpus owners, especially large corpus owners,

which perhaps could be a prediction for the next 12 months. But I think ultimately it's perhaps a question over what is the incremental value of like additional data points in this broad data set and if it's not clear that you know an individual contributor to a broad data set like really moves the needle on model performance then to what degree can an individual really like influence like this this debate basically or would there have to be like a on mass sort of um demand to like not have work be present in a training data set to influence like this question

of ownership but i think it's just like quite quite evolving and probably like long-term like these systems if they are trained on like everyone's data shouldn't necessarily be owned by a single party whether that's a company or national i do think that the the sort of the economic model is massively in in in flux right now because of what you know a lutheran stable, Stability AI have done to the centralized kind of closed source platforms. You know, if you were planning to have an API that monetizes a sort of generative image model and, you know, you have that sort of behind a sort of paywall,

and then suddenly there's a sort of an open source project that offers the same, you know, quality experience in a self-service kind of, you know, you know, non-commercial way, you're going to see a real, a real sort of tension. And we don't really yet know what the like the end state is in terms of how this is, how this is monetized and, and who it's ultimately owned by. So it feels like a very fertile time where people are really experimenting with quite different models to both owning and monetizing these systems.

Okay, thanks. Well, let's wrap up with one final topic this time from research. So it's something that we've actually discussed before, the whole conversation around scale, really, and whether just scaling things up can lead to emergent properties, to which, by the way, there has been some research that tries to set some light in the past year. And the initial findings that the people discovered was that well maybe it's not a definite conclusion but it seems that there are indeed some emergent properties in large language models so the quick question I guess would be and you know there's also the whole discussion or argument debate whatever it is you want to call it that has broken out lately between two of the figureheads

of the two different schools of thought around AI. So, Jan Lekan and Gary Marcus, which I'm sure you're probably aware of. So, has there anything transpired in the last year that has either impressed you or made you change the way you see things around this fundamental divide, let's say, in AI? Yeah, I think one... Sorry, go ahead.

No, you go. One of the reasons why we made the statement that large kind of like large ai systems or just ai research in general is sort of going a bit more towards cognitive science is some recent results around um like asking language models to reason step by step or think step by step so instead of going you know from input to output saying first do this then do this then do that and how that's um you know sort of a baby step in demonstrating that um providing some logical reasoning can actually help performance but also like trustworthiness of these systems so i think that's that's been like a like a cool kind of inch in that direction i'll let you know yeah i mean i think that like inch in that direction? I think what Gary Marcus is doing is extremely unhelpful.

Personally, I think that it's the opposite of ringing the fire alarm. And I think that the pace of progress here should merit us to take a more precautionary view. It reminds me a little bit of the early days of COVID where there were a lot of people basically saying, oh, yeah, it's just that, who kind of are skeptical about the sort of severity of the response to you know new information about something growing exponentially and i think that you know um you know the work that you know the

the um you know people at eliezer yudikowski have been doing for you know frankly decades at this point is is kind of some of the most important work because they are they're they're framing this this this point that actually we need to take this these risks far more seriously and i think that kind of just like constantly moving constantly sort of like um critiquing the you know the the capabilities of these systems isn't helpful. Okay. Well, actually, that's a topic that has lots and lots of meat. But, well, I know that our time is actually over.

So I guess we'll have to wrap up here. And thanks again for your time. And, well, again, great work. And I guess a lot of people will get to enjoy by reading it. I hope you enjoyed the podcast. If you like my work, you can follow Link Data Orchestration on Twitter, LinkedIn and Facebook.