The AI Daily Brief: Artificial Intelligence News and Analysis - The Things We Know About AI So Far

Episode Date: January 13, 2024

A reading and exploration of: https://www.oneusefulthing.org/p/signs-and-portents ABOUT THE AI BREAKDOWN The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI breakdown, we're reading and discussing a future-looking prediction post from Wharton's Ethan Mollick. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown.network for more information about our YouTube, our Discord, and our newsletter. Hello, friends. Welcome back to another Long Reeds episode. Today, I am excited to read the latest post from Ethan Mollick, a very insightful professor

at Warton who always has interesting things to say about artificial intelligence. His substack is called One Useful Thing. You can find it at One Useful Thing.org. And this post is called signs and portents, some hints about what the next year of AI looks like. I'm actually going to turn it over to AI me from 11 Labs, for the majority of the reading, but then I'll be back to discuss some of Ethan's discussion points.

As we begin the second year of our AI moment, it is still too early and dramatic to call it the AI age. It is time to consider the future. To be clear, nobody can tell you the future of AI. accurately, except that AI development seems to be happening much, much faster than even experts expected. We can be confident about that because a new paper just came out surveying almost 3,000 published AI researchers, following up on a similar paper published a year earlier. The average estimated date for when AI could beat humans at every possible task shifted dramatically,

moving from 2060 to 2047, a decrease of 13 years in just the past year alone. And the collective estimate was that there was a 10% chance that it would happen by 2027, With so much changing so quickly, we need to take predictions with a grain of salt, but that doesn't mean we can't say anything useful about the coming year in AI. To ground ourselves, we can start with two quotes that should inform any estimates about the future. The first is Amera's Law. We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run. Social change is slower than technological change.

We should not expect to see immediate global effects of AI in a major way, no matter how fast it's adoption, and it is remarkably fast. yet we certainly will see it sooner than many people think. While Amara's law speaks to aggregate change, real change often originates in smaller communities and pockets, among user innovators and those with extreme needs, or in the research being done in labs and universities. If we want to understand what AI will do,

we need to look for these early effects. This brings us to our second quote by William Gibson, who famously wrote, The Future is Already Here, it is just unevenly distributed, a point backed up by decades of research on user innovation. So rather than predict the future by speculating, it is worth looking at the places where it is already occurring.

Here is a rapid tour of the signs and portents that signal the future of AI. Section. AI is already impacting work. There are now enough careful studies of the use of AI in real work to draw three conclusions about how GPT 4-level AI's impact work performance. One, AI boosts overall performance at complex work tasks. In the large-scale controlled trial that my colleagues and I conducted at Boston Consulting Group, we found consultants using

the same version of GPT4, everyone in the world has access to 12.2% more tasks on average, completed tasks 25.1% more quickly, and produced 40% higher quality results than those without the tool. A new paper looking at legal work done by law students found the same results. As have studies on writing, programming, and innovation, all of these papers rely on chatbots, but the impact seems to persist even when AI is integrated into other software. Microsoft found large-scale performance improvements in meetings and emails from their GPT4 power co-pilot AI, which is built into office applications, though it is their product, so take it with a grain of salt. Overall, these are very consistent results across many studies. A 20% 80% improvement

across wide ranges of tasks without training or integration work is unheard of. And it suggests that AI is going to play a big role in work. Two, the effects are largest for lower performers for now. Another universal result, which you can see in the graphs from the legal paper above, is that AI acts as a leveler, helping low performers more than high performers. This may temporary. Current AIs are good enough that they essentially do solid work on their own, raising the level of lower performers who use them. In the long run, it may that AI ends up helping high performers more

or follow some other pattern entirely. We will learn a lot more about this in the coming year. 3. The Jagged Frontier. AI is better at some tasks than others. Multiple studies have identified what we refer to as the jagged frontier, because AI is excellent at some tasks that seem hard to humans, and bad at some tasks that seem easy, it is hard to know what it is good at in advance. So the only way to understand what AI can do

is to use it and see what happens. Only with experience can you understand the shape of the frontier and learn to avoid relying on AI in cases where it does not operate well, though the frontier of what AI's can do is constantly expanding. Despite these early findings and as you might expect from Amara's law, many companies have been slow to deploy AI. Actually, it is slightly worse than that. Most companies have either ignored AI, though their employees are using it all the time, or decided to treat it as some sort of standard knowledge management tool, a task that LLMs are not actually that good at. This failure of imagination will likely continue and continue to hurt companies, but more leaders are going to wake up to the nature of AI as transformational in the coming

year. I know this because anxiety in the C-suite is unevenly distributed, and there are already exceptions to the general AI complacency. For example, Eric Vaughn, the CEO of Software Company Ignite Tech, saw the implications of AI quickly and made its use mandatory throughout the company this summer. He gave all employees access to a chat GPT Plus subscription and some training, with the expectation that everyone in the company would engage in experimentation. Eric was extremely serious about the need to transform that company. He fired everyone who did not at least try to use AI by the end of the month and gave cash prizes to people who came up with good prompts to improve their work. A much more dramatic embrace of AI than I have seen elsewhere, but other companies

are considering transformation as well. Another organization I spoke to altered their hiring policy. Before a new hire is approved, the team has to spend a couple hours trying to automate the potential job as using AI. They can only post new openings after they see how much AI can supplement or replace the need for a new employee. The experiences of these early adopters suggest that productivity improvements could significantly alter the nature of jobs. And indeed, there are some troubling signs about the potential job market impact of AI on freelance workers. A new study of a major freelancing platform found that posting for jobs that could be done with AI declined 21% after ChatGPT was introduced, and a 17% drop in graphic design jobs after image-creating AIs were released. We are likely to see similar effects ripple through the economy,

though a survey of economists still sees net job growth as a result of AI, there will certainly be many changes to individual jobs that policymakers need to consider. Despite the potential for disruption, another common finding in studies is that people who use AI for work are happier with their jobs, because they outsource the boring work to the AI. Leaders and managers of organizations need to think hard about how to capitalize on the positive aspects of AI-driven change while avoiding the negative. A successful 2024 for AI's role in work will be one in which considerable effort goes to

thinking about how to transform organizations, taking advantage of the inherent power and weirdness of AI in order to help both workers and companies flourish. Section AI is already altering the truthway back in February I wrote about how easy it was to create a deep fake of myself. In the last year, the technology has come so much further. In addition to voice and video, AI images have become incredibly convincing. Realistic images of almost anyone and any scene are easy to make.

They are already circulating, and you will certainly be seeing a lot more of this, especially with upcoming election years in many countries. Almost every area of information security is going to be altered in the coming year. For example, AIs can now complete CAPCHAs and send convincing emails to get people to click on malicious links. These changes are now inevitable. Open source models, which are free to use and modify, can already fake voices, photos, emails, and more. And they can run on a home computer. Even if we shut down AI development, the information landscape post-2023 will never be the same as it was before. I don't think most

people are ready for what that means for privacy, security, come up with a secret family password to prove your identity now, and global politics. Section. AI is already effective at helping learning. If you have been reading this substack, you know I am especially interested in the use of AI in education. I have written about our papers and prompts for teaching with AI as well as classroom experiments. I hope to have a really in-depth dive into this topic in my next post, including lots of new examples of prompts and approaches. Many other instructors are also experimenting with AI and sharing what they are learning, but we haven't had good data from experiments using AI and education until very recently.

Now we are starting to see some early results, and they suggest that the potential for AI's for teaching and mentoring is quite high. A new large-scale, pre-registered, controlled experiment using GPT4 and tutoring found that practicing with the help of GPT4, especially when the AI was given a simple prompt to provide good explanations, significantly improved performance on SAT math problems. To be clear, GPT4 is not yet a universal tutor, and the experiment was more about guided practice problems with AI help than a deep educational experience. However, given that billions of people around the world have free GPT4 access through Bing,

and will be able to get a similar quality AI through Google whenever they release Gemini Ultra. This is a potentially very important finding. Moving beyond the classroom, we also have our first findings on the real world impact of AI mentoring. A fascinating new paper reports on a six-month long experiment using GPT4 to provide advice to small business entrepreneurs in Kenya. They found that getting AI mentoring boosted the performance of the best entrepreneurs by 20%, a very large effect for an educational intervention. Interestingly, low performers actually did worse after getting AI mentoring help, because they tended to ask questions that the AI was not good at answering or got advice they could not take. Many of the low performers had businesses that were already in

trouble. To me, this suggests that AI instruction can have real-world impacts, but we need to carefully design these tools to work for students of all ability levels. Of course, there are also massive risks associated with AI in classrooms. The AI can basically do everyone's homework. And remember, AI writing is undetectable. A lack of clear information means that there is confusion over how AI works and how to use it, and the biases and issues that AI brings to the classroom are still poorly understood. We are in the early days of exploring AI and education, but given its widespread accessibility and potential for improving educational outcomes, this seems like an area where it is imperative that we actively experiment to find positive

use cases. I expect to see a lot of progress this year. Section. Current tech is good enough for transformation, but more tech is coming. One clear theme of recent academic work on practical uses of LLMs is that there is a lot of potential left in GPT4. It is still best AI available, despite being a system that is already year old. Consider this paper showing a well-prompted GPT4 beats the best specialized medical AI and most doctors, or this paper finding that GPT4 can help automate novel scientific research, or this one suggesting that GPT4 can navigate webpages visually with the right tools. even if we never exceeded GPT4 level performance, very unlikely, we know that good prompting connecting

the LLM to other tools and other simple approaches greatly expand the AI's capabilities. I believe that we have five, ten years of just figuring out what GPT4 and the soon-to-be released Gemini Ultra can do, even if AI development stopped today. There are so many real-world tasks that are at least somewhat tractable by the current set of GPT4 class, LLMs with the right processes and tools. But of course, technology is not going to stand still. Open source models are coming out every week that exceed chat GPT 3.5 levels of performance and can run for free on a gaming computer.

New models that will beat GPT4 are in development and likely to be released this year. AI is going to be integrated into your Google and Microsoft applications, with uncertain implications. And the specter of possible AGI, that potential machine smarter than a human, haunts us. Among this broad acceleration, I think people who are worried about AI are often paying too much attention to scattered signals that AI might be hitting limits. The New York Times, for example, recently sued Open AI over how it trained on Times data, including how it can, under some circumstances, reproduce copyrighted articles, and fake other articles the Times did not write.

I am no legal expert and cannot speak to the merits of the case, but I think it is extremely unlikely that any legal finding will do much to put the AI genie back in the bottle. There are already companies like Adobe that only train AI on data they have unambiguous legal rights to, so there are paths forward for large companies even if the time suit is successful. Additionally, open-source models already released into the wild and being developed all over the world cannot be stopped. And different legal rules in different countries, Japan appears to view copyright as not applying to training data, suggests that AI, as a global phenomenon, will continue. Most likely, AI development is actually going to accelerate for a while yet before it eventually slows down

due to technical or economic or legal limits. While how far AI comes this year is not yet clear, I do know that this may be the critical time to assert our agency over AI's future. Managers, educators, and policymakers need to recognize that we are living in an AI-haunted world, and we need to both adjust to it and shape it in ways that increase its benefits and mitigate its harms. We need to start now, because we are facing exponential change, and that means that even the signs importance I have discussed in this post are quickly becoming prophecies of the past, rather than indicators of the future. All right, back to the non-robot NLW here.

for just a little bit of discussion and analysis. And the section that I want to hone in on is Ethan's section about AI already impacting work. Just by way of reminder, he had three conclusions. One, that AI boosts overall performance at complex work tasks. Two, that the effects are largest for lower performers. And that three, there is a jagged frontier. In other words, that AI is better at some tasks than others. Now, what I think is interesting about all this is that one of the things you see right now playing out

is a narrative battle around how significant AI is likely to be. There is a constant tension in the media, where because this technology is so obviously powerful, there's a temptation to run the other direction, and outlets are particularly excited about opinions and evidence that maybe it's actually less powerful than we think. There is a big hunt, given that last year was seen as the height of inflated expectations to look for that trough of disappointment.

And to some extent, I think that we are maybe going to get some of that this year. I think you're going to see lots of negative stories when Gemini Ultra isn't necessarily hugely farther beyond GPT4 or GPD4.5 or GPT5 or GPT5 doesn't seem as seismic as a leap as GPT4 was. But I think that if you only pay attention to the frontier, you're going to miss a lot of the most important story in AI, which will instead be a story, a quiet story, of people around the world, in offices in normal boring jobs, slowly integrating all sorts of different AI tools into their workflows, not because anyone told them to, not because of some wide-based

enterprise adoption from the top, but because, simply put, they are useful. Indeed, I think that AI enterprise adoption is going to be largely a bottoms-up phenomenon, where employees that happen to have tried these technologies slowly start to use them at work, and then they tell their colleagues and they show their colleagues how they work, and then all of a sudden, multiple people on multiple teams are using these tools, and then it gets kicked up to a boss somewhere, someone starts to expense, some new learning, and bingo-bang-bongo, all of a sudden,

without there ever being a mandate from on high, AI has been adopted. That is a much less captureable process, a much less sexy process, than the sort of big transformation than everyone has imagined. But I think it's going to more accurately reflect what actually happens. And I think at the end of it, the significance to how we work and what we work on will be even greater than the change that happened over the last year, even if we had no

technological advancements from here. It's going to be really interesting to watch, and I'll be sure to keep you posted with all the new evidence that says whether I'm right or whether I'm wrong. For now, though, that is going to do it for today's AI breakdown. Big thanks once again to Ethan for another insightful and thoughtful piece, and thanks to you guys for listening. Until next time, peace.