ACM ByteCast - Pattie Maes - Episode 38

Episode Date: June 1, 2023

In this episode of ACM ByteCast, Rashmi Mohan hosts Pattie Maes, a professor at MIT's Program in Media Arts and Sciences. Pattie runs MIT Media Lab's Fluid Interfaces research group, which does research at the intersection of Human Computer Interaction and Artificial Intelligence with a focus on applications in health, wellbeing, and learning. She is also a faculty member in MIT's center for Neuro-Biological Engineering. She has been a researcher, a serial entrepreneur and mentor, a book and journal editor, and a recipient of numerous awards, including recognitions from Newsweek, TIME, AAAI, Fast Company, the World Economic Forum, and Ars Electronica. In addition to her academic endeavors, Pattie co-founded several venture-backed companies, including Firefly Networks, Open Ratings, and Tulip. She is also an advisor to several early-stage companies, including Earable and Spatial. Pattie recounts her path to computing as one of the first people to major in computer science in Belgium and, later, as the only woman in the AI lab at MIT. She provides historical perspective on the cyclical nature of the field of AI and explains her passion for building systems that make people, rather than machines, more intelligent. She also recalls some of the designs and applied technologies she has worked on throughout her celebrated career, including recommender systems (before web browsers) and wearable devices (before cell phones). Finally, Pattie offers her thoughts on building diverse teams and what she’s most excited about in the field of AI.

This is ACM ByteCast, a podcast series from the Association for Computing Machinery, the world's largest education and scientific computing society. We talk to researchers, practitioners, and innovators who are at the intersection of computing research and practice. They share their experiences, the lessons they've learned, and their visions for the future of computing. I am your host, Rashmi Mohan. and the plethora of information we have online makes our in-person interactions a tad bit ineffective. Our next guest, however, has been on a quest to solve that problem for us.

Augmenting human intelligence and access to information in a way that seamlessly blends into our physical world has been Paddy Ma's career goal. Paddy is a professor at MIT's Program in Media Arts and Sciences and the head of the Media Labs Fluid Interfaces Research Group. She is a pioneer in the research area of human-computer interaction and artificial intelligence. Through her extensive and celebrated career, she has been a researcher, a serial entrepreneur and mentor, a book and journal editor, and a

recipient of numerous awards. Being named Newsweek's 100 People for the New Century, a member of the Cyber Elite by Time Digital, and the Global Leader for Tomorrow by the World Economic Forum, she's no stranger to the spotlight. Patti, welcome to ACM ByteCast. Thank you, Rashmi. I'd love to lead with our question that I ask all my guests, Patti, if you could please introduce yourself and talk about what you currently do and also give us some insight into what drew you into the field of computing. Yes.

So I studied computer science about 35 years ago, believe it or not. I was one of the first ones actually to major in computer science in Belgium. The degree only had been introduced a year earlier. I chose computer science initially because there was an economic crisis and I really wanted to make sure that I would have a job when I graduated. But along the way, I became really interested when I was introduced to the field of artificial intelligence, because basically it sort of connected this whole domain of computer science to people. And ultimately, people is what I'm really interested in. So I continued for a PhD at my university in artificial intelligence. And then after my PhD, moved to MIT first for some

internships, and eventually permanently, I got a job here, became a professor. Now, while initially, I was interested in sort of modeling human intelligence in machines, which is primarily what at that time AI was about, I actually personally started realizing that I was much more interested in helping people become more intelligent. So intelligence augmentation, basically, rather than making machines more intelligent. And that is really what has sort of motivated my whole career as a professor to build systems that help people with basically being their best self or with increasing their self-potential. So all of my work has been about helping people with finding information that may be relevant

to the problems that they are solving or the issue they are thinking about, helping them with making decisions, helping them with even other types of issues like attention, sustaining attention, having a better memory so they can function more productively. So yeah, that's been the journey that I've been on. And my recent work is still very much about using AI and using human-computer interaction and machines to help people become their best selves.

I love it. I love it, especially because of two things that you said, right? One is you took a very practical approach to picking computer science. We often hear of people who say that, you know, they were introduced to a computer when they were very little, or they played video games, and it really sort of drove their passion in it. But I know that there is a very large contingent of people who, like us, who got into it because it felt like a viable career, right? And then fell in love with the technology. So I love that story that you just brought up. And the second thing that I also enjoyed about what you just said was, you know, there's so much, especially for somebody who's

not deeply embedded in the artificial intelligence world, there's a lot of fear around what AI is going to do and how it's going to take over our lives and our jobs and we'll have nothing to do. But the way you describe it is it's making our lives better. It's augmenting our existing capabilities and making us better versions of ourselves, which puts a spin, which is a lot more comforting, a lot more exciting and something to look forward to. Yeah, I think when I started studying computer science, it was very much a field that only attracted people that were interested in the technology and the algorithms and so on for the sake of the technology itself. And I'm glad to see that the fields of computer science and AI has broadened to include people who really not just want to make smarter, better algorithms and machines, but people who think about how these amazingly powerful technologies can improve people's lives and can improve our society. You see in many universities now more interdisciplinary programs that include or sort of combine computer science and AI with other

disciplines. And I really hope that that trend will continue because ultimately computers and AI are already to a large extent defining the lives that we live and running our society. They're like the operating system of a lot of how our world runs. And it is important that we don't just have engineers thinking about what kind of computer run world we want to live in. Yeah, no, excellent, excellent points. You know, I definitely want to dig into a lot of those a little bit later. But I'll go back to one thing that I wanted to ask you, Patti, which is, you were working on artificial intelligence, you know, way before it was trending as it is today. So what was that trigger that you said, hey, this is the area that I want to go into? And did you anticipate that it would

explode in the applications and need as it has now? Well, personally, for me, what got me very excited about AI is what I mentioned earlier, that it really connected what I was learning about programming languages and architectures for machines and all of that. It was connecting that with people because AI, at least back then, was very much about, can we model amazing things that people can do with machines? So that is what attracted me to the field at that time. Of course, at that time, it wasn't as big a field as it is right now. Although I have over my entire career definitely seen a couple of sort of AI summers, they call them, as well as AI winters. So an AI summer means suddenly there's a lot of interest from outside the AI research community in the field and a lot of money gets poured into it, etc. And then an AI

winter, of course, is when suddenly everybody's disappointed. And it's again, just the researchers at the conferences and so on. So yeah, I've seen multiple cycles like that in AI, which is interesting because you start seeing some parallels, actually. And specifically, I think the parallel that I see is that often when there's some promise for how AI can help with real world problems, we again leave it too much to the engineers to develop systems and think about how AI can be used in the real world. And I think the problem has been over and over again that not enough people with different backgrounds, for example, psychology or management science and so on. Not enough people with different backgrounds have helped develop these AI based solutions. pure engineers, and then they get dropped into, say, a doctor's office, for example, who then has to use the AI system to make better diagnoses or something. And that approach doesn't

work. Over and over, the people who then have to use the AI systems have not trusted the systems because they weren't involved in developing them. They don't know what is under the hood or what is in the black box. They don't understand the systems, don't understand the limitations or why a system may come up with a certain recommendation and more. And I fear that we are, again, actually, making that same mistake a little bit and not thinking enough about the human elements really of AI and how it can fit in our society, in our workflow, and so on. Yeah, I love the summer-winter analogy. I mean, we're definitely in the scorching summer

right now, I think, with so much interest and scrutiny on how to build these AI systems. And you're clearly saying that, you know, there's not enough interaction from people who are in the field who will be using this technology in a way to maybe administer healthcare or other services to the world. Are there forums that are bringing, you know, these interdisciplinary experts together to actually talk about these issues, to define some standards and be able to make sure that we're basically being guided in the right direction? Or do you think there's room for that? are some people out there who have learned this lesson also, or who have this same sort of insight. For example, people like Fernanda Villegas, who works at Google as well as is a professor at Harvard, has come to the same realization, really, that we have to think more about the people who we are building these AI systems for. And we have to include them, not just at the end, but really in the whole development of these systems, so that they can be built in a way that benefits them, and they can trust these systems and will actually

make usage of them. Yeah, no, I mean, I think that's a very, very critical point that you bring up. But I'm going to go back a little bit to sort of talk about your journey as to how you got here and talk about a previous summer, maybe. You're credited with the forming of one of the very early collaborative filtering experiences, personalized recommendation systems, or even the first social network. So I would love for you to talk to us a little bit more about that. Yeah, so this work actually started before there were browsers. There was an internet, but we didn't have a worldwide web yet. We weren't just interested in helping people find media or information that could be relevant to them.

I mean, it's a very common problem that all of us struggle with all the time. You want to find a movie or a TV show that you may like or a book or maybe some web pages about a particular topic that may be useful for you and so on. And we realized that maybe one way of helping people with that was to let them benefit from what other people know, other people like them. So we started creating these recommendation systems that basically recommend, we worked on many different kinds of media, movies, books, music,

web pages, all of those systems that recommend these pieces of information or media based on other people who have similar tastes, who have similar interests, basically. We did this work initially over email because again, it was pre-browsers in 92 and 93. And so there was an email address where you could send your science fiction, favorite science fiction books, and then you would get back an email for other science fiction authors and books that you may like based on the ones that you enter them based on what other people that liked those books, what they liked. So that actually, the following year, when browsers became more common and sort of the whole World Wide Web started emerging, we started creating the first systems like that online. And we at one point, believe it or not, with our system had one of the 20 most visited websites ever. And it quickly grew in usage. And

this was before Facebook, before any of these other things existed. And people were just very excited. It was kind of a mix of a system that gave you recommendations and a social network, because once you sort of said what your tastes were in some area, you could then look up the other people that were like you, or that liked some of these same things, if they chose to not be anonymous, you could contact them. And several marriages even came out of our research experiment because people found some other people that they really resonated with. So that was a lot of fun. Yeah. We initially tried to sell this idea to companies like at that time, there was Blockbuster and Barnes and Noble and so on. Many of them, not Barnes and Noble, they actually ended up using our

technology. But many of these companies, we were way too early, basically, and they didn't understand what we were trying to do. They were still trying to figure out what this whole World Wide Web was and whether they should pay attention to it and whether people would ever want to buy things online and so on. So we were too early. We were telling them, hey, you need this system that will give them recommendations so they're more likely to find the things that they're interested in. So we created a company because we thought, well, somebody has to do this and all these existing companies aren't ready for it. And that company

was called Firefly. And we had an interesting ride sort of with Firefly as entrepreneurs. It was me and my students and then a couple of people that we brought in from Harvard Business School to give us a little bit more business knowledge. Yeah, we just had a very interesting adventure with this Firefly website and recommendation engine and ultimately sold it to Microsoft at one point because there was at some point a downturn, again, sort of a winter. But then in e-commerce where everybody was convinced that people would never want to buy things online, it was too dangerous to enter your credit card. And at that time, basically, it was very hard to raise additional funds for our company because nobody believed that e-commerce would ever take off. And so we sold it to Microsoft and moved on, basically went back to research.

That is a phenomenal story. I mean, just the idea of online commerce not taking off seems so bizarre if you think about it in today's context. Not that long ago. That's in 1998. It's 25 years ago. 25 years. I know. It's incredible, the story you tell. Also, to think about businesses that are not able to see just around the curve to see what kind of potential some of these ideas have, right? I'm sure that there was at least some amount of data you could provide around adoption, etc. But just not being able to visualize where the new business is going to come from is kind of crazy to think of. Yeah, and we saw that all the time. There were other companies at that time,

like Kodak, didn't believe that digital cameras were going to take off and totally replace, pretty much totally replace film. Well, Blockbuster that I mentioned, I don't think they even exist anymore. They went bankrupt because they weren't listening to us. But the pivot to entrepreneurship, Patti, I mean, that's a bold move, especially if you're coming from a research world. How did you navigate that? I mean, the challenges of running a business versus building the technology. Yeah, so I never really intended to be an entrepreneur. It was more that we believed so much in this idea.

And we saw this website that we had created grow so rapidly. And again, we had no takers. We went to Apple, by the way, as well. I mean, many companies that we approached, they just were not interested in what we were doing in sort of this whole social network plus e-commerce type of site that was highly personalized. So we just said, well, we have to keep doing this because people love it. And so we started the company. So it was really kind of by accident that I got into this, but it's been really interesting. Of course, the whole entrepreneurial world is different from the research world because, well, you have to learn to think differently. You're not just creating some amazing technology and proving how, whatever,

wonderful and better than previous stuff it is, but you really have to think about what problem out there are you solving and for whom, who's your market and how can you best service them. And so the technology isn't always central in the entrepreneurship world, but in general, I'm happy when I learn things, when I learn new things. So for me, it was a very exciting time. And we made some money as well. So that was good. But it was just very exciting to have this totally new way of looking at things and to try to do something really real for people out there. Right. Just that validation from having your product be used by a large group of people. And like you said, your motivation for getting into this field was to help people. So anything that will, you know, aid you in reaching to, you know, as many people as possible, I'm sure is an extremely satisfying experience. And clearly you were built for entrepreneurship as much as you're built for research because you've been a serial entrepreneur. I think you've

had multiple startups since. Yeah. But yeah, we'll get into that in a second. ACM Bytecast is available on Apple Podcasts, Google Podcasts, Podbean, Spotify, Stitcher, and TuneIn. If you're enjoying this episode, please subscribe and leave us a review on your favorite platform. I want to kind of go back to the work that you do with the Fluid Interfaces Group at the intersection of AI and HCI. How did that come about?

Yeah, so initially when I sort of had a research group here at the Media Lab, I was primarily focused, like I said, on helping people find information, helping people retrieve information also from their own previous interactions and so on. But gradually, I realized, and this relates to your introduction of the interview here, that this whole online world was so disconnected from our physical lives. And I really wanted to make these types of systems that help you make decisions, that give you information, et cetera, at the right moment about the topic you're currently thinking about. I wanted to make that available in our physical lives. So I then started to do a lot of work on wearable devices. And this was before the smartphone and before the cell phone existed. So we had to make our own hardware, really carry a laptop in a backpack that would compute and where a person was based on beacons, et cetera,

and information you entered. And the systems we built would give you just-in-time information based on what you were talking about, who you were with, basically the topic, the time of the day, the location, etc. So we called that particular system the Remembrance Agent. And it was a system that was constantly running in the background, trying to find information relevant to what a person currently was doing and who they were talking to. That sort of became an area of research that then multiple people worked on. And that was very exciting. And that still isn't really available in our commercial devices and services. We build systems like the Sixth Sense device, which I gave a TED Talk around, that in that system, it was an augmented reality

interface that would constantly project information as well as interfaces onto the objects and even the people around us so that you could, like you said, if you picked up a book, it would tell you, well, this book only gets three stars at Amazon, or you could even leave a message in a book for someone else so that if they picked up that book, suddenly the message appeared, et cetera. And unfortunately, that still isn't as easy as it should be. Although companies like Apple are rumored to come out with augmented reality glasses soon. So who knows, maybe in my lifetime, we will still see this. Yeah, I saw that TED Talk of yours, the Sixth Sense one, and I was fascinated. And I was like, why do I not have this available to me today? So what do you think are

the barriers, Patti, that have not made this commercially viable? Yeah, it mostly has been the hardware, I would say, and batteries and so on. Like in the Sixth Sense device, we actually used projected augmented reality. And the nice thing was that that was totally hands-free and so on, but it really only works indoors. You can't really do it in the daylight, and battery power is an issue and so on. Augmented reality headsets just haven't been very good yet. And they've had other issues like with Google Glass, which isn't really augmented reality, but it's related. There was a big issue,

of course, around privacy and people being able to just record conversations without others being aware that they were recording and so on. But yeah, it's been, I think, the hardware, the form factors that have hampered this. Of course, we have handheld augmented reality, meaning that you can point your phone at something and see, for example, the translation of some text in some language you don't understand or you can't read. But it requires that you get out your phone, translation of some text in some language you don't understand or you can't read. But that is,

it requires that you get out your phone, you start the app, etc. And it's too cumbersome. People want all of these types of things to be more seamless and to take less effort. Yeah, no, no, absolutely. And I mean, I think that also gives me, I mean, while it is true that it's not available today, it gives me hope that, you know, we're probably, you know, a few iterations away before we find the solution that would work more effectively than we have. At least we've had a few tries. But it's definitely something that I can see being extremely valuable from just because of the way, like you said, that our lives are just so intertwined with our digital personas that at some ways you're going to have to find those two worlds blend. Yeah, I do think one of the big problems that we really have to solve as researchers is that the technologies we have invented, they're wonderful, but they have really made our attention very fragmented. And this is one example of that. We have our physical presence and we're surrounded with people, et cetera.

But then what we do online or on our phone is usually completely not related to whatever we're doing physically and the people around us physically and so on. So we're constantly switching between these two worlds, the people around us and the physical space around us and objects, and then our phones or laptops or whatever device we use. And even on a phone and on a laptop, our attention is so fragmented. If you take something like email, it's still the main way that we communicate, I think, online. And it's so awful because we are constantly switching

from one issue to the next. I read one message about my kids' curriculum night, and then the next message is about a conference I'm co-organizing, and then the next message is about a paper that I'm working on. And we're constantly switching context like that. And it makes us ultimately, our attention is so fragmented and we're not as productive because we are switching context. And then, of course, with every time you switch context, you risk that you then sort of fall into a black hole of you get one advertisement that looks intriguing.

You click on it and then you think, oh, yeah, I do need a new pair of shoes and you start looking for shoes or something. So I think that there's so much potential still to completely change the way we interact with our devices so that ultimately our performance, our attention, and really our wellbeing also is optimized a lot more than it is today. Yeah, no, I mean, you know, guilty as charged. I mean, I know there's this phenomenon of second screen, and I am so guilty of that, because we'll have like a movie playing, but I will always have my phone with me. And so I'm constantly switching between two digital interactions. Plus, there's people around you that you know, you're also talking to. And so, yeah, it's a wonder that anything gets done at all.

But I'd love to go back to the point that you brought up about just our lack of attention. I know that, you know, some of the work that you've done is around looking at wearable medical technology. And I know one specific area, I think it was either reading something that, you know, an interview that you had done that spoke about your work to see how do we use this AI and medical technology to help with helping build concentration and inducing behavior change. So you, the way I interpreted it was that, you know, we could have interventions that could actually change the behavior of somebody from being,

you know, this sort of distracted being to sort of helping them concentrate more or helping them get out of a state of maybe unhappiness or heading towards depression. I know that there is work around, you know, feedback, but yeah, that area just seems so fascinating because that's right on in terms of like, you know, how can I help somebody in the most stupendous way? Yeah, in the last eight or so years, my work has completely really, or almost completely shifted in that area. I started realizing that computers and the devices that we rely on so much and have with us every day that we shouldn't just think of them as information devices or communication devices. They can do so much more. They can really help us with some of the more cognitive skills that we may struggle with. Given that we have a device with us,

practically 24 by 7, a lot of us, there's this opportunity for these devices to help us, for example, with developing the ability to sustain attention or help us with memory. We are doing some fun things in that area right now where we're trying to help people remember things by helping them create a little mnemonic so that they can remember the name of somebody they just met, etc. We're helping people with behavior change, another huge area that lots of us struggle with. And I do think that the devices that we carry with us have the opportunity to help us with all of these types of issues.

In these systems, we typically have more sensors, sensors that can sense the physiology of a person or the behavior or sometimes even brain wave activity, eye gaze, et cetera. And you can, as a user, say what it is that you need help with what skill you want to develop and the system will try to help you with that by giving you interventions in the moment that really hopefully help you develop that particular skill that is weak, whether that is, say, adopting a more healthy behavior, or again, being more attentive, or being able to learn and remember more easily, being more motivated. When you're in a classroom, for example, these are all things that all of us struggle with to some degree, and the devices that we carry, I think, have the potential to help us with those issues.

That would just be so path-breaking, Patti. Just the vision that you describe fostered through your career is, how are you building these systems in this integrated way that we spoke of earlier, which is really because we're making these decisions on how these interventions should impact people that use it. But you obviously need a lot of input from the medical professionals or psychologists. So I'm wondering, how do you foster that really diverse team that's giving you the input and building this together? Yeah, so I do believe that the best research really results from inviting a lot of collaboration and feedback from people who are very different. So my team here, for example, at the Media Lab has

neuroscientists, psychologists, designers, electrical engineers, AI people, all in one group trying to work together. And they can all bring their perspective, their skills, their knowledge to collaborate on a particular project. But going beyond that, I think it's important also to talk to non-researchers. And we work constantly with target users, for example. Like one area that we're looking at is memory augmentation for people that have early memory decline. And it's really important that we talk to and work with people that have that problem in developing these systems so that we're more likely to really develop what it is they need

and want and would actually use. So we work with target users. We work with experts, psychologists, psychiatrists, and so on. And we do a lot of iterative user prototyping and user testing. Yeah, no, that sounds excellent. And sounds the absolute right way to sort of go about this. But to go back to your fact that you're a serial entrepreneur, I'm wondering, do you find that, you know, I don't know what your target is or your goal is. Is this something that you think you want to build out as a company?

Or do you feel like there are larger partners there who will have the right, the funding, the visibility to take this much further than you possibly could? Yeah, mostly I want all of the things that we work on to ultimately be available for people and help people. And to be honest, I don't care so much about how that happens. So one way is through startups. And every couple of years, I have some students who start a company based on the research they do here. But we also work with industry, with large companies that do have money to invest in research. They are interested in research, but they often don't really embrace a new technology. And often that is the case because it doesn't fit into one of their existing markets and products or services, often the things we build are kind of new and they may even compete or with something that this company say is already

selling. So in my experience, starting companies has been a better way to make sure that these techniques make it out there and that the work that we do really ultimately has a real world impact. Yeah, no, I completely get that. I mean, I think there's probably a reason why startups tend to be sort of that hotbed of innovation because there's risk involved in sort of starting up, but at least in terms of, you know, maybe breaking through into a new market, you're not sort of held to the standards of profitability or other metrics that you're using to measure larger companies,

which are probably more risk averse. So I'm really, really happy to hear that, you know, you're embracing that and sort of taking these amazing solutions out to the world because, you know, I mean, who knows how many Barnes and Nobles are out there that are not seeing where this could go. Yeah, that's great. So Patty, one thing I do want to cover in our conversation is you started working in AI many, many years ago before it was even a buzzword. And I've heard that you were

often or maybe the only woman when you first started in this area at the AI lab in MIT. Help me through that journey. How was that experience? What do you think helped you in being sort of more successful and confident? What would be your advice to young women out there who are just sort of embarking on their career journeys? Yes. So when I came to MIT, indeed, I think there was actually one professor, one woman at the AI lab of MIT, which was the mecca for AI research, of course, in the world at that time. And she left soon after she had a baby. So then I came in initially as a visiting postdoc. And then eventually they

invited me to be a visiting professor, actually. And I was the only woman, the only woman at the AI lab at that time that was basically a professor or visiting professor. Things have changed a lot since then, but I think we can still do better. And I think it relates to this point that I made earlier, that I think often women can bring a different perspective. And I'm, of course, generalizing here where I shouldn't. Different people are all different. But on average, women are more interested in, I think, having a positive impact on people, on the environment, etc. And that can motivate them more than, say, the pure algorithms, technology, science, etc. And I do think we need more women in the fields because they will make the fields better. I talked earlier about how things like the problems with Facebook, et cetera, that we had around the election and so on, maybe things like that wouldn't have happened if there were more people involved in creating these services other than engineers, young and often male engineers.

And I think this is actually, in my experience, it has been in some ways easy to be a woman because my perspective was different. And so I would do things that maybe a lot of the men were not interested in, or I would be interested in topics that they wouldn't work on. So I do think that ultimately, women are in a great position to do very well. And we need more women, of course, to then be role models and mentors for the next generation and so on. But the fact that women, I think, have more broad motivation for their work can really be a benefit. That's great advice. And you're definitely an inspiration. And so I hope we find a lot more women who choose to get into these fields that are up and coming and so many wonderful problems to be solved. Patti, for our final bite here,

I would love to hear from you. What are you most excited about, you know, in the field of HCI and AI over the next five years? Well, that's always a hard question. I am very excited right now. And I mean, not just right now, for the last five years or so, we've been working in the area of generative AI. And I know that it is really scary what it might, sort of the big earthquakes that it may organize sort of in a society. But at the same time, I think there's a lot of positive use cases that can be developed with these technologies.

They can really be used to improve learning, well-being, et cetera. And I think that's a really interesting area to explore, but it has to be explored very carefully, of course, because of the sort of dangers in the technology not being totally accurate and be biased and so on. That's amazing. This has been such an engaging and incredible conversation. Patti, thank you so much for taking the time to speak with us at ACM ByteCast. Thank you, Rashmi. It was a pleasure.

ACM ByteCast is a production of the Association for Computing Machinery's Practitioners Board. To learn more about ACM and its activities, visit our website at learning.acm.org slash b-y-t-e-c-a-s-t. That's learning.acm.org slash ByteCast.