Microsoft Research Podcast - Ideas: Building AI for population-scale systems with Akshay Nambi

Episode Date: February 11, 2025

In this episode, guest host Chris Stetkiewicz talks with Microsoft Principal Researcher Akshay Nambi about his focus on developing AI-driven technology that addresses real-world challenges at scale. Drawing on firsthand experiences, Nambi combines his expertise in electronics and computer science to create systems that enhance road safety, agriculture, and energy infrastructure. He’s currently working on AI-powered tools to improve education, including a digital assistant that can help teachers work more efficiently and create effective lesson plans and solutions to help improve the accuracy of models underpinning AI tutors.

For me research is just not about pushing the boundaries of the knowledge. It's about ensuring that these advancements translate to meaningful impact on the ground. So yes, the big goals that guide most of my work is twofold. One, how do we build technology that's scaled to benefit large populations? And two, at the same time I'm motivated by the challenge of tackling complex problems. That provides opportunity to explore, learn, and also create something new. And that's what keeps me excited. You're listening to Ideas, a Microsoft research podcast that dives deep into the world of technology research and the profound questions behind the code.

behind the code. In this series, we'll explore the technologies that are shaping our future and the big ideas that propel them forward. I'm your guest host, Chris Stetkiewicz. Today, I'm talking to Akshay Nambi. Akshay is a principal researcher at Microsoft Research. His work lies at the intersection of systems, AI, and machine learning, with a focus on designing, deploying, and scaling AI systems to solve compelling real-world problems. Akshay's research extends across education, agriculture, transportation, and energy.

He is currently working on enhancing the quality and reliability of AI systems by addressing critical challenges such as reasoning, grounding, and managing complex queries. Akshay, welcome to the podcast. Thanks for having me. I'd like to begin by asking you to tell us your origin story. How did you get started on your path? Was there a big idea or experience that captured your imagination or motivated you to do what you're doing today? If I look back, my journey into research wasn't a straight line.

It was more about discovering my passion through some unexpected opportunities and also finding purpose along the way. So before I started with my undergrad studies, I was very interested in electronics and systems. My passion for electronics kind of started when I was in school. I was more like an average student, not a nerd or not too curious, but I was always tinkering around doing things, building stuff, and playing with gadgets. That kind of made me very keen on electronics and putting things together, and that was my passion.

But sometimes things don't go as planned, So I didn't get into the college which I had hoped to join for electronics, so I ended up pursuing computer science, which wasn't too bad either. So during my final year of bachelor's, I had to do a final semester project, which turned out to be a very pivotal moment. And that's when I got to know this institute called

Indian Institute of Science, which is a top research institute in India and also globally. And I had a chance to work on a project there, and it was my first real exposure to open-ended research. Right? So I remember where we were trying to build a solution that helped to efficiently construct an ontology for a specific domain, which simply means that we are building systems to help users uncover relationships in the data and allow them to query it more efficiently, right? And it was

super exciting for me to design and build something new, and that experience made me realize that I wanted to pursue research further. And right after that project, I decided to explore research opportunities, which led me to join Indian Institute of Science again as a research assistant. So what made you want to take the skills you were developing and apply them to a research career? So interestingly, when I joined IISc, the professor I worked with specialized in electronics, so things come back, so something I had always been passionate about.

And I was the only computer science graduate in the lab at that time With others being electronic engineers and I didn't even know how to solve it But the lab environment was super encouraging Collaborative so I kind of caught up very quickly in that lab Basically, I worked on several projects in the emerging fields of embedded device and energy harvesting systems of embedded device and energy harvesting systems. Specifically, we were designing systems that could harvest energy from source like sun, hydro, and even RF signals.

And my role was kind of twofold. One, I designed circuits and systems to make energy harvesting more efficient so that you can store this energy. And then I also wrote programs software to ensure that the harvested energy can be used efficiently. For instance, as we harvest some of this energy,

you want to have your programs run very quickly, so that you are able to send the data, send it to the server in an efficient way. And one of the most exciting projects I worked during that time was on data-driven agriculture. So this was back in 2008, 2009, right? Where we developed an embedded system device with sensors to monitor the agricultural field, collecting data like soil moisture, soil temperature, and that was sent to the agronomists who were able to analyze this data and provide feedback to farmers.

In many remote areas, still access to power is a huge challenge. So we used many of the technologies we were developing in the lab, specifically energy harvesting techniques, to power these sensors and devices in the rural farms. And that's when I really got to see first-hand how technology could help people lives, particularly in rural settings. And that's what kind of stood out in my experience at IIC, right, was that it was end-to-end nature of the work and it was not just writing code or designing circuits. It was about identifying the real world problems, solving them efficiently, and deploying solutions in the field. And this cemented my passion for creating technology that solves real world problems, and that's what keeps me driving even today. And as you're thinking about those problems that you want to try and solve,

where did you look for inspiration? It sounds like some of these are happening right there in your home. That's right. Growing up and living in India, I've been surrounded by these kind of many challenges. And these are not distant problems. These are right in front of us. And some of them are quite literally outside the door. So being here in India provides a unique opportunity to tackle some of the pressing real world challenges

in agriculture, education, or in road safety, where even small advancements can create significant impact. So how would you describe your research philosophy? Do you have some big goals that guide you? Right, as I mentioned, my research philosophy is mainly rooted in solving real-world problems through end-to-end innovation. For me, research is just not about pushing the boundaries of the knowledge. It's about ensuring that these advancements translate to meaningful impact on the ground. So yes, the big goals that guide most of my work is twofold.

One, how do we build technology that's scaled to benefit large populations? And two, at the same time, I'm motivated by the challenge of tackling complex problems. That provides opportunity to explore, learn, and also create something new. And that's what keeps me excited. So let's talk a little bit about your journey in Microsoft Research. I know you began as an intern and some of the initial work you did was focused on computer vision, road safety, energy efficiency.

Tell us about some of those projects. As I was nearing the completion of my PhD, I was eager to look for opportunities in industrial labs and Microsoft Research obviously stood out as an exciting opportunity. And additionally, the fact that Microsoft Research India was in my hometown Bangalore, made it even more appealing. So when I joined as an intern,

I worked together with Venkat Padmanabhan, who now leads the lab, and we started this project called HAMS, which stands for Harnessing Automobiles for Safety. As you know, road safety is a major public health issue, globally responsible for almost 1.35 million fatalities annually, and with the situation being even more severe in countries like India. For instance, there are estimates that there is a life lost on the road every four minutes

in India. When analyzing the factors which affects road safety, we saw mainly three elements. One, the vehicle. Second, the infrastructure and then the driver. Among these, the driver plays a most critical role in many incidents, whether it's over speeding, driving without seat belts, drowsiness, fatigue, any of these, right? and this realization motivated us to focus on driver monitoring which led to the development of HAMS. In a nutshell, HAMS is basically a smartphone based system where you're mounting your smartphone on a windshield of a vehicle to monitor both the driver and their driving in real time

with the goal of improving road safety. Basically it observes key aspects such as where the driver is looking, whether they are distracted or fatigued, while also considering the external driving environment. Because we truly believe to improve road safety, we need to understand not just the driver's action but also the context in which they are driving. For example, if the smartphone's accelerometer detects sharp braking, the system would automatically check the distance to the vehicle in the front using the rear camera and whether the driver was distracted or fatigued using the front camera. And this holistic approach ensures a more accurate and comprehensive assessment of the driving behavior, enabling a more meaningful feedback.

So that sounds like a system that's got several moving parts to it. And imagine you had some technical challenges you had to deal with there. Can you talk about that? One of our guiding principles in HAMS was to use a commodity off-the-shelf smartphone devices, right? This should be affordable in the range of $100 to $200 so that you can just take out regular smartphones and enable this driver and driving monitoring and that led to several technical challenges. For instance, we had to develop efficient computer vision algorithms that could run

locally on the device with cheap smartphone processing units while still performing very well at low light conditions. We wrote multiple papers and developed many of the novel algorithms which we implemented on very low cost smartphones and once we had such a monitoring system right you can imagine that the several deployment opportunities starting from fleet monitoring to even training new drivers right. However one application we hadn't originally envisioned but turned out to be its most impactful use case even today is automated driver's license testing. As you know, before you get a license, a driver is supposed to pass a test. But what happens in many places, including India, is

that licenses are issued with very minimal or no actual testing, leading to unsafe and untrained drivers on the road. At the same time, as we were working on HAMs, Indian government were looking at introducing technology to make testing more transparent and also automated. So we worked with the right set of partners and we've demonstrated to the government that HAMs could actually completely automate the entire license testing process.

So we first deployed the system in Derradun RTO, which is the equivalent of a DMV in the US in 2019, working very closely with RTO officials to define what should be some of the evaluation criteria, right? Some of these would be very simple like, oh, is it the same candidate who is taking the test who actually registered for the test, right? And whether they are wearing seat belts, did they scan their mirrors before taking a left turn and how well they performed in tasks like reverse parking and things like that. So what's been the government response to that? Have they embraced it or deployed it

to a wider extent? Yes, yes. So after the deployment in Dehradun in 2019, we actually open sourced the entire hams technology and our partners are now working with several state governments and scale HAMS to several states in India. And as of today, we have around 28 RTOs where HAMS is actually being deployed. And the pass rate of such license tests is just 60% as compared to 90 plus percent with manual testing. That's the extensive rigor the system brings in. And now what excites me is after nearly five years later, we are now taking the next step in this project where we are now evaluating the long-term impact of

this intervention on driving behavior and road safety. So we are collaborating with Professor Michael Trammer, who is a Nobel Laureate and Professor at University of Chicago and his team, to study how this technology has influenced driving patterns and accident rates over time. So this focus on closing the loop and moving beyond just deployment in the field to actually measuring the real impact, right, is something that truly excites me and that makes research at Microsoft is very unique.

And that is actually one of the reasons why I joined Microsoft Research as a full time after my internship. And this unique flexibility to work on real-world problems, develop novel research ideas, and actually collaborate with partners both internally and externally to deploy at scale is something that is very unique here. So have you actually received any evidence that the project is working or is driving getting safer? Yes, these are very early analysis and they are very positive insights we are getting from that. Soon we'll be releasing a white paper on our study on this long-term

impact. That's great, I look forward to that one. So you've also done some interesting work involving the Internet of Things, which with an emphasis on making it more reliable and practical. So for those in our audience who may not know the Internet of Things or IoT is a network that includes billions of devices and sensors and things like smart thermostats and fitness trackers. So talk a little bit about your work in this area. Right, so IoT as you know is already transforming several industries with billions of sensors being deployed in areas like industrial monitoring, manufacturing, agriculture, smart buildings, and also air pollution monitoring. And if you think about it,

these sensors provide critical data that businesses rely for decision making. However, a fundamental challenge is ensuring that the data collected from the sensors is actually reliable. If the data is faulty, it can lead to poor decisions and inefficiencies. And the challenge is that the sensor failures are always not obvious. What I mean by that is when a sensor stops working, it always doesn't stop sending data but it

often continues to send some data which appear to be normal. And that's one of the biggest problems, right? So detecting these errors is non-trivial because the faulty sensors can mimic real-world working data. And traditional solutions like deploying redundant sensors or even manually inspecting them are very expensive, labor-intensive, and also sometimes infeasible, especially for remote deployments. Our goal in this work was to develop a simple and efficient way to remotely monitor the health of the IoT sensors. So what we did was we hypothesized that most sensor failures occur due to the electronic malfunctions. It could be either due to short circuits or component degradation or due to environmental

factors such as heat, humidity or pollution. Since these failures originate within the sensor hardware itself, we saw an opportunity to leverage some of the basic electronic principles to create a novel solution. The core idea was to develop a way to automatically generate a fingerprint for each sensor and by fingerprint I mean the unique electrical characteristic exhibited by a properly working sensor. We built a sense system that could divide these fingerprints for different types of sensors allowing us to detect failures purely based on the sensors internal

characteristics that is the fingerprint and even without looking at the data it produces. Essentially what it means now is that we were able to tag each sensor data with a reliability score, ensuring verifiability. So how does that technology get deployed in the real world? Is there an application where it's being put to work today? Yes. This technology, we worked together with Azure IoT and open-sourced it, where there were several opportunities and several companies took the solution into their systems,

including air pollution monitoring, smart buildings, industrial monitoring. The one which I would like to talk about that today is about air pollution monitoring. As you know, air pollution is a major challenge in many parts of the world, especially in India. Traditionally, air quality monitoring relies on this expensive fixed sensors, which

provide limited coverage. On the other hand, there is a rich body of work on low-cost sensors, which can offer wider deployment. You can put these sensors on a bus or a vehicle and have it move around the entire city, where you can get much more fine-grained, accurate picture on the ground.

But these are often unreliableable because these are low-cost sensors and as reliability issues. So we collaborated with several startups who are developing this low-cost air pollution sensors who are finding it very challenging to gain trust because one of the main concerns was the accuracy of the data from low-cost sensors. So our solution seamlessly integrated with the sensors

which enabled verification of the data quality coming out from this low-cost air pollution sensors. So this bridge the trust gap, allowing government agencies to initiate large-scale pilots using low-cost sensors for fine-grained air quality monitoring. So as we're talking about evolving technology, large language models or LLMs are also enabling big changes,

and they're not theoretical they're happening today then you've been working on LLMs and their applicability to real-world problems can you talk about your work there and some of the latest releases so when chargeability was first released I like many people was very skeptical however I was also curious both of how it worked and more importantly whether it could accelerate solutions to real-world problems. That led to the exploration of LLMs in education where we fundamentally asked this question, can AI help improve educational outcomes?

And this was one of the key questions which led to the development of SIGSHA Copilot, which is a GenAI-powered assistant designed to support teachers in their daily work starting from helping them to create personalized learning experience, design assignments, generate hands-on activities and even more. Teachers today universally face several challenges from time management to lesson planning and our goal with Sikshakopilot was to empower them to significantly reduce the time spent on this task. For instance, lesson planning which traditionally took about 60 minutes can now be completed in just 5 minutes using the Sikshak Copilot.

And what makes Sikshak unique is that it's completely grounded in the local curriculum and the learning objectives, ensuring that the AI generated content aligns very well with the pedagogical best practices. The system actually supports multilingual interactions, multimodal capabilities and also integration with external knowledge base making it very highly adaptable for different curriculums. Initially many teachers were skeptical some feared this would limit their creativity. However as they began starting to using CICSA they realized that it didn't replace their expertise

but rather amplified it enabling them to do work faster and more efficiently. So Akshay, the last time you and I talked about Sikshya Copilot, it was very much in the pilot phase and the teachers were just getting their hands on it. So it sounds like though you've gotten some pretty good feedback from them since then. Yes. So when we were discussing, we were doing this six month pilot with 50 plus teachers where we gathered overwhelming positive feedback on how technology is helping teachers to reduce time in their lesson planning.

And in fact, they were using the system so much that they really enjoyed working with Sikshak Copilot where they were able to do more things with much less time. Right? And with a lot of feedback from teachers, we have improved Sikshat Copilot over the past few months. And starting this academic year, we have already deployed Sikshat to 1000 plus teachers in Karnataka. This is with close collaboration with our partners in Sikshat Foundation and also with the government of Karnataka. And the response has been already incredibly encouraging. And looking ahead, we are actually focusing on again, closing this loop, right, and measuring the impact on the looking ahead, we are actually focusing on, again,

closing this loop and measuring the impact on the ground, where we are doing a lot of studies with the teachers to understand not just improving efficiency of the teachers, but also measuring how AI-generated content enriched by teachers is actually enhancing student learning objectives. So that's a study we are conducting, which hopefully will close this loop

and understand our original question that can AI actually help improve educational outcomes? And in the deployment primarily in rural areas or does it include urban centers or what's the target? So the current deployment with thousand teachers is a combination of both rural and urban public schools. These are covering both English medium and Canada medium teaching schools with grades from Class 5 to Class 10. Great. So Shiksha was focused on helping teachers and making their jobs easier. But I understand you're also working on some opportunities

to use AI to help students succeed. Can you talk about that? So, as you know, LLMS is still evolving and inherently they are fragile and deploying them in real-world settings especially in education presents a lot of challenges. With SIGSHA, if you think about it, teachers remain in control throughout the interaction making the final decision on whether to use the AI generated content in the classroom or not. However, when it comes to AI tutors for students, the stakes are slightly higher where we need to ensure the AI doesn't produce incorrect

answers, misrepresent concepts, or even mislead explanations. Currently, we are developing solutions to enhance accuracy and also the reasoning capabilities of these foundational models, particularly solving math problems. This represents a major step towards building AI systems that's much more holistic personal tutors, which helps students understanding and create more engaging effective learning experience. So you've talked about working in computer vision and IoT and LLMs. What do those areas have in common? Is there some thread that weaves through the work that you're doing?

That's a great question. As a systems researcher, I'm quite interested in this end-to-end systems development, which means that my focus is not just about improving a particular algorithm, but also thinking about the end-to-end system, which means that I kind of think about computer vision, IoT, and even LLMs as tools where we would want to improve them for a particular application. It could be agriculture, education, or road safety. And then how do you think this holistically to come up with the best efficient system that can be deployed at population scale? Right? I think that's the connecting story here that how do you have the systemic thinking which kind of takes the existing tools, improves them, makes it more efficient, and takes it out from the lab to the real world.

So you're working on some very powerful technology that is creating tangible benefits for society, which is your goal. At the same time, we're still in the very early stages of the development of AI and machine learning. Have you ever thought about unintended consequences? Are there some things that could go wrong wrong even if we get the technology right? And does that kind of thinking ever influence the development process? Absolutely. Unintended consequences are something I think about deeply. Even the

most well-designed technology can have these ripple effects that we may not fully anticipate, especially when we are deploying it at population scale. For me, being proactive is one of the key important aspects. This means not only designing the technology at the lab, but actually also carefully deploying them in real world, measuring its impact and working with the stakeholders to minimize the arm. In most of my work, I try to work very closely with the partner team on the ground to monitor, analyze how the technology is being used and what are some of the risks and how can we eliminate that. At the same time, I also remain very optimistic. It's also about responsibility. If we are able to embed societal values, ethics into the design of the system and involve diverse

perspectives, especially from people on the ground, we can remain vigilant as the technology evolves and we can create systems that can truly deliver immense societal benefits while addressing many of the potential risks. So we've heard a lot of great examples today about building technology to solve real-world problems and your motivation to keep doing that. So as you look ahead, where do you see your research going next? How will people be better off because of the technology you develop and the advances that they support?

Yeah, I'm deeply interested in advancing AI systems that can truly assist anyone in their daily task, whether it's providing personalized guidance to a farmer in a rural village, helping a student get instant 24- 7 support for their learning doubts, or even empowering professionals to work more efficiently. And to achieve this, my research is focusing on tackling some of the fundamental challenges in AI with respect to reasoning and reliability, and also making sure that AI is more context-aware and responsive to evolving user needs. And looking ahead, I envision AI as not just an assistant,

but also as an intelligent and equitable copilot, seamlessly integrated into our everyday life, empowering individuals across various domains. Great. Well, Akshay, thank you for joining us on Ideas. It's been a pleasure. Yeah, I really enjoyed talking to you, Chris. Thank you. Till next time.