The Science of Everything Podcast - Special Episode: Visual Processing in Mice

Episode Date: December 30, 2020

I discuss the methods and results of my Master's Thesis, in which I analysed electrophysiological data of the mouse visual system using a variety of statistical and computational techniques. I consider some of the major research questions addressed in my work, including population encoding, stimulus representation, neural network modelling, and state space modelling of information transformation. Recommended pre-listening is Episode 113: Visual Processing.   If you enjoyed the podcast please consider supporting the show by making a paypal donation or becoming a patreon supporter. https://www.patreon.com/jamesfodor https://www.paypal.me/ScienceofEverything

you're listening to a special episode of the Science of Everything podcast, and I'm your host, James Fodor. In this episode, we are going to talk about visual processing and encoding of information in the mouse visual system. And this is the topic that I have written about in my master's thesis, which I just submitted last week. So I thought that it might be interesting to talk about the research that I've been doing this year with you guys and sort of share some of the questions and approaches that I've been looking at. So this will be a bit more of an informal episode. I'm just sort of going to talk a bit through my thesis and some of the questions that I looked at. No real recommended pre-listening, although it might be useful to have a bit of a background in some of the basic ideas of neuroscience. So you could look at episode 38 neurons and synapses for that, but I'll try to explain things as I go through them.

So the basic idea is I'm going to kind of explain some of the background behind my thesis, and then I'll talk about some of the methods that I use to kind of answer the questions that I'm looking at and talk a bit about the results and some of the implications that I think can be drawn from this. So although I studied the mouse visual system, I think, I mean, the idea is that mice are used as a model organism in neuroscience increasingly, and there's a hope that we can learn something more generally about encoding of information and processing of information that's applicable to other animals as well, not just mice, although of course the focus of my research was, on the mouse visual system because that's where the data came from. So if you're interested more generally in sort of neural processing and encoding of information,

then this episode is for you. So let me start with a bit of background at what I was looking at. So as I was just saying, the mouse has been increasingly used as a model organism in neuroscience in the last decade or two. Traditionally, some of these studies in terms of vision have been carried out actually in cats, So some of the very oldest research was conducted in sort of studying the visual receptive fields was conducted in cats. More recently, the primate visual system, so that includes humans and also like macaque monkeys,

has been subject to a lot of study. Primates have very advanced vision, so it's naturally a good model for that. Also, of course, we're interested in how humans perceive and process visual information. But mice are much easier to study. They're very easy to breed in a laboratory, environment and you can you can insert electrodes much more readily than you can for monkeys and

there are fewer sort of ethical concerns there I mean there are still ethical concerns but potentially lesser than than for say monkeys or certainly humans so for some of these reasons they've been used more increasingly in recent years so that's why we're looking at mice specifically but the broader questions here are about how visual information is processed and encoded in the brain and so mice is just again a model organism here one of the extant questions here is whether mice have the same level of higher visual processing as is present in primates. So to explain what I mean by that, the current understanding of the visual system, and I've done a series of episodes on this that actually could be worth looking at,

so episodes 45 to 47 are on vision. So I guess this could be considered kind of a sequel to those, although it's not necessary to have listened to those, but it may help because those will go into the background in more detail. But our current understanding of the visual system is that it is a hierarchical processing system. Again, this is especially true in primates, but probably true for other animals as well. So that means basically what you have is the initial sensory information is picked up by cells in the retina, which is then processed, sent to cells in primates in the lateral geniculate nucleus.

There's a corresponding region in the mouse as well. I forget what it's called. That's part of the thalamus, which is kind of like a relay station, which information is sort of sent through, which leads to the rest of the brain. So the thalamus is not part of the cortex, which is sort of the more evolutionarily recent part of the brain. So the signals are sent through the thalamus up to the visual cortex, which is at the back of the brain, at least in primates. And from there, it goes first to V1, so that's just the first part of the visual system. And then it's processed and sent to progressively higher regions. So in primates, again, there's V2, V3 and 4, and then regions like the intratemporal cortex, the IT cortex, and other regions as well,

where more advanced processing takes place. So the idea is at each successive level, the processing becomes more sophisticated. So in V1, cells are mostly responsive to, like, lines of a particular orientation and maybe spatial frequency of, like, a grating pattern. At V2, the cells are responsive to maybe, like, specific shapes or textures or perhaps colors as well.

There's a whole color system as well that's embedded into this. And then higher visual regions, the cells are responsible. maybe to particular classes of objects, like maybe they're responsive to faces or tools or particular types of animals. And so this has been found in both monkeys and in humans that they have these sort of semantic category specific responsive cells. So this is what I mean by hierarchical processing, that the processing goes from sort of detecting simple features to more complex features. So if I use the phrase features, this means like just some aspect of the visual,

scene that is detected in a sort of systematic way. So a feature could be as simple as a line of a particular orientation up to, you know, like a specific face and sort of anything in between. So one question is, how much of this hierarchical feature processing does there exist in the mouse visual system? That's one of the questions that I wanted to address in my research. Another question is exactly how does the information processing work as you go from sort of one region, brain region to the next or, you know, one part of the brain to another. We often talk in a vague way about information processing or signals being sent from one region to another, but obviously it's not just a simple like copying of the information because then nothing would happen. You know,

you would just have the same representation in different regions. What we need is a transformation of that representation so that it extracts new information so that you sort of go from the simple raw data that comes from the retina and eventually you can extract features and progressively more complicated features from that until you are able to recognize particular objects and scenes and then extract sort of behaviorally useful information from that. So there has to be some process of which this information is processed, kind of like a computer does, but exactly how this works is not very well understood. So that's another question that I wanted to address. So how is the information processed from going one region to another? How is the signals extracted and the information

sort of collated in a relevant way. And I guess the final question, sort of general question that I was interested in, is how is the visual information represented at a particular brain, in a particular brain region? So in a computer, in a digital computer, information is represented as a series of bits,

so zeros of ones. I've done a whole series on how computers work, if you'd like to take a look at that. But the brain doesn't work like that. There's no like storage modules where particular information is represented as a series of bits. We don't represent an image,

in our brains as a series of pixels of like, you know, colors of, so there's three primary colors, and then the intensity of each color at a given location is just represented by a number that gives how intense the light is, and then you have, so three numbers basically for each pixel, and then you have how many pixels in an image, and that is how you describe the image. You can represent an image like that in a computer, but that's not how the brain represents visual information, or really any sort of information. So the question is, how does it represent the information? Obviously, it's represented as some sort of pattern of neural activity, so firing rates of different neurons is one hypothesis. But how does that work in detail?

What's the structure of that? And so that's the sort of third question that I'm sort of wanting to address in my research year. So to summarize, the three basic questions are, is there, to what extent is there hierarchical processing of visual information in the mouse visual system compared to like primates? How is visual information coded and represented in different visual regions of the mouse visual system? and how is that information processed or transformed going from one region to another in order to extract relevant information? So these questions are obviously interrelated because, you know, hierarchical processing involves representation and processing from one region to another. And the representations in different brain regions will, like you get from one representation

to another through some sort of information, transformation or processing. So these are all kind of different aspects of the same question of information, representation, and processing in the mouse visual system. So that's basically what I'm trying to address in this study. There's a few more sort of background concepts that I wanted to discuss before I start to talk about methods. So the mouse visual system has different brain regions. I've been talking about that. It's got V1, so primary visual cortex. That's the same as primates and humans have V1 as well. In mice, this is the largest of the cortical regions. So information is first sent to V1 via the thalamus.

But after that, the information is processed in two separate streams they're called. So you can think of these as neural pathways. is sent sort of either one way or the other way. Of course, you know, the same information may be sent both directions, but it's processed differently. So these are called the ventral stream and the dorsal stream, which just describes basically their location in the brain, but we're not going to worry too much about that. The ventral stream progresses through a region called L-M and thence to a series of regions including P-O-R, L-I, and P. These all stand for different things. I'm not going to use the full names, just because it's not very helpful. I just generally

use the abbreviations in my work, and it doesn't really matter that you remember these. I may refer to them at some point in the future. That's why I'm mentioning them now. But I'm just trying to illustrate the point that there's a successive series of regions that the information goes through. So in the ventral stream, it's LM, and then there's higher regions there.

And then on the dorsal stream, it's AL, and then goes up to higher regions such as PM, AM, and RL. So those are the main ones that I'm looking at. And these are all part of the visual system. And then from there, of course, the information is further propagated into like motor regions

or somatosensitive regions or wherever else. So you can think of them as two pathways, ventral and dorsal, and basically there's kind of like three layers. There's V1, that's sort of the starting point, and then there's the second layer of LM in the ventral case and AL in the dorsal in the dorsal in the dorsal in the dorsal in the dorsal in the dorsal in the dorsal streams, as well as V1. And what we're looking for is propagation of information and processing across these different streams. and one of the questions is how much higher-order processing occurs in these higher-order visual areas. So remember I said that in primates, as you go sort of up the hierarchy of visual regions, the processing becomes more precise and sophisticated and, I guess, specific to particular types of stimuli. You know, it goes from like just basic line segments to shapes and patterns to more complicated shapes and patterns,

and to finally like object recognition. It's not known whether that also happens in the mouse visual system. So that's one of the questions that I'm looking at here. you move up into these, you know, through the ventral and the dorsal streams, do we have more sophisticated, sort of precise and increasingly, I guess, specific features that are being extracted. Now, in terms of the question is how is information encoded in the mouse visual system? Traditionally, this has, this question has sort of been addressed by looking at the receptive fields of individual neurons. So the traditional model here basically says that certainly in V1, and maybe in other visual regions as well,

each neuron is tuned to a particular orientation and spatial frequency of a line grating. So by spatial frequency, I just mean like how close together are the lines. If they're further apart, then that's a low frequency. If they're close together, then it's a high frequency. And of course, the orientation is just kind of like you rotate the orientation of the lines around 360 degrees. It turns out that if you combine an arbitrarily large number of oriented line segments like this, so of different orientations and different spatial frequencies and also at different positions in the visual field, then you can form any image. So just like you can form any image from just adding pixels of different intensities, you can also form any image by combining these oriented line gratings.

And these oriented line gratings therefore represent a different basis for representing an image. That's a mathematical concept. If that doesn't mean anything, don't worry about it. But you can sort of transform between representing an image in sort of spatial terms as like a, pixel intensities or in terms of frequency terms, which is what these oriented line segments are about. If that doesn't mean anything to you, don't worry about it, some of the listeners will know what I mean there. The point is, though, that this is a different way to represent images. And this has been thought to be the way that, well, not just mice, but humans as well, represent images.

Basically, each neuron, especially in V1, is thought to be responsive primarily to stimuli of a particular orientation and a particular spatial frequency at a specific location in the visual field, so like in the retina. So if I see a line grading of particular orientation at, you know, the bottom left part of my visual field, then certain neurons will response to that, certain neurons won't. And then if you add up the responses of a bunch of different neurons, then you can sort of form a representation of a more complex image. That's sort of the traditional model for understanding this. And so the way that typically these studies have done, have gone about this, is by trying to measure the visual receptive fields of neurons in different brain regions. So like what orientation

are they specific to? What frequencies are they specific to? And how do those compare across different regions? And so I review some of that in my discussion. Now, I think there are important limitations to this approach, although it has been very useful, but I think that it can't get us all the way to understanding vision properly. One of the problems is that most of the stimuli that have been presented in these sorts of studies are just these oriented line gratings, or what are called Gabor stimuli, which is basically the same thing, but it's just spatially localized. So the difference is I present a grating, line grading stimulus, that is just all the mouse sees. It's like the whole visual field is just line gratings. A cabal stimulus is spatially localized. So like maybe most of

the visual field is just gray or blank or whatever, but then there's a localized grating in a particular region. So I won't necessarily distinguish a lot between those two because they're sort of conceptually very similar, but just bear in mind that there are sort of subtle differences here. But anyway, the point is that traditionally most studies, including in primates, but especially in mice, have been conducted using these artificial, very abstract stimuli, because it's thought that, well, what's happening is a bottom-up processing of these very simple line segments, and then they're sort of added together to form a more complex stimuli. So to understand how this works, we just need to understand the receptive fields, so the responsiveness of different neurons to, you know,

which orientations and which frequencies and so on. However, I think there's a big limitation to this, because we know that visual processing is not just all bottom-up. It's not just the case that, you know, V1 neurons get a stimulus and they process that and send that to a higher, you know, in the ventral stream, say, and then that processes and sends the signals up to a higher level in the, in the ventral stream, and so forth, likewise for the dorsal stream. That's been sort of the paradigm model, but we know that that's not the case, that it's just feed forward processing from sort of low to high. There's also a lot of lateral connectivity, so that means connectivity between neurons, but also between regions, between different brain regions. So like at the same

level of the hierarchy, there's connectivity across regions, and again, between neurons within those regions. And also there's feedback connections or back projections, which means connections from higher levels to lower levels. And these connections are known to be pretty widespread and significant. You can see this in, I show a figure in my paper where I show how many of these connections there are that go backwards and that go sideways. and the traditional models just kind of ignore this, and I think this is an issue because we have a lot of evidence from psychology and other areas that these sorts of things matter a lot.

But for example, expectation effects matter in vision. If I'm expecting to see something, then you can think of that as kind of, there's a part of my brain where that expectation is represented, and that sends an image kind of back to the lower levels of my visual systems, essentially saying, like, look out for this thing. And so it's not just that the lower levels are saying, hey, this is what I'm seeing,

but the higher levels are kind of saying back, kind of look for this and so fit it into this framework. And so what you see is shaped by what you're expecting. And also we have priming effects where basically if you, if you present, like if you pair a stimulus with something else, then you would be sort of primed to more readily detect that, the pairing in the future. And again, that can really only be understood as a sort of a feedback effect, not as just a feed forward. And there are computational models as well that involve showing how if you have feedback or what are called recurrent projections that that can help you to store memories. But this involves basically the feed forward processing being influenced by the

existing patterns of stored memories. So effectively what this means is if I'm looking at an image, I'm trying to make sense of what it is. And so I'm trying to fit it against things that I've seen before. So I'm like, this thing looks a bit weird. Say I'm looking at an animal. I'm like, what animal is this? I'm not just constructing that from the bottom up. I'm trying to fit it against things that I've seen before. Like is it a cow? Is it a horse? Is it a donkey? What is it? And I'll sort of fit it against things that I've seen before. And so that represents a back projection of information from like where I'm storing the memories to the V1 cells, which is sort of processing that information. So the point is there's a lot of lateral and feedback projections

that are all contributing to how the information is processed. And so if you just look at low to high, feed forward projections, you're going to miss all of that. And especially because if you're presenting very simple stimuli. Again, I don't think you're going to have much of those sort of feedback and lateral projections that you're probably just picking up very simple feed forward projections there because, you know, I mean, there's not a whole lot of expectation effects probably or memory comparisons that I'm making if I'm just seeing a grating stimulus. There's just not a lot there. Plus, it's also known that we remember things and process them a lot more like intensely and focus on them more and so forth if they're a relevant stimulus. So if I show you a Goli

line grating, you're like, okay, so what? You know, this is kind of boring, it's not interesting, this is irrelevant to me. So I'll still see it and process it, but I'm not really going to be very focused in terms of attention-wise, but also just my visual system's not really, hasn't really been adapted to process line gratings. And so there's not a lot in my memory that corresponds to this. Compare that to if I show you an image like, you know, of a tiger running at you or a car that's about to hit you or, you know, a picture of someone that you know.

these are all things that are salient to you because they're relevant to your experience, they're things that you've seen before and have been encoded into your memory. So the hypothesis here is that there's going to be a lot more and richer processing occurring for stimuli like this that are, the phrase is ecologically relevant. It just means they're sort of relevant to the animal and the sort of environment that they would live in and probably have seen before. So you're going to have richer and more robust responses, more differentiated responses as well to ecologically relevant compared to ecologically irrelevant stimuli that are,

like the simple line gratings. And so again, I think that we're going to be missing a lot of the processing, a lot of the interesting stuff that's happening when you present ecologically relevant stimuli. So as a combination of the fact that the artificial stimuli that are used are very simple, they're not ecologically relevant, and they're not going to elicit the same degree of lateral and back projections because they're things that have not been seen before and they don't correspond to anything meaningful in the memory and so forth, a combination of these factors means that I think that if we just focus on the feed-forward receptive fields of the neurons to simple simuli, we're going to miss a lot of what's happening in visual processing.

And so what I'm interested in doing is looking at responses to complex stimuli. And so what this means is basically natural scenes. So this just means a photograph of some real thing, often like animals or plants or, you know, buildings or really anything. It's just anything that you could see in the natural world as a natural scene, as well as videos. So, you know, moving images. So these are some of the examples of complex stimuli, which I'm kind of particularly interested in looking at, because I think that in order to understand how the encoding and processing works,

we're going to need to move beyond the sort of very simple stimuli that are not very meaningful to the animals. So that's one important aspect of what I'm doing in the study. It's not been done a little bit before, but not so much. He's looking at the responses to these complex, more naturalistic stimuli instead of the very simple ones. Now, there's another aspect of what I'm doing here, and that is applying not exactly a new parrot, but it's a slightly different paradigm in terms of looking, in terms of analyzing the neural data. So the basic idea of the data that we're going to get is we're going to, well, I didn't do this personally, but the idea is that you stick electrodes in a mouse's brain and measure activity

from a bunch of different neurons while the mouse is viewing the stimuli. So this is called, these are called electrophysiological recordings. You're measuring the electrical activity of multiple neurons at a time. Generally, it's like a few dozen to a few hundred neurons at a time. This is only a tiny fraction of the millions of neurons that are going to, going to be involved in visual processing, but it's at least better than measuring from only one or two or three neurons at a time. And the idea here is that we're going to need to measure patterns of activity across many neurons. When I say activity pattern, what I mean is that, suppose I have three neurons, I can specify many different combinations of activity. It could be

neuron one is highly active, neuron two is moderately active, neuron three is inactive. Or it could be neuron one is moderately active, neuron two is highly active, and neuron three is also highly active. You know, there's lots of different combinations. And of course, it's not just like qualitative. You can put a number on the firing rate of the neurons. And so that combination of activities is what I call a pattern. You can represent that just as a series of numbers, you know, the activity, the firing rates of how many neurons that I'm measuring from. And typically what we're doing here is taking activity from a particular brain region. So say from V1, maybe I'm measuring from 50 different neurons in V1 in response to a given stimulus in a given animal. And I've got

firing rates of 50 different neurons at a given time, and then I see how that changes over time with different stimuli. And so when I say activity pattern, I mean a vector of 50 numbers, which are the firing rates, in response to a given stimulus, and they change, obviously, when you change the stimulus. So the firing rate is averaged over the time that the stimulus is presented for. So this is what I mean when I say activity pattern. It's just a series of numbers that represent the firing rates of different neurons. The important thing about activity patterns is that we're not just interested in questions like, is the total amount of activity higher or lower? or something like that, or which type of stimulus is each neuron most responsive to?

That's a more traditional way of looking at it by studying the receptive fields of different neurons. So that traditional approach would ask, like, which line orientation is this neuron most responsive to? Which line orientation is this neuron most responsive to? And so on. And then you could maybe construct a model on the basis of that. But that's not what I do in this study. That's been done a lot, and I think that it's interesting, but we need to move beyond that because there's more that we're not getting out with that approach, which is just,

the very feed forward focused on simple stimuli. Instead, I adopt what's called an activity state space approach. So remember I talked about a pattern of activities that you can represent this as just a vector of numbers. You can sort of visually represent that or imagine that as a vector that points in a space. So suppose I have three neurons, I can represent the activity pattern of those three neurons as a vector of three numbers, which in turn I can plot as just an oriented line in three dimensional space. Each axis corresponds to the activity of one of the neurons, and the line,

the vector points in a different direction depending on which neurons are more active compared to each other. So you can visualize this in three dimensions. You can't really visualize it in four or more dimensions because you run out of directions to plot your axes in, but you can still describe it mathematically. So this is what a state space is. It just represents different possible states that the system can be in, in this case a state corresponding to a particular pattern of activity. So if I have 50 neurons, I can describe the total set of possible sets of activity patterns as a vector in 50 dimensional space. Again, you can't visualize all 50 dimensions, but you can still describe them mathematically. And so my approach here is to say that a particular visual stimulus

is represented by a particular activity pattern in n-dimensional space, where n is the number of neurons. In practice, it's going to be, you know, tens of thousands of dimensions, probably. Although there's also redundancy, so it's not the case that every neuron is going to be a unique dimension, but that gets to more complexity than what I get into here. For our purposes, you can just think of one neuron means one extra dimension, and we're dealing with maybe a few dozen to up to it, maybe 100 dimensions at a time. The idea, then, is different stimuli will be represented by different patterns of activity, which correspond basically to vectors that point in different directions.

And the hypothesis that I'm making here is that there's a similarity relation. That is, more similar stimuli will be represented by activity patterns that are more similar to each other. So you can think of that as vectors pointing not in exactly the same direction, but a more similar direction. An example that I give in the paper is if I'm looking at an elephant, each time that I look at an elephant, the activity pattern in my brain is going to be slightly different. I mean, there's noise and there's variations. Also, there's change in the stimulus. I don't see the elephant from exactly the same angle. It's a different element maybe, or the lighting is slightly different, whatever. But in all of those cases, the activity patterns are going to be pretty similar because it's still an elephant, right?

And in contrast, if I see a duck, again, there's going to be some variation. The pattern is not going to be exactly the same, but it's going to be probably pretty similar because I'm looking at the same thing. However, the elephant patterns and the duck patterns are probably going to be more different from each other because they're different animals. And maybe they'll still be more similar to each other than if I'm looking at a skyscraper, which is not even an animal. It's something wholly different again, right?

So the idea is that the more similar the stimulus is, the more similar the activity patterns are. Effectively, that means the more similar the vectors are, so they're pointing in a more similar direction. So that's the activity, state space, and similarity pattern structure. In fact, what I hypothesize is that as you move across regions of the cortex, like higher visual regions that I talked about, the representations of different objects or different types of similar become more distinct. So the idea here is what the brain is doing is it's performing pattern separation. If I just see an elephant and a duck, if I just look at that at a purely, you know, like a pixel level, you know, there's some pixels that are dark and there's some that are light. Like, you know,

what's the difference at that level? At that level, it's maybe sort of hard to see the difference. There's going to be a lot of overlap. But what I need to do to understand the distinction between these images is to extract the relevant features. You know, elephants have tusks and they have big ears, whereas, you know, ducks don't have that. Ducks have feathers and maybe ducks are often found in water. There's lots of differences, right, that help you distinguish an elephant from a duck. The idea is that this is happening in the visual system.

We're not necessarily conscious of it, but it is happening. And what different regions of the visual system are doing is extracting these features, and therefore representing progressively more and more distinct differentiable patterns of activity. So at the low levels, there'll be some differentiation because ducks and elephants are different, but it won't be as pronounced because basically that lower region of the visual system is representing lots of things that are irrelevant to categorizing the stimulus. Whereas at higher visual regions, they focus on extracting the features that are relevant to distinguishing those objects or the different scenes,

and therefore the patterns of activity become more distinct. So you can think of this as vectors that start as pointing in a similar direction, but become progressively more and more differentiated. They point in more and more different directions as you move across into higher and higher brain regions. So that's the hypothesis of how this is working. So the hypothesis is that the visual information is represented as a vector, as a pattern of activity in the state space,

and that the transformation of this information occurs as the brain extracts relevant features and thereby changes the patterns of activity and therefore effectively the direction that the vectors are pointing in as you move from one brain region to another. And under this model, understanding processing in the visual system would basically correspond to understanding specifically which features are being extracted by different brain regions and what the functional transformation is that we can use to describe how a pattern in one region is transformed into a pattern in another region because you can describe that basically as a matrix transformation. If you, again, if you have a bit of maths background, you know what I mean.

If I have two vectors, I can describe the transformation from one vector to another by a matrix transformation. So in theory, then, you could predict the activity pattern in one brain region from another brain region by applying a matrix transformation. And therefore, if we could learn what that matrix transformation was, then I could understand how the visual system is processing information. And so that relates to one of the models that I construct that I'll talk about in a little bit. So hopefully that sets the stage for how I'm sort of conceptually thinking about these issues of

representation and information processing across the brain, the visual system specifically. So now I'm going to talk a bit about some of the methods that I use to address this. I've already said that I'm using neurophysiological data, so this is recordings of activity across a bunch of neurons in different brain regions of the mouse, where the mouse is presented with different visual stimuli. So this includes both some simple like oriented line stimuli, as well as more complex natural scenes and video stimuli. I focus more on looking at the latter. As I said, most of this data collection was not performed, well, none of it was performed by me directly. Unfortunately, I wasn't able to devise my own stimulus set to use, which I would have

really liked to do. What I did is reanalyzed data from the Allen Brain Project, which is a really cool project that you might want to look up. They've got a lot of good data online, not just from mice, but from like primates and other organisms as well, but I've obviously looking at the mouse data. They've got dozens of experimental sessions, so each experimental session is with a different mouse, and then they present them with many different stimuli. So, for example, there's a set of natural images. I think there are 118 of them, and then they present each of them a few different times.

I think it was like 10 or 20 times, and record the resulting activity. So what I'm doing is analyzing this data. I'm downloading that data. It's quite a lot of it's like 100 gigabytes or something that are downloaded, and then processing all of that. So trying to understand what's happening in these different brain regions and based on the different levels of activity.

We also did collect some data in our lab, so we did one experimental session with a set of 500 images, which was good to add to the data set. Again, I didn't collect that directly, but I did work with processing that into the form that I could use. So mostly it's the Allen Brain project data, but we also did have a little bit of in-house data that we added. So that's basically what I'm doing. In terms of, in analyzing the data, there's a bunch of different statistical tests and computational models that I applied. I won't go through that in great detail, because it's a little technical, hard to describe without showing you equations or graphs.

Plus, it's maybe not that interesting. I'll talk a little bit about it as I go through this. One of the things that I did do is, or try to understand the processing of information by using neural network models. So I should describe those. A neural network is an artificial model, so it's like an abstract mathematical or computational device that's used to understand how, well, it can be used to understand how the brain processes information.

But, I mean, these things are why they're used in artificial intelligence these days as well. If you've heard about deep learning, that's what this is. It's just artificial neural networks that have lots of layers in them. Basically, the idea is you have a series of nodes, organizing different layers. There could be just as few as three. Well, actually, as few as two really, but say you could have three layers. An input layer, a hidden layer, which is just like the middle layer, and then an output layer. And the neurons in each layer are connected to all of the neurons or nodes in the next layer. So these are fully connected networks. It's basically just imagine it as a sort of mesh of wires connecting. nodes together. And what happens is that you train a network like this to learn patterns. So, for example, I could just give them, well, an example, if I talk about the deep neural networks that

have been trained to recognize images, I could provide as input a series of pixel intensities, which represents my image. And as the output, I could have a series of categories, so just like a number between 1 and 20, if there are 20 different categories. And the category represents, the category that I've assigned the image to. And so what I'm going to do is I'm going to train my network to learn to associate the correct category label with the input image that I provide. And to do that, of course, I need a lot of data, a few stimuli and go to work. Generally, these are trained on millions of stimuli with hundreds of different categories. And of course, there's not just like one hidden layer. There's, well, it depends on the network, but these days you can have hundreds

of hidden layers. So you need lots of data, you need lots of computational resources to train these, to train the big ones at least, but they can get become very accurate. This is essentially how Facebook or other tools recognize faces in images or recognize other types of stimuli. They can tell what they are because they've been trained on, they've trained to neural networks on very large datasets essentially. What I'm interested in here is whether these models, these artificial neural network models can be used to understand how the actual brain works.

This is where the idea of neural networks came from that came from an inspiration of the brain, but it's sort of since diverged from that. but I'm asking the question as to how much they can be useful in explaining the actual process of information in the real brain. And so that's one of the approaches that I use. I train neural networks to see if I can predict the activity in one brain region from the activity in a previous brain region, just as I described before. Hypothetically, if you could learn that matrix transformation between one region and another, you could predict the activity at better than chance levels. And so that's one of the techniques that I applied.

another technique that I applied using neural network models is to take an existing what's called a convolutional neural network. That's just a type of deep neural network that's been trained to categorize images. Retrain that on the images that were presented to the mice in these experimental studies. And then see if the way in which that network, the convolutional neural network, represents the stimuli, is similar to the way in which the mouse brain represents the stimuli. There's ways of doing this comparison, which I might get to in a little bit. but just the basic idea is if the way that the convolutional, the artificial neural network represents the stimuli, is similar to the way in which the mouse represent the stimuli,

then that's evidence that there's similarities in processing going on. We can't necessarily be too specific and say exactly what is similar, but it seems that, for example, if we have significant similarities at different levels in the convolutional neural network, then perhaps the mouse is also extracting features in a similar way to the way the convolutional neural network is, because these convolutional neural networks are sort of trained, they're structured specifically so that they extract features across different regions. Just like I discussed before, this is how the visual region is thought to work

by extracting more and more sophisticated complex visual features as you go across the visual hierarchy. It's thought that this is how the brain works, but in order to test that, we can compare the representations to those learned in the convolutional neural networks and see whether there's any similarities. So that's another thing that I did. I also just looked at the raw experimental data

and tried to see whether there was evidence for this pattern separation. So remember, I hypothesized that stimulus is represented by a particular pattern of activity, and these patterns of activity are transformed across brain regions, and I hypothesize that there's a similarity of relationship here. That is that more similar stimuli are represented by more similar patterns of activity. So that's something that I tested for. I also tested for whether there are differences in the representations in different brain regions, because remember the hypothesis is that higher brain regions extract more complex stimuli

and probably respond more specifically to more complex stimuli, like the natural scenes, as opposed to cells in the lower regions like V1, which are probably more specialized to just responding to oriented line segments or gratings, and so those might be more specifically responsive to the simple artificial stimuli that I mentioned. So these are the kind of things that I'm looking at in the analysis that I conducted. And so there's a lot of, you know, I had to write a lot of code and conduct statistical tests and computational models and so on. But let's talk about what I found, because that's the interesting part.

I'm not going to talk about every single result. I'll just try to give the broad picture. So one of the things that I found, remember, I was looking for similarity relations between the stimuli and the activity patterns. One way that I looked at that was by comparing the similarity of pattern activities for experiments, experimental trials in which the mouse was presented the same stimulus, and compare that to trials in which they were presented different stimuli. So what I would expect is that the patterns of neural activity would be more similar

when the mouse is presented with the same stimulus. So if I present an image of a dog 20 different times, the resulting neural patterns are going to be different because there's a lot of noise and there's other things going on, right? But I would expect them to be more similar than if I just compare that those patterns in response to the dog to the patterns in response to all the other stimuli. Because there's variation, you know, in any case,

but there's also variation due to the stimulus changing in the latter case, whereas in the first case, the stimulus is always the same. So what I predict, if this is, if this hypothesis is correct, is that there's a systematic and significant difference between the same stimulus pattern similarities and the all stimulus pattern similarities, because, again, the same stimulus should be encoded in a more, in a similar way compared to all the other stimuli. And that is what I found. I found this for every different type of stimulus that we looked at, from the simple simulator to the complicated stimuli and the movies as well. Also, I looked at this in different brain regions, and I found the same

thing in each brain region. It's pretty consistent across the different brain regions. So the difference is not large. I measured the similarities of neuralactivity. on a scale that varied from 0 to 1, from 0 being completely different to 1 being exactly the same. And what I found was that the difference in the sort of same compared to all stimuli varied from maybe about 0.05 to 0.2 in the highest case, although mostly it was around to 0.1 as an average. So that's about 10%, which is not huge, but it's definitely significant. And remember, we're only looking at a very small number of neurons. And so I wouldn't necessarily expect to see a very large difference because many of the neurons are not going to be very responsive to the specific stimuli

that I'm showing, or that were presented. This is what's called sparse encoding, which means that like most of the neurons are not doing that much, or they're only responding a little bit. They're only going to differentiate their activity a little bit in response to the stimulus. I'd say many of the neurons that are differentiating most between the stimuli were not measured in the experiment, and therefore they won't show up here. So the idea is I would potentially expect to see a larger difference if I could somehow measure all of the neurons that are relevant. The basic idea is I did find what I expected here. There is a pattern similarity difference,

which essentially is evidence that different stimuli are encoded differently in terms of the patterns of activity. But what I'd really like to do is see if there's a graded relationship here. By that, what I mean is that more similar stimuli are encoded in a more similar way. All I could do by looking at it in this way is to say that if it's a different stimulus is it encoded more differently compared to if it's the same stimulus. But what I'd like to do is have a gradation of more, from very different to more similar, progressively more similar stimuli.

Now, unfortunately, the stimuli that were presented in the Allen Brain data are not conducive to that, because they're not designed it for that sort of analysis. This is why it's unfortunate that I wasn't able to collate my own set of stimuli to use because I could have designed it so that there was a gradation in stimuli. Similarity. You could imagine, for example, the same stimulus seen from progressively different angles or further and closer away, or I could progressively change features of the stimuli so that it's more and more different.

That's the sort of thing that I've wanted to do, but unfortunately wasn't able to do. However, there was one type of stimuli that I was able to do a graded similarity test with, and this is with the movies. Because remember, the nature of a movie, well, at least most movies, is that you have a picture that gradually changes. Like, imagine people walking across the screen. The image is at first very similar if you just look at it frame by frame, and then the legs gradually move, and then the people gradually move across the screen,

you know, cars move in the background, this sort of thing. Images that are one frame away from each other are going to be very, similar to each other, but 10 frames away, it's going to be more different to 100 frames away. After a certain point, you know, it's probably not that much of a difference if it's 200 or 1,000 frames away, because basically you're just talking about completely different scenes then. But at least up to a certain point, there's going to be a gradation in similarity then. So starting off most similar and becoming more different. And so what I was able to do is use this temporal interval, basically how close are the frames

in the movie, and see whether the pattern activities for stimuli that were close to each other, for frames that were close to each other, became progressively more different to frames that were further and further apart from each other. And what I found is that, in fact, you do observe this. So again, the effect is not massive. It's not like the points fit neatly along a line. There's a lot of variation, right? But when you fit a line to this, and I'd show figures in my thesis, you can see an effect here. Thankfully, the Allen Brain Stimuli included the same movie, one with the frames in order and one with the frames shuffled. So randomized. And that's good because I can make a direct comparison. If the frames are shuffled,

I shouldn't expect to see any relationship between how close the frames are and how similar the activity is, because it's just all over the place. The frames aren't in any order. Whereas if they're in order, I should expect to see that relationship. And so I can compare the two cases and see whether it's just the ordering effect that's making the difference, which should mean then that it's actually the similarity in the stimuli that's making the difference. And that is what I observed. So in the shuffle case, I observed an average correlation. Well, again, it depends on the brain region you're looking at, but say V1. In the shuffle case, I observed an average correlation of about 0.01 to 0.02, say.

Whereas in the case where in order, the correlation was about 0.2. So that might not, it's not a very high correlation, but it's a lot higher than 0.01 to 0.02. It's about 10 times the correlation. So in the first case, the correlation is basically insignificant. It's basically 0. Whereas in the second case, sure, it's not high, but it's very significantly different from zero. So again, that's significant evidence that the more similar stimuli are being encoded in a more similar way, in a graded relationship as the frames progressively change,

the patterns of activity that are measured in result, that result from those become more and more different. And this is all presenting, and if you look at the, it's extracted from an old movie. The videos in question are of like people walking across the street and cars moving and things like that. So this is not ecologically relevant to a mouse, right? The mouse has never seen this before. And so I think it's quite significant that we see these results, even when the stimuli are not ecologically relevant. I would predict that we would see more significant results when the stimuli are ecologically relevant, because what I would predict is that the mouse's brain is going to extract more

relevant features and recognize essentially the objects and the scenes as more distinct from each other and therefore produce more differentiated patterns of activity when it's actually relevant, and it's seen it and learned it before compared to irrelevant stimuli. But even with non-ecologically relevant stimuli, we're still seeing a very significant effect. So this is one of the results that I think is particularly interesting. I looked at the differences between different visual regions. I won't go into the details of that, but I did find some systematic differences. So some of the regions were distinctly different and consistently so than others,

so they represent the information differently. In particular, I found that V1 was significantly and consistently different from the higher visual regions, which is a result that has been found before, but I looked at it in a different way, looking at these patterns of activity. So that's consistent with the hypothesis that the activity patterns are being transformed from lower to higher visual regions.

I didn't find a consistent difference between the ventral and dorsal pathways. Remember I talked about those two different pathways. I may have predicted that there would be a difference found, but I didn't find that difference, although I think that there's reasons why that may not have occurred. For example, the ecological relevance of the stimuli, as well as the fact that only one region from,

I think it was the ventral stream, was actually recorded from in the study. I don't know why that was the case, but that's the data I had to work with. And so it's a little bit hard to compare the difference between two streams if you're only measuring one region from the ventral stream and four from the dorsal stream. So the idea that patents of activity progressively transformed across the hierarchy of regions was sort of partially supported here. There was an effect from V1 to higher regions,

but not a very clear relationship between those higher regions. So that's partial support there. Remember I also talked about the neural network models that I trained, So I talked about the convolutional neural network where I compared the representation of intermediate layers in the artificial, convolutional, deep neural networks that have been trained to recognize stimuli compared to the actual experimental mouse stimuli. The hypothesis is that there are going to be significant correlations, basically similarities between how the stimuli are represented in the two different cases. And this is indeed what I found. This has been reported before for primates. I don't think it's

have been reported for mice or for rodents. So this might be the first time that this has been done in rodents. But so I'm replicating here results that have been found for primates, which shows a similarity here. So the correlations that I found, I use a comparison method called representational similarity analysis, which basically means that I'm looking at the correlation matrix between activity patterns in response to different stimuli, and I'm comparing those correlation matrices to each other.

So I'm not directly comparing activity patterns. I'm comparing correlation matrices. If you don't know what that means, then don't worry about it. I'm just, for those you are interested, that's what this is. So basically, I'm just doing a comparison of how does this neural network represent the stimuli compared to how does the brain represent the stimuli, or how do different regions in the brain represent those stimuli? That's what I'm looking at here.

And the relationships that I find are significant, not large, but still statistically significant. So it does depend on the stimuli that I'm looking at. There's two different movies that we used, and then there's the natural scenes. that I also looked at, but we're talking correlations of around, say, 0.05 at the lowest to 0.2 at the highest. So, again, say like 0.1 as a rough average. So again, that's not a high correlation. Correlation, remember, varies from 0 to 1. A high correlation might be like 0.7 or something, but it's still statistically significant. It's very significantly different from 0. So again, even for these non-ecologically relevant stimuli and measuring a small number of neurons,

we're still getting significant results. And so this is some evidence that the mouse is extracting features in at least a somewhat analogous way to how a convolutional neural network is doing it, which I think of some evidence that these models are actually getting at something like what the brain is doing, which is quite interesting. The final result that I wanted to mention is that I looked at predicting patterns of activity going from one brain reach into another. So this is by using an artificial neural network. This is different from the convolutional case, which is not predicting any neural activity patterns. this is just seeing is how the artificial neural network represents the stimuli, similar to how the real neural network does it.

This case here is actually using a neural network to directly model the transformation of activity like neural firing rates from, say, V1 to LM or V1 to AAL or like whatever brain regions it is. And the hypothesis here is that the model, even very simple models, can do this accurately. And basically the idea here that this would provide evidence in favor of the kind of matrix transformation that I described before. You start with a pattern of activity described as a vector in one region. That just transformed into a new activity pattern in a different region, and you can represent this transformation as a matrix.

If we can learn that matrix, then we will understand better how the processing occurs. So the attempt of this simple network here is to learn that matrix transformation. Again, it's not going to be very good because we don't have that much data to train it on and only a few neurons to work with. So I didn't expect it to be great, especially as it's a very simple model. I didn't try to optimize all the parameters to get the best performance because I just wanted to compare it across different regions and so forth. Even so, what I found is that the accuracy in terms of measured as to how accurate are the predictions compared to the actual activity patterns. The accuracy was significantly better than chance.

It's a little tricky to how to measure chance levels here. One way that I measure chance levels is just to assume that is to compare it to a case in which the network always predicted that the output, region, so like the subsequent higher visual region that you're trying to predict, just always predicts that it has the average output activity. This is basically like saying, if I ask you to predict people's heights, how accurate would you be if you just always guessed the average height, right? And on average, you're actually not going to do too bad because, like, most people have around the average height, right?

So how accurate that is depends on the variance of the data you're looking at. But I think that's actually a good measure of sort of chance levels here. So if I just always guess the average, how close do I get there? And you do all right, just always guessing the average. Again, partly I think that's because we're not dealing with ecologically relevant stimuli here. We're only using a small subset of neurons. So there's a lot of noise in this system, basically. A lot of the variation activity is just noise.

It's not actually representing the stimuli. But some of the variation does represent the stimuli, and so we're trying to extract that bit of it. And what I found is that the activity, the accuracy difference was significant in all cases and varied from about 0.02 to 0.07. So let's say roughly 0.03 or 4 as an average. So you can think of this as like 3 or 4%. So that's not a very large effect, but it is statistically significant. And again, given how simple the model is and the data that we're working with,

I actually think that's quite an impressive result. So again, what this is showing is that we can describe the transformation of activity patterns from one brain region to another using a very, very simple. So there's only one hidden layer in this network here. So much, much simpler than the deep networks that I talked about before. Even this very, very simple model here trained on a very small amount of data. So like there's only a few hundred trials generally that it's trained on, with a small number of neurons, can still predict at greater than chance levels

the activity of subsequent layers. And so I think that this is evidence that that kind of matrix transformation type processing, that something like that is happening in the brain. Bearing of mind also that this very simple model completely neglects all of this lateral and back projections and stuff that I mentioned before, because that would be too complicated. We don't have enough data to train all of that. So even just the basic feed-forward aspect, you can do greater than chance levels of prediction based on these simple models.

And so I was quite impressed by even how very simple the models were and how little data they had to be trained on that they were able to perform at the levels that they did. So that's kind of a proof of concept that we can analyze processing in this way. And the idea would be to try to get larger data sets, record from more neurons, and gradually increase the complexity of the models to get more and more accuracy. So those are the basic results that I reported. So there's the similarity relations that I mentioned. I found evidence of a graded similarity relationship between the stimuli and the resulting activity patterns. I found evidence for different representation and encoding of information in different brain regions, particularly V1 compared to the higher brain regions.

And I found evidence that both the convolutional deep neural networks and also the very simple neural network models that just predict firing rates from one region to another, I found evidence that they were both accurately describing the representation and processing of information in the real mouse brain, at least to a significant extent. So what does this all mean? What's the implications can we draw from this? Well, I've kind of been explaining this as we go, but the basic idea is I think that the data that I present provides support for the model that I was using. Remember I talked about the state space approach where I'm talking about how patterns of activity vary in proportion to the similarity of the stimulus. and how those are progressively separated more and more as you move to higher visual regions, and that this is basically what the brain is doing when it extracts features and represents the stimuli.

I found evidence for that. The one thing that I didn't find evidence for was an increasing pattern separation. So I didn't find that the degree of a difference between the stimulus similarity and the patent similarity. I didn't find that that increased as you move across higher visual regions. If anything, I found that it decreased slightly, although I don't necessarily think that that's significant. but it seemed that it was pretty consistent. So again, I think that the reason for that is probably because the stimuli that we're dealing with are not ecologically relevant. So think about how we think that this works.

We think that this works by the mouse learning, or the organism, but in this case a mouse, learning to associate particular stimuli with particular behavioral responses or other stimuli that they might receive. So if the mouse sees food, for example, it may also smell the food, may detect certain things with its whiskers. Meister not very visual. They're actually used their whiskers to detect things. So it's probably detecting certain stimuli with its whiskers at that time. It probably sees other things that go along with that.

And then there's a behavioural response, a reward that follows at some point soon after it because it gets to eat the food, right? So there's all of these other things that can be used to help learn and reinforce the visual stimuli that are associated with that scene or that object. None of that is going to happen for things that are not ecologically relevant. It doesn't know what an elephant is

or what a street scene is with people walking along the street and cars moving. None of these things are relevant to the mouse, so it hasn't learned to separate those patterns and basically learn those objects. It's basically like if I showed you, if you don't speak Chinese and I showed you a bunch of different Chinese characters, it's true that you can tell that they're different at a low level. You can see, well, you know, the strokes are in different places and different orientations and so on, they're different shapes.

But what you're not going to do is be able to represent those at a higher level as meaning anything more significant. So you wouldn't expect to see this sort of pattern separation. that represents high-level feature extraction, because there's just no high-level features that mean anything to you. So this is basically, I think, what we're doing when we present these very simple stimuli to mice. It's kind of like showing them Chinese characters, which when they don't speak Chinese. It doesn't mean anything to them. So you're not getting a lot of differentiation. You're getting some differentiation, because at a very low level, they can see that they're different

just from the base features level, but you're not going to get that separation across higher visual regions. And so my hypothesis would be that if we could present more ecologically relevant stimuli, and maybe as well as we might need a slightly larger number of different stimuli as well to make sure that we get significant results, I would expect to see this pattern separation across visual regions. So that's a prediction I would make for future studies. Likewise, I think that the data that I, the results that I found provide support for the idea that the stimuli are progressively transformed from one region to another using something like a matrix transformation. Of course, that's an abstraction that's simplifying it, but I think the results that I show indicate that, both with the convolutional neural networks and the simple neural network case,

that this is a useful approximation that provides real insight into what's happening here, and so the idea would be to try to build on and expand those models. That takes us back to the questions that I was addressing about the similarity structure, so I did find evidence in favor of that. I found some evidence that higher visual roles play different roles to lower visual regions, like V1 compared to the higher regions. I did not find evidence for the increasing pattern separation. I just explained why I think that I didn't find that.

And I also found evidence that the neural network models that I were using and the matrix transformations that sort of they embody are useful models for explaining how the information is processed in the mouse visual system. So my recommendation for future study here would be to build upon this basic framework of the space state representation with the sort of matrix transformations from one region to another by presenting a, by devising a stimulus set that is more ecologically relevant to mice and that also involves a graded similarity relations, the different images. And the hypothesis would be, and maybe we'd also need to measure from a few more neurons, so that's it, there's partly a technological limit there, but also you can measure across more mice and average it across experimental sessions to helping increase sample size, essentially. But the hypothesis would be that you would see an increasing separation of these patterns of activity as you move across the higher reaches of the mouse visual system, as the mouse basically perform object or scene recognition by extracting more and more sophisticated visual features. Ideally, we'd also be

able to construct so to train convolutional neural networks from scratch on very large sets of images that are presented, a subset of which are presented to the mice. I wasn't able to do this because that would require a much larger set of images than I had and much greater computational resources than I had available. So I just retrained existing networks that were trained on mostly different sets of images. And so obviously that's going to lead to inferior results than if you had specifically trained those models just for the types of stimuli that the mouse are going to see. So that would be another extension to train these models with more specifically relevant results. I also mentioned that the simple models that I use for predicting activity for one region to another, you could make

those more complicated. You could train them on much larger data sets and see if you can improve their accuracy. Oh, and one other thing that actually could be done that I mentioned in the discussion is combining the visual stimuli with some sort of behavioral output. So it's going to be particularly relevant to the mice. If you can associate visual stimulus with some sort of behavioral response, they're more likely to remember and focus on the stimuli. And so trying to develop an experimental paradigm in which this was possible, I think would also be a way of extracting sort of more relevant and richer processing information and activity patterns. So those are a few, there's some other things that I discussed, but those are some of the sort of implications and

future directions that I discuss in my discussion section. So that basically concludes what I wanted to say. So the sort of the short takeaway is that I think that this states-based representation is a powerful way of understanding how visual stimuli are represented. And I think that the basic framework that is supported by my results is that complex scenes are represented by patterns of neural activity. These patterns are progressively separated across higher visual regions, of course, the hierarchical structure of the visual cortex.

as features are recognized and the representations are transformed across these reasons, and this can be modeled by an artificial neural network and the sort of matrix transformations that occur from one region to a next. But this is dependent on the mouse having sufficient background and kind of experience with these images, and therefore they're being sort of ecologically relevant. And I think that this paradigm will be quite helpful. It's not, I didn't invent this paradigm, but I think I've been the first to really systematically test it with respect to the...

the mouse visual system specifically, and I think this will be quite sort of helpful in understanding how we were able to recognize complex scenes from, you know, starting from some very sort of simple, just patterns of activity coming from the retina. So, hopefully you found this interesting. If so, you might want to consider recommending the podcast by going to iTunes and giving the podcast a favorable review or on whichever aggregator you prefer to use. You can send me an email if you're interested in making suggestions or asking questions or giving other feedback. My address is Fods toave at gmail.com. That's FODDS12 at gmail.com. If you are particularly generous, you can also support the podcast financially, which helps me to get more

content out there. You can make a one-off donation via PayPal, or you can become a Patreon supporter. The links should be in the show description. Or you can go to my website, just type, probably the easiest ways to Google the Science of Everything podcast, and you'll find the website there. In terms of upcoming episodes, the next regular episode will be an episode on special relativity, which I've been spruking for a little while. I'll sort of finally get around to doing that. And then I think after that we're going to do a couple of episodes on cell membranes, which I've been reading up about lately. So stay tuned for those. Thanks again for listening, and I'll talk to you next time.