Algorithms + Data Structures = Programs - Episode 83: 🇺🇸 ARRAY 2022, mdspaces, & More!

Episode Date: June 24, 2022

In this episode, Bryce and Conor talk about Bryce’s ARRAY 2022 Keynote talk, mdspaces and more!TwitterADSP: The PodcastConor HoekstraBryce Adelstein LelbachShow NotesDate Recorded: 2022-06-13Date Released: 2022-06-24PLDI 2022ARRAY 2022Bryce’s ARRAY 2022 Keynote Talk (not online yet)Conor’s ARRAY 2022 Talk (not online yet)C++23 std::mdspanC++23 std::mdarrayC++23 extentsArrayCast Episode 28: Rank and Leading Axis TheoryArrayCast Episode 29: TransposeJ |. (transpose)Dyalog APL ⍉ (transpose)BQN ⍉ (transpose)All PowerPoint ShortcutTools 3.0 Keyboard ShortcutsPowerPoint Morph TransitionReddit: CPP Cast is over, what are you all listening now?CppCast Podcastcpp.chat PodcastNDR PodcastTLB hit 💥 PodcastTwo’s Complement PodcastMagic Read Along PodcastApple Keynote Magic MoveGreat Impractical Ideas in Computer Science: PowerPoint ProgrammingCppNorth ConferenceIntro Song InfoMiss You by Sarah Jansen https://soundcloud.com/sarahjansenmusicCreative Commons — Attribution 3.0 Unported — CC BY 3.0Free Download / Stream: http://bit.ly/l-miss-youMusic promoted by Audio Library https://youtu.be/iYYxnasvfx8

The other one, the other comment though, that I just find hilarious is that it was the criticism that our podcast lacks structure. Welcome to ADSP, the podcast episode 83, recorded on June 13th, 2022. My name is Connor, and today with my co-host Bryce, we talk about Bryce's Array 2022 keynote talk, empty spaces, and more. Now we're Googling Blue Yeti signs. All right, we're just going to do it. So stereo, omnidirectional, bidirectional. You see, I told you. I told you. Boom.

Do you have it? Boom. I told you it was that one. Okay. Now I'm going to say something. And now I'm going to say something. Looks good. Looks good.

All right. High five. Woo! That was the sound of... This is so weird. Have we seen each other since we started this podcast? We have not. No.

No. I actually think the last time we were... Face to face. Wait, was the last time we met each other was when we were on our 2019 trip? No, no, no no no no Prague 2020 oh yeah that's true and we were you working at NVIDIA at the time?

no definitely because I started that trip was right after I started at NVIDIA and we started this podcast in November of 2019, correct? Did we? We started it pre-pandemic? 2020, no.

No, we started it post-pandemic. Listener, do you remember when this podcast started? Oh, yeah. The reason I'm thinking it was 2019 is because in a talk that I gave, I showed a photo of Dublin being like, oh, yeah, Bryce were here. But it was actually – it was a year later that I was – and I think it was meeting C++ and it was an online presentation. And I said almost one year ago was when this photo was taken of Bryce and I who – big announcement. We're starting a podcast. We're starting a podcast. So that was taken of Bryce and I who big announcement we're starting a podcast so that was November of 2020 and that was definitely uh post pandemic yeah so yeah this is the first time

in over two years then so yeah February of uh 2020 was the last time we saw each other and yeah we're here in San Diego. It's very weird. I have to be honest. It's extremely weird because I'm sitting like two feet away from Bryce. We could also sit much closer and whisper into the microphone. Honestly, I'm very happy we live in separate cities. I mean, not in general.

Wow. Wow. Not like it would be nice if we were in the same city but for the sake of this podcast i don't know if i could handle doing this on a regular base just standing there's way too much eye contact i'm just not used to the and that's the thing is I'm usually this close to the mic but here because of my uncomfortability I'm coming in or going out in and out this is great

this is great production value it's like if we're trying to get serious and you lean in we're both leaning in and then there's literally like six inches between it's we need to do this with multiple mics next time I brought my mic you're the one who didn't bring your mic because dear listener live from pldi 2022 adsp the podcast that doesn't explain why i didn't bring my mic it doesn't but it explains

why we are we i was explaining why we didn't have our microphones with us because we are at the PLDI conference. Correct. Well, I have my microphone with me. Connor doesn't have his microphone with him. I mean, I flew out on Saturday and you flew out yesterday. Yeah. And you pinged me. I'm just doing a pop-in.

I'm just popping in keynote, popping out. And Bryce pinged me on Saturday being like, should we record in San Diego? And I was like, well, I'm already here. Also, the room we're recording in is very cold. So we're both like shivering. Yeah. I'm wearing a big, bright, pastel pink hoodie that I picked up in La Jolla yesterday. As Bryce pronounces it, La Jolla.

Because you didn't bring it. You didn't bring a sweater, did you? I mean, we're on the border of Mexico. Do you even own pants? I think you only own shorts. No, I did bring a pair of Lululemon pants. Of course. Which I will be... Please sponsor us. Yeah. Also Bubbly. Yeah, Bubbly, what's up?

I mean, I think it's much less likely we'll get a Lulu sponsor. So what are we talking about today? Well, Bryce just had his keynote. Are we going to talk about PLDI? Yeah, we should talk about PLDI. Today is the screen just went black. I think it's fine. I think it's fine.

It's just a blank screen screensaver. Yeah. Oh, I don't have screensavers. I've never had a screensaver. Gotcha. I don't believe in screensavers. Okay. You want to expound on that or should we get back to PLD?

I always have the default desktop background and the default screen saver. It's just, I don't know. It's my thing. All right. Real expression of creativity there, but okay. But here, yes, we just finished attending the array workshop. So the way PLDI, for those that aren't regular listeners or have forgotten or aren't aware, it's the Program Language Design and Implementation Conference.

It's actually starting on Wednesday. So Wednesday, Thursday, Friday is PLDI. And then the Monday and Tuesday and a little bit Wednesday as well are co-located workshops. So I think there's a three-day PLMW, which is the Programming Language Mentorship Workshop, and that focuses on technical research, blah, blah, blah. There's a bunch of other workshops, but the workshop that Bryce was keynoting at and I was speaking at was the Array Workshop. How'd that go, Bryce? I think it went great. I finished my slides about five minutes before the talk, so just-in-time slide delivery.

I can confirm that was a missed opportunity to say just-in-time slide something something. I did. I just said that. But you didn't say just-in-time. You said just-in-time slide delivery. Did you say just-in-time? I literally said that. He doesn't listen to me, listener.

I'm just too weirded out. By the way, I'm used to interacting with people through monitors now has picked up some bad anti-social tendencies i mean picked up they were always there i think it's just uh with regular practice you know are you an introvert or an extrovert oh definitely baseline introvert people don't believe me yeah everyone say they don't believe me but that's the thing it's, where do you like, what's your recharge method? Is it I've got spreadsheets and publisher schedule and things that I like to update daily like little graphs you know diet stuff and if I go too long without updating that like it starts to give me anxiety because I know that like seven days from now I'm gonna have to go and like try and backfill everything that I missed and the whole life

gets out of whack I run my life on spreadsheets and like if I don't if I'm not able to update that but I mean I'm a very extroverted introvert. I used to think that I was an introvert, and then recently some people close to me have pointed out that I am, in fact, the most extroverted person that they know. I'm trying to get my flight back upgraded, and we're waiting here on the live chat from the airline to see whether it can happen and and it's just the suspense is killing me it's killing me well

we'll find out how we've been recording for what nine minutes now and there's nine minutes before this break ends yeah a lot of technical difficulties not no shade on z who is a professor at carnegie mellon who's running the morning session of Array. He had nothing to do with it, but they started trying to set up the technical live YouTube stream slash gather stream slash whatever, like two minutes before the hour when Bryce was supposed to start speaking. And it didn't take off until 9.23. And then...

Connor knows exactly. And Connor kindly timed my talk. So it took 56 minutes. It's like I never... Like these days, I don't really kindly timed my talk. So it took 56 minutes. It's like, I never, I like these days, I don't really rehearse my talk at a time. I just like sort of like intuitively know like what I'm going to say and like how long it's going to take. But it always surprises me how, like how on target I am for, I was like, this is roughly an hour of content and I clocked in or no, I think I gave him the estimate before the

talk. I think this is going to be 50 minutes or so. And I clocked in at 56 minutes. Yeah, it's quite impressive. Do we want to talk? I mean, we'll link it. It'll be online by the time this airs in probably a week and a half from today. It's June 13th.

This will probably air, yeah, two Fridays from now. Do we want to talk about empty spaces? Did you just come up with this in the last four days? It was a thing that I invented over the course of the past week or so. So I'll just let you sum up in the next five to ten minutes. Well, the basic premise of my talk, of the back half of my talk, is that iterators are a bad abstraction for multidimensional iteration and that maybe we need

a new abstraction and so the abstraction that I came up with is sort of like an extension of ranges but for multiple dimensions and so I'm calling it spaces and the idea is that you have a

you can call begin and end on a range but on a space you can call md begin or md end on it. And when you do that, you give it as a template parameter the extent which you're calling it upon. And then that returns you an iterator over that extent. And when you call md begin and md end, you can feed into it a tuple of stuff,

sort of arbitrary stuff. And the idea is that that is how the outermost loops pass things to the innermost loops. So like the outermost loop, you know, is just going to be like an iterator over maybe indices, you know, of ints. And so then you take that int, and then you feed it in to the md begin and md end call for the next innermost loop, and that means that that next innermost loop has access to that, to whatever iteration it's in,

the index of the iteration that it's in in the outermost loop, and so from that you can sort of incrementally build up the final index that your iterator is gonna, that the iterator in the innermost loop is going to return. And so I think it's a kind of clever little protocol. And I prototyped a library based for each equivalent for it.

And I imagine that we could have a sort of a space-based for loop, just like how we have a range-based for loop that understands this protocol. And I also think that it's a model that'll be composable too, because we could make it pipeable, we can make spaces pipeable just like ranges.

And the default would be that you could pipe in range adapters and those would operate on the innermost iterator but you can also imagine having some new adapter like call it on extents that would say hey I want to apply this particular transformation to just one of the particular extents. And through that way you can get to a nice compositional model. And I think that this model will let us get good code gen, but also have all the properties that we're looking for, like composability and an ability to write generic multidimensional code. And that's going to be rank agnostic and location agnostic and layout agnostic, et cetera.

Yeah, we should dedicate a whole episode or maybe two in the future to not necessarily diving into details, but talking about MD-SPAN because I feel like the direction that this podcast is going and my work's going and your work's going is like MD-SPAN is going to start to come up a lot more, but without diving into a complete explanation or high level view, uh, extent, um, cause that's something for a while. I didn't, I didn't get what that was. Would you like to explain to the listener what, what, what the extents object is or what an extent is both the latter though first. Well, an extent is just like a um you know it's an axis of a dimension you know um uh so like the you know the the c or if you have like a 2d matrix

you know it has two extents and each one of those has like a length to them right so so for our listeners and this is why i guess it wasn't necessarily confusing, but the first time I heard extent, it was just not in my lexicon. Yeah, I don't know why we picked that word. I did not know why they called it that. Yeah, we could have called it dimension. I think you would typically think of it as dimension, but there's a reason that we didn't do that. Colloquially, it's referred to as dimension, but in the array world and array languages, they refer to them as axes. And that's actually, we kind of talked about that one or two episodes ago um i think it was last episode which

actually hasn't been released yet well that it'll be coming out this friday but listeners already listened to it potentially a week ago and yeah so to rephrase it in like array language speak when you have a matrix aka a rank two array aka a two-dimensional array you have two dimensions or two axes the first and the second or zero and one and so in bryce's talk uh you'll see a lot of code that is you know has on extent and then zero one or two and it's very thought provocative because this is actually, and maybe we're going to have to bring Marshall on because I mentioned in, in,

in the most recent array cast episode, I talked about the evolving Connor and Bryce cinematic podcast, cinematic universe, because now we're going to have a panelist from array cast coming on ADSP. We've got my third podcast coming out, which I still haven't recorded my first episode for. So it's going to get complicated, folks. We're not talking Game of Thrones complicated, but

it is going to get complicated and there's going to be people crossing, you know, podcasts. We'll bring Marshall on. But the most recent episode we talked about on ArrayCast was how you only get a certain amount of access to drilling down using this leading axis theory

so you can operate on a certain rank but you might have to reshape your data in order to specifically work on a specific specific chunk and you then need something called a dyadic transpose so like a transpose of a matrix 2d matrix pretty straightforward but what if you have a uh three or n dimensional array that is more than just a matrix what does a transpose look like there interestingly it's like different across the array languages in apl and j i believe they just reverse the shape. So if you have a cube and the dimensions are 2, 3, 4, you just reverse it and it becomes 4, 3, 2. In BQN, they do a one rotate. So they take the first dimension and move it to the back. But more specifically, you can do like any

arbitrary reshape, which is what a dyadic transpose is it's a transpose that takes two arguments like your matrix and then this new basically like grade if you will so it's the reordering of the axes aka extents and md span speak and now you're talking about this like new potential model and they're completely different and i don't understand yours well enough but definitely you could do everything you can do with dyadic i think like they're completely different. And I don't understand yours well enough, but definitely you could do everything you can do with dyadic. I think like they're probably isomorphic, but admittedly array people, they talk about dyadic transpose as being like one of the most complicated things. And my question is like all of this complexity, obviously there's a certain set of problems

that necessitates you being able to do this stuff. But is this like, is this mostly driven by industry problems like an hpc you've got a bunch of stuff that requires this like whoo extent extent dot dot shape shape or is it like the hardware that you know and having to do like efficient algorithm implementation with tiling and stuff multi-dimensional um uh multi-dimensional spaces are logical, not physical in hardware. On GPUs, there's a little bit of a notion of that, but even that is actually physical,

is logical, not physical. And when I say physical, I mean physical in terms of how processes address memory. I do recognize that there are some types of memory that are actually multidimensional. But no, normally we think of memory as a linear store, typically. And so whatever multidimensionality we add on top of it is a logical layer. I guess, I mean, I'm not sure if that was, I asked my question correctly, but like all of this, like the complexity that you have to introduce in order to give the user a completely generic way to drill down to like any extant or axis of some multi-dimensional thing is that like i don't know i've yet to come across problems but

like i don't operate in the hpc space and i usually am writing libraries not using them i just don't know like uh you know the complexity comes from the applications yes it does i mean the the complexity of the like on the interface on like what you want to do the complexity and like how we implement it certainly comes from the hardware you know like as we discussed in my keynote today there's you know there are a number of sort of natural approaches to how you do multi-dimensional iterators in c++ which just lead to terrible code gen. But, you know, the application problem is like, yeah, they're naturally, they have this

structure where they have multiple dimensions. I don't disagree that the multiple dimensions exist. I mean, wanting to do things like tiling is sort of like performance driven. And that's what my thought is, is that a lot of the way that, at least, it doesn't necessarily have to be GPUs, but that's we work at NVIDIA i and that's what my thought is is that like a lot of the you know the way that

at least you know it doesn't necessarily have to be gpus but that's we work at nvidia and that gpus a lot of the times if you can translate your you know matrix multiply and some some sort of tiled matrix multiply you know the gpu goes nom nom thank you for the more food um uh but that's like that's not driven by that's not driven by some application-level problem. This is the way the hardware works, and therefore it'd be nice if we had a generic way to write the implementation of this algorithm such that it could be tiled super simply.

Yeah, yeah. I mean, if you look at the naive way that you write a matrix multiply versus how you write an efficient matrix multiply, they're very different, and they're very different because, you know, the way that you write to get performance is just going to be very, very different than the naive way because you have to think about

things like locality and memory access patterns. Oh, speaking of which, this is totally random and the listener won't get it, but those three diagrams that you had, row major, column major, and then what was the third one was? Morton order. Morton order. Did you pull those off of or did you make those? No, I made those. You made those? I made those. In PowerPoint i made those in powerpoint i'm in powerpoint wow very listener yeah you

should just go watch this keynote uh just for those is that the first time you've you've snuck that into or did you have that before so you made it specifically for this deck you know you know how i do do things like that uh you've seen my little red boxes i don't so so so what one of my secrets to powerpoint is like so arrows and powerpoints connect to shapes um but they don't always connect to shapes in like useful places um and so like sometimes like when you're wanting wanting things to be nice and aligned instead of just like having arrows and lines that you try to manually align, it can be really useful to make little boxes and put them in the right places.

Then you can connect lines to those, and then you can make those boxes 100% transparency, which is not 100% transparency is nice because it means that you can put them on the top layer so you can click them and get to them if you need to move them around but nobody can see them. And I always make these little boxes red so that when I turn the transparency back down again it's easy to see them. So on that little 2D matrix that I had where I put the lines of the orders

in the center of each one of those matrices is a little red box that I used to connect up the lines. Because, yes, dear listener, PowerPoint is my day job, which means I have acquired some PowerPoint skills. I don't know if we've talked about it on this podcast before. No, this is the first time I've heard it. No, the shortcut tools. I had this PowerPoint plug-in that I don't remember the exact name of it, but we'll put a link to it in the show

notes. But this plugin is amazing because it gives you a couple of very useful PowerPoint actions, like copy position of a thing. So instead of copying the whole thing, you just copy its position and then you can paste the position to put some other object the same position, which is really useful when you're trying to align things across lines. It also has key bindings for all of the alignment things because usually you have to go into a range and then click on one of those things. I never have to do that. I just got my key bindings.

I am a PowerPoint wizard. Connor's an Excel wizard. Connor's life runs on spreadsheets, as he just mentioned earlier in this episode. My life runs on PowerPoint. I don't think I'm your level, but if you're like a triple black belt, I'm probably like a single black belt. You did teach me today about the morph thing. Yeah. I mean, well, so what you just said you have keyboard shortcuts for.

I mean, this derailed. Actually, we're going to derail this derailment. I found that Reddit thread that you said when you mentioned that some people were saying you're too nonchalant. And so there was a Reddit thread that was entitled CPP cast is over. What what do we listen to now? And it was basically a 50% were recommending sort of the the newer C++, you know, plus because some of them talk about other things like ours. So, you know, ADSP, TLB hit, Two's Compliment, NDR. So those are the sort of the four. And then there was a bunch of other ones outside of C++. But then there was one comment that said like, yeah, you know, I really like ADSP.

But then there was like a bunch of responses, which in it said, yes, Bryce is too nonchalant. But it also said that the podcast. Yeah, I don't really care. See, it's just me being nonchalant. But it also said that the podcast... Yeah, I don't really care. See, that's just me being nonchalant. The other comment, though, that I just find hilarious is that it was the criticism that our podcast lacks structure. please go back to the beginning and listen when this podcast was inspired by Magic Readalong a 20 to 25 minute telephone conversation if you will between Dr. Boolean

and I can't even remember the other words because they would never even introduce ours has an intro little jingle theirs literally like it would just start hey man how's it going wouldn't say their names for like up to episode 40 I didn't even know Dr. Boolean because they never plugged their Twitter handles they never plugged for ratings which is why i basically said only once every 50 episodes are we going to say go to your blah blah blah

we're not going to do it now that doesn't count but the point is is like we modeled this pod i modeled this podcast off of magic read-along which doesn't even um record or release anymore but it was specifically the lack of structure that I liked. And so I just thought it was funny that they're like, yeah, you know, this could do with a little more structure. I'm not denying that our podcast probably could do. It's by design. But it's by design that it's not there.

So back to the PowerPoint, Morph Transition. Oh, no. So before Morph, I was going to say the things that you have key bindings for, I have the shortcuts. So you can add little icons. So it's still a single click. But, yeah, I have all those. Like basically the distribute evenly and horizontally, the align left, right, center, middle. Those are like bread and butter.

You have to have it. Yep. And the thing today, and this is probably the number one question I've been asked post-talks. It has nothing to do with my technical content. It's always like great transitions. How do you do the thing where you get code, one piece of code to transform into another? And in Microsoft PowerPoint 2019

and some later editions of 2017, but usually you just want to get 2019, they added a transition called Morph. Keynote, Apple's keynote, the PowerPoint clone, has had something like this called Magic Move, I believe, for years. Some people are going to take issue with you categorizing it as the PowerPoint clone. I mean, admittedly, didn't their software suite come out like years after Microsoft? I don't know. I just said, my point is just we're going to get letters.

All I know is Keynote is they got that one animation where the text falls down and then creates a sort of little dust cloud. And I'm like, oh, that's Keynote. Because that's what they always used to do. Or Steve Jobs, you know, they'd have some number and then it'd be 10 billion devices sold or whatever. Anyways, Morph was added. But the key thing is that some

people will try morph and every single animation and transition or most of them have like uh effect properties or transition properties and so you need to go and click on that and there's three different ones object words and characters and for code transitions you always want to choose either words or characters and the default is objects so it'll actually pick out the words that are you know consistent between two slide examples and i was telling bryce how does it do that that's really it must be a very very like and that's the thing is some people have criticized especially when it's apl because like it looks really nice when you have like c++ or python code and it transitions because no one's

going to be able to track you know exactly what went where it just looks nice but with apl sometimes you only have like four characters on the screen and then you transition but what it does is it it finds the closest word or character so like sometimes especially if you're dealing with a an array of ones and zeros and you're trying to like add 10 to it or something a one that really was being turned into a two will actually slide across and become the first one in the 10 and that that's not actually what you want. And you actually could have seen that in, I did a first 10 odd numbers when I went from one to 10, then two to 20, and then one to 19, the ones and stuff were sort of moving around. So it doesn't work perfectly, but it's basically like some kind of shortest distance algorithm to some object identifier. But do you think they pre-bake that? I don't think that they can. What do you mean

pre-bake? Like, do you think that they pre-bake that transition? Like they pre-bake that? I don't think that they can. What do you mean pre-bake? Like, do you think that they pre-bake that transition? Like, they pre-bake the compute of, like, what they're going to map to what, and that gets saved in the PowerPoint? Or do you think that it's done dynamically? I'm trying to even... Like, it has to be done, like, on the fly. Yeah, yeah. I mean, like, they could do it either way,

but why would why would you go to the effort of like saving it because you just have to update it every single time you change something on the slide it wouldn't make any sense i wonder if we can we could probably come up with some like some stress tests that would really slow powerpoint down for this you put a lot of words on the screen oh yeah yeah maybe i mean uh maybe. I mean, well, so we'll link in the show notes. And I think I've mentioned this before. What algorithm do you use to do that? I'm sure it has a name of just like shortest object, shortest distance.

But like there's no way that's like linear. It's got to be worse than linear. Oh, no, yeah, it is. If you have a complicated like it'll start to, especially if you have like a huge wall of text. Yeah. Characters are not too bad. But like as soon as you're doing words, words I think is actually a little bit trickier.

No, words has to be worse. A wall of text is just going to be bad. But I'll link in the show notes, the way I learned this was this crazy PowerPoint hacks. And basically what this guy was showing, the presenter, is that obviously if you've got a rectangle that turns you know from a purple rectangle to a blue square it can figure that out and it'll actually transition the colors and the shape but like if you're trying to make a star object turn into a circle well like those are actually two different objects but what you can do is group groups of

objects together and then you can make one of them like the primary there's some way to like you know you put a square but make it invisible and then take a circle and a star and then it'll it'll see that we have two bundles and they both include a square and then they will morph like the star into the circle and that like and he starts and he shows and so if you really take this crazy and then he builds this whole animation that looks like it was something designed in, you know, whatever, 2005 Flash. And, yeah, PowerPoint, like I said, I might not be a triple black belt. We're going to have to look up what algorithm you'd use for this, and we're going to have to talk about it. I would love, I mean, I said this back when we did the Excel episode, to bring some of the Microsoft. If we happen to have any Microsoft devs that work on PowerPoint or Excel or anything

in the office suite. I actually added a couple people on LinkedIn when I was at CppCon 2019 because I ran into them and they were like, oh yeah, we work on Excel. And then I just lost it. I was like, I love Excel so much. I used to have three versions. Microsoft, Excel people, call us. Yeah. We want to talk to you.

Yeah. So this became an episode mostly about PowerPoint. I don't think that this episode was mostly about anything. Yeah. Yeah. Should we wrap it up? We should probably wrap it up and go back to the conference. Anything else we want to say to the listener?

When's the next time we're going to be in the same spot? Oh, yeah. In a month. CPP North. CPP North. Yeah. Stay tuned. Who are we bringing on?

So we've got Tony's going to be there. We have to record a bunch of episodes between now and July because I'm going to be in London and then Tokyo. Yeah, we can do it. So like, Tony's going to be at CPP North. So is Titus. I think Chandler's going to be there as well. And Sean, parents, also going to be there.

Anyways, we've got a bunch of people that have been on our list of people to talk to. You said you want to bring Titus on. Yeah. I said I wanted to bring Sean and Tony. Maybe we could have a big, do you think Chandler and Titus would come on at the same time? Yeah, sure. Why not? Anyways.

Two for the price of one. I think Sean and Tony are the – actually, that was half the people I just listed. Yeah. Sean and Tony together are good. Yeah. Anyways. And maybe we'll do another live. We definitely got to do a couple live ones at Live's Ones.

Although I feel like what people expect from the live episode is not us sitting in an empty conference room, but like the atmosphere. Like we should go find some people. Well, so that's the problem is like we, this is an academic crowd. I know a lot of these people from having watched the virtual ones, but like who do we, we know Doug Greger here. Yeah, but we should wrap it up because this room is freezing and I need to be somewhere warm yeah it's surprisingly cold in San Diego yeah I'm wearing a like

I said a pink hoodie yeah which I will try to change out before we go to dinner tonight yeah we're not going to someplace cheap for dinner all right listener signing out from San Diego see ya soon

except not because we don't see our listeners we might see some of them conference season is starting that's true

that's true like I said like I just said yeah Sean and Tony definitely listen yeah so we'll be seeing both of them

anyways till next time till next time later thanks for listening we hope you enjoyed and have a great day