CppCast - Unicode for C++23

Episode Date: October 31, 2019

Rob and Jason are joined by JeanHeyd Meneide. They first discuss an ISO paper about the pros and cons of introducing ABI breaking changes into the C++ standard. Then JeanHeyd talks about the talk he gave at CppCon and his efforts to get unicode support into C++23. JeanHeyd also gives an update on his std::embed proposal. News ABI Now or Never Eliminating the Static Overhead of Ranges The Case for C++ C++ On Sea CFP Links CppCon 2019: JeanHeyd Meneide "Catch ⬆️: Unicode for C++23" Text for C++ Embed (Library) Embed on Godbolt Support (Patreon/GitHub Sponsors) Sponsors Write the hashtag #cppcast when requesting the license here One Day from PVS-Studio User Support

Episode 221 of CppCast with guest Jean-Huidh Manid, recorded October 30th, 2019. Sponsor of this episode of CppCast is the PVS Studio team. The team promotes regular usage of static code analysis and the PVS Studio Static Analysis Tool. In this episode, we talk about the pros and cons of breaking C++ ABI. And we talk to Jean-Yves Manit. Jean-Yves tells us about his efforts to get proper Unicode support into C++ developers by C++ developers. I'm your host, Rob Irving. You're my co-host, Jason Turner. Jason, how's it going today? I'm all right. Rob, how are you doing?

Doing good. You finally over that cold? No, not completely, but it's only been two weeks, and it'll be fine. Yeah. We were just talking about how a guest we had on a couple weeks ago, Jerry Ellsworth, just finished up her Kickstarter. Right. And met her third stretch goal right in the final minutes, I think, actually. That's awesome.

Yeah. It started off, they were looking for, what, 500,000, and it went up to over 1.7 million. Yeah. It started off, they were looking for, what, 500,000, and it went up to over 1.7 million. Yeah, and that third stretch goal means that it's going to include five more free games or something like that. That's pretty cool. I did not buy the Kickstarter, but I am definitely looking forward to that coming out

and hopefully getting my hands on it sometime. Yeah, I also did not buy it, but I did see tweets from several people and messages on slack from people who heard about it here and decided to go for it it's pretty cool i look forward to seeing how it is when it is more available yeah yeah cool well that's our episode like three piece of feedback uh this week we got a tweet from andrew saying hi robin jason i've been reading lots about abi lately and catching up on CPCon talks and enjoyed this video,

which is a link to a talk Louis Dion did on C++ ABI at CPCon 2019. And this post on Reddit, which is an article that we were about to go over in the news section, actually, and said it would be great to have Titus and or Louis to talk about ABI on the show, especially the challenges, benefits, drawbacks of breaking C++ ABI with a future release.

Love the podcast. Keep up the good work. And yeah, I think we briefly mentioned the C++ ABI with Nevin last week, but we've never had a show dedicated to discussing the ABI and definitely not talking about the potential drawbacks of breaking it. Right. Yeah, we haven't. And while I've got some more comments once we get to the news item too about this, but yeah, it's interesting topic. Yeah, I definitely think we should follow up with this and maybe do an episode devoted to it. Okay. Well, we'd love to hear your thoughts about the show. You can always reach out to us on Facebook,

Twitter, or email us at feedback at cbcast.com. And don't forget to leave us a review on iTunes or subscribe on YouTube. Joining us today is John Heid Benid. John Heid is a student at Columbia university in New York. Most of their programming is for fun as a hobby, even if their largest open source contribution soul two is used across many industries.

They're currently working towards earning their own nickname, climbing the academic ladder, while spending as much time as possible contributing to C++ standardization and development. Their newest and biggest project is Unicode for C++. They very much love dogs and hope to have his own in a year or so. They also like

TWRP's Feel Pretty Good from the album Together Through Time. John Hegid, welcome back to the show. Thanks, Rob. Earning your own nickname. Yeah, I forgot to actually mention your nickname. Yeah, so I mean, I'm just trying to make sure that I can become a PhD because I have the nickname The PhD.

So I can eventually get one so people can actually call me Doctor or not be like, are you a PhD student? I'm like, just working on it, still working on it. But yeah, it would be nice to formally earn that name amongst my peers and be able to kind of cash in on that promise I made for my future. So I think you were, were you already a student at Columbia when, when you were last on the show? Yes. Yes.

So I am taking, I guess a bit of a gap year basically to make sure I, you know, get a clean up a couple of things in my life and, you know, monetarily wise, I can go back to school and finish up the, the last part.

That's a, those gap years are interesting. Sometimes, you know, you accidentally end up starting Apple or something during that time frame. Yeah, I've got to be very careful. People have been warning me about this all the time.

If you get a job with a salary, you'll get used to living. You're not a peasant and all that stuff. You've got to be careful. You can't get lured in by the money and having an apartment and having regularly dispensable income that you can just like, oh, I want to buy a new embedded board. I guess I'll just go get one because I have the income to do that now

and stuff like that. Or things like, I haven't eaten ramen in six months. That's weird. Yeah, stuff like that. Very cool. How long are you on the academic ladder? So I just have one last remaining semester. Okay.

Oh, wow. That's great. After that, I have my bachelor's, and then I'm going to go for a master's, PhD program, hopefully in research, and what I was supposed to be doing which is visioning graphics but i haven't been doing a lot of lately mostly because i've been stuck doing like unicode and compiler stuff and all this other stuff that's like not at all related to what i went to school for and what i wanted to do so what is your actual curriculum that you're in right now so the curriculum i'm actually in and that i'm interested in is visioning graphics.

So running simulations, the typical illumination equation, BRDFs and all the other great stuff that comes with it. Also sound simulation, which is becoming a part of... There's been a lot of new research over the last year or two for doing physically accurate sound simulation. And so a lot of that comes from the same simulation you do in graphics, but you attach the physical properties that allow you to do things like, say, emulate the sound of what happens when you take a jug of water

and you pour it into a glass and you hear that sound. And the sound you get isn't actually the sound you get of the water flowing into the cup. It's actually the bubbles. So you simulate the bubbles to faithfully reproduce the sound of like you pouring water right and so instead of having to like use foley techniques to do that you just use like you know actual physical simulation and so there's a lot of research in that that space right now but that's you know that's that's all the the stuff out there that's not you know what i've been doing i've been watching with with great interest but haven't been able to participate for a little bit.

I just recently had a class of students who are physicists doing nuclear simulations, like what happens when a neutron or whatever is bounced around in this space. And I realized that's very similar to ray tracing. And it seems like that kind of physical modeling, like has lots more applications than I ever realized it did. Yeah. The space for what happens when you when you're trying to basically simulate, you know, whether it's whether you're trying to simulate like direct physical reality, or whether you're trying to, you know, do an approximation, whether it's like, like for ray tracing, or whether you're trying to do movie production, where you're trying to do an approximation, whether it's for ray tracing or whether you're trying to do movie production where you're trying to go for a specific effect or whatever else, there's a whole lot of space in the simulator.

Even for things like you mentioned, molecular interactions, but other physical interactions too that make this, even though it's called vision and graphics, it's highly applicable to a lot of other different fields. Okay. Well, Jean-Huid's highly applicable to a lot of other different fields. Okay. Well, Jean-Huid, we've got a couple of news articles to discuss. Then we'll talk more about the work you've been up to lately, okay? All right.

Okay. So this first one we referenced in that feedback. This is an ISO paper written by Titus Winters, ABI Now or Never. And he's talking about how we haven't had an ABI break in the past few releases. We're not breaking anything with C++20 either. And he's basically worried that if we continue not breaking the ABI, we'll never be able to. I thought it was an interesting article.

He doesn't have like, you know, there's this key feature that we avoided doing because we don't want to break the abi it's more you know he's got a list of much smaller things that we could do with an abi break um but he's definitely advocating this position of we should uh break the abi and uh not be afraid of it yeah so this was um i've actually been uh actually been I don't know if victim is the right word, but I've definitely been on the bad side of the ABI debate. I presented a paper

from Alex Christensen who works at Apple with J.F. Bastian and a bunch of other people. He presented a paper to take initializer list and make it so that you could move elements out of it, right? I am fully on board with that concept.

I knew you would be because, you know, I've attended your CS Plus Now talk about, like, how much standard initializer list sucks and ways that we could do it better or fix it and stuff like that. But basically, this paper was proposing that. Now, you wouldn't think it, but the idea behind the paper is that, oh, we'll just, we'll take the const off the initializer list like iterator returns and everything, right, and just make it so that you can return

non-const stuff, and so if you move out of it, it's just fine, right? The problem is, the problem with that is that if you were to have a library that takes an initializer list at an API boundary? So you have a function foo.

It takes an initialized list of, not unique pointer, but of std string, right? Mm-hmm. Okay. You have that function. You compile it with, say, this new code. Say this was approved in this window, C++20, right? And so we have this new initialized list. You can move out of it, C++20, right?

You take an initialized list on an API 20, right? And so we have this new initialized list, you can move out of it C++ 20, right? You take an initialized list on an API boundary, right? And you move the elements out of the initialized list into your vector or whatever, right? You then have an older library that's from like, say, C++ 14 or 17, that creates this initialized list, right? That creates this initialized list filled with strings, and passes it to, you know, this function. If you were to do that in a loop or something else, what would happen is that you would break the old application that passed you this initialized list of strings because your C++20 DLL or whatever moved elements out of this list, but the C++14 application wasn't expecting this, right? So it doesn't recreate the strings. It just keeps reusing the same old storage that you've already moved out of.

And so instead of getting the behavior where you have your strings copied into this thing, like, multiple different times, you end up in a case where you pass it once, it moves everything out, and then you're just copying, like, a bunch of, like, empty moved-from strings over and over again. Yeah, and so that's – and this is the kind of thing you run into with ABI. So long story short, that paper died. That paper died in San Diego, which sucks. Because it would be an ABI break.

Yeah, because it would be an ABI break. So if we implemented this paper, we put it in Compile Today, we wouldn't be able to reconcile the fact that we have all these initializer lists that depend on the fact that it's copies only, not moves, right? And we just like totally destroy all that code. And that's, you know, that's the problem, right? And you would think that doesn't affect a lot of things, but I've, you know, started to keep my ear to the ground for a lot of different libraries, like libc++ and libstdc++ was the GCC standard library. And just like the number of things that affect ABI

is just like astoundingly high. So for example, there was a change that somebody wrote for, I think it was one of the iStream iterators, right? And so there's a difference between the size of the struct that derived from these things in C++ 11, 14, 17

versus the, or 11, 98, 11, 14 versus 17. Now, this bug kind of wasn't detected for a very long time, and Jonathan Winkley only found it now, but the fact that we can't change the size of this, like, struct thing means that anything that you pass through an API, anything that you pass through an API layer needs to be stable. So Jonathan Wickley couldn't make the change that he was going to make or that had already been pushed to the code base because that would change the size from being 24 to 16. And because of that, that would change the way GCC passed it on the stack to the function call, right?

And by doing that, that breaks ABI, right? And so it's just all these little things, right, that just, like, completely destroy, like, the whole way that old code that, you know, say you pulled it from a, you know, like, apt or something, right, on Debian or whatever, or you got some precompiled code and you put it, you know, on your path or whatever to work with it in Visual Studio, all of that code and all those things break in the face of ABI changes. And so these are the kinds of things that I think that Titus is talking about in his paper, right? And there's a lot of things you can do to get around it,

but there's also sometimes not a lot of things you can do to get around it, but there's also sometimes not a lot of things you can do to get around it. And it does accumulate a lot of performance penalties. It does accumulate a lot of fixes and changes that you just can't do. And it's incredibly limiting to have to stare down the barrel of ABI for C++. And to the irony, if I understand it reading this article, is that the standard doesn't technically say anything at all about the ABI and the standard committee doesn't have any specific direction. They just kind of quote, avoid doing it. Yes, exactly. And so what happens is, is that,

you know, people can propose papers and a bunch of different ideas and the committee can say like, oh yeah, we like that. But then what usually happens is an implementer looks at that paper sees the implications of an abi break and says oh no this is an abi break and then everybody votes it down um i actually had that feeling right i was talking about that initialized list paper i was in the room like i know i had a presentation everything i was like i was really selling it i was like you know this is going to be great it allows all these different code you know it makes everything better right and everybody was like people were nodding their heads like i was feeling good and then one person raised their hand and says this is an abi break and just the whole thing was just hosed from there wow yeah

and it's it's just and it's just it's it's it's it has that level of power right but we don't talk about it there's nothing in the css standard that says this is you know the layout of a structure blah blah blah right we have that at a high-level abstract, non-deterministic, parametric virtual machine level, but we don't have that on the level of what people actually work with, which is what compilers compile, what ends up in code bases, what ends up in distributions, et cetera, et cetera. So what really happened is that a lot of people don't even know that ABI is a thing, or they don't know that they're even depending on it.

There's so many Linux distributions that had no idea that this was a problem. We changed std string. The biggest example is the std string change in a GCC. So in C++ 11, they had to ban the copy on write implementation. So they make that change, and there's just a whole bunch of people who have no idea what's going on. They start, they're using their strings, they recompile their code, right? They use their strings like normal, and then everything goes to hell because, you know, you're calling std string, but with a different std string that has like a different

layout, right? A different memory layout. And so you pass these DLLs, these compiled, right? Like shared objects are like the basis of any Linux machine, right? So you have all these system components, right? These very have all these system components, these very important critical components that are taking a std string that are just broken now, completely irreversible. The thing links fine, et cetera, but it just breaks. So this is the thing about ABI,

that it's very expensive and very costly to break it if you have users who for a very long time expected you or just implicitly relied on the fact that things just magically worked. Yeah, the Titus's paper here says that he conservatively estimates it would cost engineer millennia to do an ABI break across the C++ ecosystem. Yeah, he's still advocating for it. I mean, he kind of puts down here at the end that he sees three paths forward. One is we kind of say we will do an ABI break and we will be explicit about it. We'll plan for it. Two is we'll continue, we know, the current course, which is, you know, people make a present, uh, presentation like you did John heed, and it'll probably just get voted down when someone realizes there's an ABI break. And I'm just kind of curious where, where do you fall, um, with these three options, John, he does someone who goes to those parents meetings.

Oh boy. Um, so, so recently I've started to get into contributing some stuff to Libstd C++, and I might want to do some stuff for Lib C++, and et cetera, et cetera. And in watching the bug tracker fly and all the issues related to ABI certain things and things like that, I've gained quite a degree of sympathy for stability. But I don't use C++ for stability. I use C++ because it's the fastest thing I can use, because I can make great abstractions that perform at the same level, if not better than handwritten code, right?

Like that's everything I'm about, right? So for me, I'm absolutely on option number one, which is decide that there's going to be an ABI break, let everybody know beforehand. Give everybody ample. I think one of the biggest reasons that this was a problem was because nobody saw it coming. Or the people who did see it coming weren't able to effectively communicate with the community. Listen, this is going to break.

And this is also kind of part of why number three is a really bad choice to make, because just having that silent assumption that everything's going to keep working is a severe problem, because you'll never be able to communicate to people who make new proposals or, they come to the committee, like, I made this great proposal, right? And people are like, oh, yeah, that's a good idea. And then an implementer shows up and says, that's an ABI break. And then like the paper stops, so they have to go backwards, right? And so there's a cost in if we sit around and don't make any acknowledgments or don't make any choices. If we pick option number three, we end up in a really bad place where we'll be wasting a lot more of everybody's time. Option one, of course, is what I want because performance and speed and all the other good stuff, right? I always want to be able to make changes to the things that I've written. If I was stuck, say for a library like Sol 2, if I was stuck on the version one implementation,

there's no way I would have been able to beat the performance of every single other library out there if I was stuck on the version 1 performance for ABI reasons I absolutely broke ABI to fix things and I expected that people would recompile their code or whatever else and I'm lucky because my code would be used in an application layer not necessarily like a middleware

or library layer so in that case I was kind of lucky that I could make an ABI break and nobody would really complain. But I can definitely sympathize with the option number two because I've seen the effects of an ABI break. And it also just makes people a lot of money. If you pick option number two, right, like Corporation X will know that,

okay, well, I can buy a contract for 10 years and everything is going to be great, right? I think that's, you know, obviously this is hearsay, but I think this is probably why, you know, Microsoft kind of doubled down on picking an ABI. I think, what was it, Visual Studio 2017, that release is the one that they made ABI stable. And they have like a huge number of bugs they can't fix in later versions because they decided to double down on ABI stability. I think they're planning to break the ABI, I thought, though. They might, which would be great. Oh my god, I hope they do.

But, you know, this is the kind of thing that you have to deal with, right, when you start trying to pick a side on whether or not you want to break ABI. Of course, I mean, I'm always number one because I use C++ because it's fast, right? Because I can get more performance, right? Because not only can I get good designs,

but those good designs, like after the optimizer is done with them, are just as fast if I had just handwritten a bunch of code, right? And that makes it very easy to write elegant abstractions. Maybe not with compile time, maybe not with other things,

but definitely scale with developer time. That a developer can sit down, write out something quickly, and know that it's going to be just as fast if they had spent a whole week writing hand-optimized, you know, perfectly crafted, you know, you know, artisanal, yes, artisanal assembly code, right? And that's, you know, that's a really big sell for me for CSOS, right? And if we pick number two, and I think Titus went into the paper, right? And that's, you know, that's a really big sell for me for C++, right? And if we pick number two, and I think Titus went into the paper, right? If we pick number two, right, we admit to ourselves that there is room for another language beneath C++. There is room for something else to sit, to take the place of C++, right? And whether that's Rust or,

you know, something else, you know, that's just, that's just whatever the ecosystem decides. But if we do that, we resign ourselves to go eventually to the way of like Kobold and Fortran 77 and things like that where, listen, they're used in banks. They're used for X big industry. It's infinitely stable because it can't ever be changed or improved. And so is that the future we're looking for? right there it's an it's infinitely stable because it can't ever be changed or improved right and so right is is that the future we're looking for we're looking for a future where we continue to be something that the new people pick up right and that's that's the contention so you know you mentioned soul 2 or soul more to the point and i learned like super quickly with chai script

as a library like that's pointless for me to distribute binaries. Because as soon as I did, people would say, oh, your binary crashes on my system. I'm like, yeah, you just have to recompile it for your compiler. Like, that was just, it ended up being like quickly a given to the point that I'm like, like, I don't even understand why people are expecting binaries for C++ libraries to work. And I read, let's see, it was Andre Alex Andreescu's and Herb Sutter's C++ coding standards book in like 2005. And I don't remember the exact words that they said, but my effective takeaway from it was never put C++ types on an ABI boundary, because, you know, it can't work. And I took away from that. And I said, Well, that's dumb. What's the point using C++? Instead,

I'm just going to always recompile everything. And then I know that it'll work. And I've been doing that for like 15 years. So I'm kind of surprised that there are people that are like, surprised when it doesn't work, I guess, maybe, I don't know. Yeah, it's, it's, it's, it's sort of like a, it's, it's two different universes, right? If you're on the side of like, open source, and etc, etc, right, it's very easy to get into a system where like, you never get into a system where you don't even think about this stuff. It's like, oh, just recompile. You change compiler flags? Okay, just recompile it. It just makes sense.

But when you're talking to proprietary people who have proprietary source code or people who, God forbid, they lost the source and they only have the object files around and they're like, I need to link with this thing. Lost this. I never even considered that that would be a universe. It is. It's real.

It's scary, but it's real. And, you know, it's like, you know, they need to like, listen, this needs to work with a C++98 version of the string and needs to work with X and Y and Z and it had better link and run because we are not going to invest the money or we're not going to take care of our technical liability

here. We're just kind of going to walk it over and, you know, make sure that every, all the code we write in the future just magically, you know, links with this stuff, right? And so it's that kind of tension, right? It's the recompile the world tension versus the I don't own the sources tension, right? And there's a lot of companies whose whole business models are predicated off of I don't own the source. So it's a tough call to make. Well, I just, for the record, just did a quick calculation. And if Titus is correct, that it would take engineer millennia to deal with a large ABI break, and there's approximately 30 million C++ users,

that means we would each spend 0.02 days dealing with this. I am willing to do that. Yeah, I'd be willing to do that. But the reality of that kind of thing is that what happens is those people spend 0.2 days dealing with it and then they file a bug report on jonathan wakely and then he spends the next five five weeks dealing with it right and and abi tagging everything and so it's um it's it's i'm still on this side of number one but but ah stability and just like uh you could be fast or you can be stable or you can i I don't know.

I don't know. Whatever we pick, I don't want to go the way of, I don't want to be Kobol. I don't want to be, you know, I don't want to be the year 2000s Kobol. I don't want to do that. No, no one wants that. Okay. This next article we have is from Vector of Bulls blog, and it's eliminating the static overhead of ranges. And this guy writes really long

blog posts um but in this one he's kind of going over uh the pipe operator with ranges and introducing his suggestion for kind of an alternative to the pipe operator do you have a chance to read through this one john he'd oh yeah so i read this the moment it came out actually um so i was like i was super excited about this article because I've noticed that there is overhead, right? When you're using ranges, when you're using whatever else, especially because I'm working on some new Unicode stuff that's supposed to be rangy and whatever else.

And obviously the compile time cost is pretty heavy. And so when I read this article, I was really excited of the compile time cost is pretty heavy. Okay. And so when I read this article, I was really excited because the way that he specifies this feature, this, I don't know what to call it. It's like he's got the line and then like the... Pipe arrow. Pipe arrow or like, I don't know, boomerang-ish.

It's kind of like a boomerang operator. I don't know, like top hat. I don't know. boomerang ish it's kind of kind of like a boomerang operator i don't know like top hat i don't know um basically he's got this operator here um and all it does is just it just rewrites the function call um and i really like this because it it it takes no boiler plate right i have to write zero boiler plate to make this stuff start working with with like functions right i just write i write functions it rewrites the thing, it calls the thing, and it does what I expect it to do, right? Like, and so now, like, obviously, there might be like a couple corner cases where it becomes a little bit more difficult. But like, the like, this, to me, is an absolutely elegant solution. Now, I don't know if, if, you know, this will fly in the committee. Because I know it's any given day, it's hard to tell like what, what the committee wants out of something.

But this is like a very like nice clean way of making it. So you don't need to start, you don't need to, you don't need to make, you don't need to really define like callable, what do they call it? CPO object, callable customization point object,

sorry. And a bunch of other stuff that's like in the standard, like to support like this, the, like having these pipe operators and everything else and making it work the way you want it to. And so I'm really happy with the article and the solution it presents and everything else. I definitely want something like this. There's also, I believe, what was it? Isabella Morto also wrote a proposal um for some of these operators but i think uh vector of bulls um the way he writes

the operators is different from the way that uh isabella moreto's paper which is in isabella's paper it's literally just like an operator you can overload um so of course you might need some boilerplate and other things to make it work but but in the case of Vector of Bool, it's literally just a syntactic rewrite. So maybe easier to implement, maybe easier to deal with. I think you could get the same benefits from what Vector of Bool talks about here in his Eliminating the Static Overhead article by using Isabella Moratis, but that's a lot of template instantiations of an operator to do the same thing, where this looks like it's baked directly into the compiler um which is

infinitely faster just just throw that out there it's it's in if you do something in the compiler versus through a bunch of templates and operators it's always going to be infinitely faster maybe i'm missing something but i think there's a missed opportunity here to allow it to also rewrite dot call syntax. And then it could be a backend way to having universal call syntax opt-in ways. You mentioned the proposal, you know, the one that people try not to talk about. Yeah, but we're not doing that proposal. This is an opt-in. You could say, S, do this thing, and it's either the first parameter if it's a free function or it's the left-hand side of the dot if it's not a free function.

And I think that would be kind of a nice way to do it. Although overload resolution. Yeah, overload resolution becomes fun, right? How do you determine whether the member function call is better than the free function call and stuff like that? And so it gets hairy pretty quickly. But, I mean, you know, let the record show that Jason said it, not me. He brought up the universal function call syntax. But, you know, I mean, that would be something that you could do, yeah, for this.

But I'm kind of, I'm still kind of just more on the foot that I like the fact that it's just a rewrite of what pipes would be, and that it's a very elegant and nice rewrite of what pipes, of how pipes work in ranges like that.

Okay, so we have this last article, or second to last article, The Case for C++. Yeah. And this is written by John Murray. And he's basically kind of going over what's been happening in C++ over the last few years. If you're a listener to this show,

there's nothing really new to us here. But this article is definitely aimed at people outside of the C++ community, just letting them know what's been going on in C++ and that you shouldn't be afraid of this language if you were previously. Oh, and we get mentioned. Oh, are we? I didn't see that. We are an amazing weekly podcast. I didn't see that. I did see that mention the include C++ group. They go over tooling and analyzers. So definitely a nice post, but, you know, kind of aimed at developers using

other programming languages. Yeah. And then last thing is C++ on C has its call for papers open, right, Jason? Yep. Call for papers open until the end of November, I believe. And the link that I have in here doesn't actually say when it ends. There's a, on the main cpponc.uk website tells what the dates are. So get your papers in. The conference still isn't until 7th of June, but Phil's getting things started early this year. So yeah, end of November is when the call for speakers ends. Okay, great. I might have to actually submit for that one then. Yeah, so I guess for our listeners, and for you, John Heath, if you didn't know this,

this is one of the conferences that does pay your travel expenses. So yeah. And June in southern England should be much nicer than February in southern England, as it was last year nearly completely fogged in for the whole conference, which, I mean, you know, has its own certain British charm to it. But being able to see the ocean would have been neat also. Yeah. Like it was physically impossible to see the ocean for much of the time that I was there because the fog was too thick. I guess it was kind of like an atmospheric.

Maybe that would have been good in Halloween time. It was like the atmosphere, like there's fog everywhere. You'd hang up some lights and say some spooky sounds while people walk into the conference or something. Well, and honestly, walking to and from the conference venue in the morning and evening with the darkness, with the uh with the lighthouses and the thick fog through it was actually i mean it was a special moment you know uh but it would be neat to see the place as well yeah yeah the sponsor of this episode of cpp cast is the pvs studio team pvs studio is a tool for static application security testing.

It is designed to detect errors and potential vulnerabilities in the source code of programs written in C, C++, C Sharp, and Java. For those who have repeatedly heard in the podcast about the tool, but never tried it, there is a reason to do so. When requesting a license, write the hashtag CppCast and receive a trial license not for a week, but for a whole month. This will allow you the time to get a full picture of the analyzer by using it to check your personal and work projects. If there are any questions about the tool, don't hesitate to contact their team for support. By the way, on the topic of support provided by the PVS Studio team, there's also a link to a small amusing note in the description of this podcast episode. Okay, well, Jean-Huid, you gave a talk at CPCon this year on Unicode, and we wanted to ask how that talk went.

Oh, boy. So that talk actually went really well. I think I was able to kind of not only illustrate why C++ and C are really bad at handling Unicode, but also kind of present an API that I think a lot of people definitely agreed with was the right way forward for C++. And obviously, that's hard to do in a talk, so I think I did a decent job. I had people

from... STL was there, and he actually came up to me and he said that was a great talk, and he actually saw the design merits and everything of the library. At a moment, I was standing up there, and I actually came up to me and he said that was a great talk, and he actually saw the design merits and everything of the library. And so at a moment, I was standing up there, and I had my suit on and everything, so I had to pretend I was calm, cool, and collected. But I totally squealed when nobody was around that STL came up and said he liked my talk. So that was, oh, man.

That's great. Do you think STL listens to this podcast? Because if he does, I think I just kind of blew up my spot. I don't know if he does or not, actually. I'm not sure. Okay. Well, hopefully he kindly skips over this part of the thing. I don't know.

It'll be fine. Yeah. But I really think it went really well in that I was able to very much sell the infrastructure and the design and what it was all supposed to be doing. And so now what I've gotten into, after having finished the talk and everything and gotten pretty good feedback and all that, I've basically started moving forward with an implementation. Because this is the kind of thing that covers so many different use cases and involves so many different kinds of people that you definitely don't want to like, this isn't something that you can bring to the standards committee and be like, well, time to standardize this, you know.

Yeah, let's get it done. You know, I need to produce an implementation. It needs to have, you know, it needs to be pretty sound, right, you know, at least in this design aspect, right, performance and stuff. Like, I always do stuff that's high performance, so, of course, I'll optimize the hell out of this. But, you know, I need to make sure the design is sound and everything else. I've gotten, definitely I've gotten good feedback from people who work at Mozilla Firefox

or work for the Mozilla Corporation, sorry, but who work on Firefox. I've got feedback from people from Bloomberg and a bunch of other people who have seen the proposal, have seen the talk, and are pretty much like, yeah, this is definitely the direction that we should be heading in for Unicode for C++. So that was pretty exciting. I think it's worth asking, since you just said your actual love and interest is in this physical modeling world,

but you spend all of your time on an open-source scripting engine and now Unicode, why the interest in Unicode? So a long time ago, I did a little bit of charity work, and I helped somebody reimplement text encoding in their code base because it was really bad and sucked. So basically I went in and I implemented basically what my proposal is today. I basically implemented that with a couple differences. But when I was implementing that, I realized that there is a wide body

of just a complete lack of being able to properly handle encodings. And at first I thought it was like, oh, okay, but there's ICU and there's other stuff like that. But then I take a look around the world and I see the number of things that just completely fail when it comes to handling encoded text. Things that are not web-browse. Obviously, web browsers have their stuff together because they needed to have their stuff together. But I have a Russian friend.

His online handle is Elias Dallaire. And I think he also is one of the people who advocates for Sol2 and brought it to CEP Russia and other places. But he showed me a wiki link to a Russian site. And they have a site that's dedicated. Here's a phrase and here's all the different ways it can be misencoded. And this is what the Mojibake looks like. And he's been able to talk to me. And he's like, some people are so good at, like, seeing, like, badly encoded stuff come through that they can identify what this person is actually saying because they can read, like, the mojibake that comes through.

And that appears on, like, their terminals in other places. And so it's this wide, like, widespread, prolific, like, problem. And, like, the problem is I don't see anyone, like, fixing it. Like, I see people fixing it for, like, their specific use case or in their specific style. But I don't see anyone fixing it. I see people fixing it for their specific use case or in their specific style, but I don't see anyone fixing it for the entirety of the ecosystem. And there's definitely been attempts, right? And I think one of my favorite attempts so far was a C++11 library called LibOganek, written by Robot.

And for those of you who don't know who Robot is, he's the one who actually made and coined the term rule of zero that eventually Scott Myers and a bunch of other people use in their talks and books and et cetera. But he designed a library called the Boganek. And that library laid a lot of the foundation of what I thought was a good text design. The thing I had implemented was sort of like that, but missing a lot of crucial pieces that I think LibOganek filled in. And while I was kind of going through this process of seeing, I think the only problem really with LibOganek is that it's kind of not maintained. So there's a bunch of extra encodings and a bunch of performance optimization, there's stuff that probably should need to get done and have to get done.

And there's also just, and I mentioned this in my presentation, but there's stuff that probably should need to get done and have to get done. And there's also just, and I mentioned this in my presentation, but there's also small design problems. Not necessarily with Oganek, but with the limitations Oganek had. So Robot, the guy who made this, he's way ahead of his

time, like 500 years ahead of his time. He's the one who can do all those ABI fixes for us then. Yeah, probably. But he coined the term rule of zero. He wrote blogs on tuple packing before people even did tuple packing or know how to do it well. He did a lot of template metaprogramming before I even knew other people who even talked about

it. And he was also the person who, you know, who made like ranges and like good ranges stuff before like ranges were a thing. In fact, I think that's the only weakness of LibOganek is that it tried to do ranges before like C++ or C++11 or even 17 caught up. And so it has like all these issues with like dealing with like really fat iterators like because of the amount of information you's a store and stuff like that so there's there's a bunch of different like things that like like it wasn't a limitation of his design there's a limitation of the language right um but like his design is that good right that like once you take

his some of those concepts right and you transplant them on like a c and you rebase them on a c plus 17 or c plus 20 like capabilities, you know, his design like completely holds up, right? And that's like fascinating that that worked. And, you know, as I kind of went through, you know, I saw that there were a lot of people who kind of solved this stuff internally for their own companies. So I know a couple of Fortune 100s that have the design that I have, right? But it's internal to them.

And even when you work with things like Qt and everything, they have a very specific way of handling certain things that make it very, very difficult to port that outside of the Qt framework or handle it outside the Qt framework. Sorry, it's called Qt. But nobody solved it for everybody, right? Lib Oganek tried, right? They ran up against the limitations of language and they stopped, right? You know, Boost.txt is trying to solve it, which is Zach Lane's project. But he's very opinionated.

And so he absolutely made great progress because he was able to hammer down, like, oh, I'm going to pick this normalization form for you and it's always going to be stored in UTF-8 because that's the way the way the truth in life and and stuff like that um but you know even on like the the thread where he's proposing to put this into boost right there are people who are like i'm sorry i can't use your your your framework because i i standardize on normalization form d and i need normalization form d for my for my

tech processing and stuff like that right so really when really when I, when I made my, when I, when I, when I originally thought like, I'm going to see if I can like spend a year off working specifically on Unicode and C++. Um, I really just, I'm thinking of like all the people who have all these requirements that like current solutions aren't meeting and realizing that it's kind of a super fundamental problem because like, we can't like it's C++, like++ and C like the like the lingua franca systems programming languages, right? That everybody uses to talk to their embedded devices and networking and all these other like different applications, right? Like these things can't handle like other like not English, right? And that's like really bad um you know and and so we're living

in a world where like fundamentally like we have assistance programming language that's used everywhere but doesn't accommodate people's languages and cultures and like that to me is like that's like super unacceptable to me right like you know when i worked on soul 2 like it was kind of still low stakes because it's like eh you know somebody doesn't like it right they can make their own lua framework or whatever else, right? And stuff like that. But this is fundamental. We should have solved this.

I'm actually kind of a little bit annoyed that I'm working on this problem in 2019 because I shouldn't be working on this problem in 2019. This should be a solved problem. We should be working on all the other stuff, right? And so really, if you want to see something done, you have to do it yourself, right? So I'm going to do it myself.

I'm going to make a design that's flexible, that works for different code base, that works for both companies and individuals, that make it so that people can work on the systems level, right? And then on a systems programming level, you can have something that supports your languages, your use cases, and things like that. So you sound like you're being pretty aggressive about this plan.

I mean, are you hoping to get Unicode support in C++23? Yes, absolutely. I do not expect this to take more than three years, and I don't plan to work on this to take more than three years, which is hyper-ambitious. But for C++ 23 specifically what we need is a good transcoding solution, going from UTF-8 to whatever or going from whatever the magical

locale based encoding of your system is, whether it's like Windows 1252 if you're on a Windows machine or Shift JIS if you're in Japan or EUC JP if you happen to be or EUC, JP, if you happen to be on an IBM machine with a specific locale set

and you're using WCART, it's a mess. I need to fix transcoding, right? And on top of when we have transcoding, right, when you can get from the implementation-defined black box, that is whatever your string literals and whatever encoding comes from the system is, to a well-defined encoding like UTF-8, UTF-16, UTF-32, or whatever else it is, that is what

will enable you to basically go from there and stack all the Unicode algorithms on top of that, right? Because you'll have a well-defined way to go from whatever your encoding is to Unicode code points. And once you have code points, you unlock all the rest of Unicode, right? And so for C++23, I'm absolutely dedicated to getting a solution to transcoding, right? To

being able to go from some blob of text to UTF-8 or UTF-16, UTF-32 in the standard. And that is pretty much going to be my mission for the next six months to a year. To help with that, I've set it up. You know, I already have a Patreon, but I basically also on GitHub sponsors, and I'm going to be reaching out to companies and individuals, trying to see if I can get a small degree of sponsorship so I can really work on this full time and get it done and basically solve this problem for everybody in the community. Because it really is a shame that the primary systems programming language just can't handle

fundamental things that most people would just kind of expect out of the box right now. C++ is the only production-level language that doesn't solve this problem, which is not only is it kind of a shame, but it's also just really bad, especially with the use of C and C++

in so many different places. Hopefully I can get sponsors, and people will contribute, and this will be a thing, and so I'll be able to work on this full-time and solve it, and, you know, I'm really looking forward to kind of tackling and solving this problem, and kind of like making it so it isn't a thing where we have

to be like, oh, well, you know, how do I support Unicode in, you know, 2020? It's 2020, and I still don't know how to do Unicode stuff, right? So really, by the end of 2020, I want the implementation out there. I'm going to be brigadiering with the papers and all that other stuff. So how are you getting interest so far already on your

Patreon and, you said, GitHub sponsor pages for this project? Yeah, so I have some interest, and there are a couple people who sponsor me, but they've mostly been individuals with like $5, $1 donations, stuff like that. That doesn't pay the mortgage, no. No, it doesn't. I have reached out to a couple different sponsors.

I've had some interest, but then some fell off. I have a couple additional corporations that might be interested in this. So I've sent out emails, and I'm kind of just like waiting to see what they say. But I definitely haven't been doing a lot of the advocacy besides what I said at my CPCon talk. So this is kind of serving as another like advertisement that if your company is interested in having Unicode solve for all of C++, and not only C++, but also C, then please reach out to me.

I'm absolutely looking for sponsors. I'm absolutely looking for people to help me tackle this problem and get it done within a timely fashion. And hopefully this is something that we can get done for both the C and C++ world. Awesome. We'll absolutely put a link to the Patreon in the show notes. If someone wants to contact you directly, like to get their company involved, how would they do that? So they can, I mean, they can use it in any way they want to. But usually the primary way is through email. So that's phdofthehouse at gmail.com. But if somebody even sends me a DM on Twitter, I'll be more than happy to respond.

But yeah, definitely via email. If you have my resume, you can probably send me a snail mail letter as well. But I don't know why you'd go that route when email is available. But yeah, absolutely please get in contact with me. Please reach out. a snail mail letter as well, but I don't know why you'd go that route when email is available. But yeah, absolutely, please get in contact with me. Please reach out. I'm absolutely interested in making sure that this problem gets solved within the next year so that we can have a solution.

Not just for C++, but also for C. I attended the WG14, that's the formal name for the C Standards Committee. They had a meeting in Ithaca, New York, actually, this past week. And so I attended and actually presented a paper to fix some of the transcoding issues, not only at the sea level, but at the sea level. And I received, well, I can't say that I guess the actual straw poll numbers. But nobody voted against it. So I have pretty much not only support to have the old null terminated style of conversion functions, but also to add sized ones that actually take a size parameter, which is very important because with a size parameter, you can safely SIMD through an entire range without having to check for a null terminator,

without accidentally running over the edge of a buffer and creating a security vulnerability. So I was very glad that they voted both the null terminated and the size versions in, because that enables performance in a bunch of other stuff. So I'm really glad that even the C committee recognizes it, like, yeah, we need to solve this problem, and it would be nice if we do this sooner rather than later.

So that's pretty cool. So you mentioned again GitHub Sponsor. Do you currently have a GitHub project up that people can look at? Absolutely. So if you go to the, I believe it's just github.com and you go to my name

there's a little button with a heart that says Sponsor and you can click on that and it'll take you to my Sponsor page. The Sponsor page has a full list of all the things I'm working on, but it also has a link to my portfolio on text that lists not only how much I've accumulated as far as cash flows, but also lists in explicit detail, these are all the things I'm going to do given the funding to get them.

These are the stretch goals, things that are extra, these are the things that are going to do, give them the funding to get them, right? And these are the stretch goals, things that are extra, these are the things that are core to the fundamentals, right? Potential backporting to CS plus 11 if you still need a solution for that, that kind of thing. So it's a very full listing and offering

of everything I'm going to do. And so I plan to have this library obviously be open source, to be given to most people, to be given basically freely available to most people. And so I plan to have this library obviously be open source, to be given basically freely available to most people. And so the whole point is to make it so that this gets done as fast as possible. It's open source as fast as possible. And that everybody can kind of benefit so that we can stop working on transcoding utilities and start working on the really fun stuff, which is like, I have an application that needs to do X. I want to work on X, not Unicode, right?

Right. Stuff like that. Okay. I wanted to go back to what we had you on originally a year and a half ago. I think we're mostly talking about your stood-in-bed proposal. Have there been any updates with stood-in-bed since then? Oh, boy.

So stood-in-bed was... The reason I wrote stood i wrote student bed in the first place right well even if i'm even remembering back to when i was on cdpcast before what i had said was oh yeah this is like the non-controversial simple easy you should get through in a couple in a meeting or two paper right uh not the case this is i think so I think this is going to be by virtue of my first paper this is also my longest running like longest open paper and it's still open I ran into

some snags right where what basically happened is that in after Rapperswil after the Rapperswil meeting last year in the summer a bunch of like the build people and the tooling people like were upset at the state of modules and how that was all going. And I think you remember some of the articles

and the Reddit fires that started and stuff like that. But basically, they showed up at San Diego, right? And what they had basically injected into the consciousness was that if you make things hard for tooling, that's going to be like a hard no, right, from them. And std embed kind of got caught up in that, because what std embed is, it's a like a const eval, like compile time function that like takes a string view, and then like loads up some resource, whatever, right, which sounds simple enough, right? I mean,

everybody wants something like that, right? But the thing is that because it takes a string view, you can compose an arbitrary string at compile time, especially since in C++20 we're getting constexpr std string. So you can bake up a string at simple like pre-processor scanning tool to like get all the dependencies out of like the all the hash includes or whatever else right um and just know a priori or a priori a priori um that like that's all your dependencies right for this for this like translation unit right um stood in bed makes it so that you need to wait until after the compiler finishes doing basically uh up to semantic analysis before you know what it actually is going to depend on to, like, compile the file, right? So basically what's happening is that you have to run the full compilation, basically accept the code gen step to definitely know, right, this is, like, all the stuff I'm going to need out of this translation unit with std embed. And that was a hard no for some.

And so what happened is I kind of went back. I was like, okay, well, maybe if I introduce a preprocessor thing. And then so I said the preprocessor word. And what happened is a bunch of people said, over my dead body, macro-like stuff? Are you kidding me? Absolutely not. And so student bed got stuck in this really hard place between people hate the preprocessor solution, solution but yet some people love the pre-processed solution because well you can only pass string literals to that and my my you know crufty old dependency scanner can like read that string

literal know exactly what you're trying to put in right but for other people that's like that's like a hard no because like if you've ever written a shader past like 1995 then you'll know that you have like include statements and you need to like include other things into that shader past like 1995, then you'll know that you have like include statements and you need to like include other things into that shader, right? And so for that, if you wanted to process that, you can't just have a pre-process statement that just blindly includes a string literal, right? You need to get the file, process any includes that are in that file, and then recursively like dump in the information so that you can have, you know, your full, like, file, right, to work with. And so that's basically all the problems that Stood in Bed happened. And so I was getting

pulled in two different directions. I didn't know what to do. And so finally, what I decided to do is that, with all due respect, you know, not, I don't care about the standardization committee. You know, I, and so what I did is I implemented Stood in Bed in GCC, right, just kind of on my own, and decided that there was no – there's no reason that the general C++ community can't have this built in, can't have this functionality, right, because the committee is like turning over on itself trying to figure out whether or not they want to deal with dependencies or not, whether or not they want to answer a 40-year-old question, right? So that's what I've decided to do. And so actually, by the time you're listening to this CFEcast episode, you'll be able to use std embed and hash embed on Godbolt like right now. So you'll be able to kind of experiment with it. I mean, it's not obviously pushing the mainline GCC. I still have to get the patch ready. But I've basically made this implementation to kind of prove out that it's possible to have embed,

that it's probably not going to, that it probably does what people expect them to, that most people aren't going to be confused by the fact that, you know, a dependency scanner can't automatically find what sit embed does, and that it's really just something that we should just get done, right? And so if the standards committee, if the standards committee says no, even for C++ 23, that's okay, because, you know, I'm smart enough to implement it now, right? So I'll just, I'll put the built in in myself. And, you know, de facto standard is stronger than a paper one. So let's do it that way. So do you think it will get accepted as an extension in GCC? I think it will get accepted as an extension, right? So you won't be able to have,

you won't be able to do hash include, you know, open, open Chevron, embed, close Chevron, right? But what you will have is the, what you will have is the built-in. And as long as we have the built-in, right, which can be under any name it wants to without having to require some pedantic warning that you're using it, that's fine. So what I'm going to do is I'm going to create a repository called phdEmbed that just implements what the stdEmbed

magic library function is supposed to be. Call the built-in, and that way we'll have stdEmbed for everybody. Now obviously I need to do the implementation for Clang, I can't do the implementation for Clang. I need to do the implementation. I can't do the implementation for Microsoft. But, you know, maybe you can, you know, rub some shoulders and, you know, grease some paws and say, hey, Microsoft. I definitely don't have enough money to grease Microsoft's paws.

But, you know, C++ doesn't belong to just the committee. It belongs to the community. It belongs to the implementers. And so even if they say no, right, I am absolutely ready to go the other route because I receive so many emails from people like, excuse me, when is student embed getting in the standard? And I'm like, that's not my decision. Like, I don't control this. I'm trying. I promise you I'm trying.

But, you know, we I'm going to implement the built-ins, right, as best I can. I already finished the one for GCC. And you'll be able to – you know, there will be a link below and you'll be able to check it out on Godbolt. Of course, I'll probably be tweeting it out like crazy because, oh, my God, I implemented something in the compiler. Woo-hoo. But I'm working on Clang, but I'm running into some problems with the way Clang handles stuff. Like in GCC, when you want to expand a built-in or whatever, you can arbitrarily generate code at that time and stick it into the tree, into its abstract syntax tree. Clang is a little bit more strict in that it seems like they really only want to do folding and stuff like that. And by the time they create their tree,

it's very much like a const tree. Like you're not supposed to modify it. I mean, you can, but it's super dangerous. Right. So, you know, I ran into some problems there. I implemented pretty much everything up to the last step, which is just like make the array that backs this like std embed thing. Just do that part, right? Just please do that part.

I can't do that part yet. So maybe when I go to – I'm going to be going to the standards meeting, so maybe I can ask Richard Smith or Eric Vassellier, ask them for some tips, slide me some information or whatever. But, yeah, if I'm able to get it into trunk Clang and trunk GCC, then, I mean, Microsoft will always be an X factor because they just do whatever they want. But if it's in Clang and GCC and people start using it, then all the static analyzers have to start supporting it, right? So that means EDG will have to support it, right?

And then you see how this master plan kind of locks it up from there. So even if it's not, you know, in the standard per se, that doesn't mean that it won't be a de facto standard. So, you know, we'll solve the C++ community's problems one way or another. And so that's kind of where student bed is right now. I mean, if I go to the standards committee and they say, and they see the implementation, they're like, okay, this is good, put in the standard, right, then that saves me so much work. But you know, I'm like, done. Yes, I'm like, done. Yes. All right, we're good. But if not, then I have to, like, it's going to be a lot of work, but, you know, it's worth it.

And I can also stop having Crent and Jabot constantly say, so when are you going to implement sit in bed? You know? Please. But, you know, if we do this right, I think it will turn out very nicely. Cool. That sounds great. Well, you've got a lot on your plate.

I hope you get the support you're looking for so you can continue the Unicode work. And it's been great having you on the show today, Jean-Huid. Yeah, thanks, Rob. Thanks, Jason. Thanks for coming on, yeah. Thanks so much for listening in as we chat about C++. We'd love to hear what you think of the podcast. Please let us know if we're discussing the stuff you're interested in or if you have a suggestion for a topic. We'd love to hear what you think of the podcast. Please let us know if we're discussing the stuff you're interested in, or if you have a suggestion

for a topic, we'd love to hear about that too. You can email all your thoughts to feedback at cppcast.com. We'd also appreciate if you can like CppCast on Facebook and follow CppCast on Twitter. You can also follow me at RobWIrving and Jason at Lefticus on Twitter.

We'd also like to thank all our patrons who help support the show through Patreon. If you'd like to support us on Patreon, you can do so at patreon.com slash cppcast. And of course, you can find all that info and the show notes on the podcast website at cppcast.com. Theme music for this episode was provided by podcastthemes.com.