CppCast - Approval Tests

Episode Date: August 1, 2019

Rob and Jason are joined by Clare Macrae to discuss Approval Tests and how they can be used to quickly test legacy C++ code. Clare is an independent consultant, helping teams streamline their work with legacy and hard-to-test C++ and Qt code. She has worked in software development for over 30 years, and in C++ for 20 years. Since 2017, she has used her spare time to work remotely with Llewellyn Falco on ApprovalTests.cpp, to radically simplify testing of legacy code. She has enjoyed this so much that she recently went independent, to focus even more on helping others to work more easily with legacy code.      Clare was until recently a Principal Scientific Software Engineer at Cambridge Crystallographic Data Centre. She is the original author of their popular 3D crystal structure visualisation program Mercury. News Cmake 3.15 available Clang/LLVM Support for MSBuild Projects LEAF light-weight error-handling lib seeking Boost review manager Clare Macrae @ClareMacraeUK Clare Macrae's Blog Links C++ Approval Tests Approval Tests #include Happy one-of-our-birthdays #include! Sponsoring Diverse CppCon 2019 Attendees #include sponsorship for CppCon 2019 Sponsors Errors that static code analysis does not find because it is not used PVS-Studio in the Clouds - Running the Analysis on Travis CI Hosts @robwirving @lefticus

Episode 208 of CppCast with guest Claire McRae, recorded July 24th, 2019. Sponsor of this episode of CppCast is the PVS Studio team. The team promotes regular usage of static code analysis and the PVS Studio static analysis tool. In this episode, we discuss a lightweight error handling library. Then we talk to Claire McRae. Claire talks to us about approval tests and how they can be helpful for testing legacy code. Welcome to episode 208 of CppCast, the first podcast for C++ developers by C++ developers. I'm your host, Rob Irving, joined by my co-host, Jason Turner. Jason, how's it going today? I'm okay, Rob. How are you doing?

Doing okay. How are you doing? I mean, I know a lot of the East Coast is going through a heat wave right now, although we had a pretty big storm here yesterday, so it cooled off a bit. Did that affect you at all, like a week ago or so? Our weather's actually been normal for here. Like, we did get around 100 degrees, but that's what you expect, July, August in Colorado. But we've also been getting seasonal, like, afternoon thundershowers, which cool things off, which apparently used to be normal, July, August in Colorado. But we've also been getting seasonal like afternoon thunder showers, which cool things off, which apparently used to be normal like 30 years ago, but it's never been

normal since I moved here. So certainly that's super welcome. And actually, I'd say this is like, like, I don't know, it seems like the most normal, calm possible weather we could expect at the moment. That's good. I know that some parts of the country are suffering right now, but hopefully we'll get some relief soon. No, we are getting ozone alerts for poor air quality though. And if you hear that in my voice in this recording, that might be why.

So our pollution is terrible, but the weather's great. That's something going on there. Okay. Well, at the top of our episode, we got a little piece of feedback. This week, we got an email from another jason uh saying jason and rob i look forward to listening to you guys every week to

keep up on c++ i write uav autonomy software which c++ is an integral part i've recently started exploring the ros2 framework and got to thinking about thinking it would be awesome if you guys could interview morgan quigley lewis poobel or dirk thomas from the open source robotics foundation uh i'm not sure the best way to contact them but it looks like dirk and lewis are on linkedin and lewis has a website on his on twitter um i feel like we have talked about open robotics a long time ago when we had jack Kay on, right? That might be right. I was going to say I had no idea, but yeah, that sounds entirely possible. Yeah.

So we should either maybe try to reach out to her again or reach out to someone else who's working on this open robotics foundation. I think we're going to have to have a conversation about this whole other Jason thing, though. I'm not aware of any other J's in the C plus plus world. I don't know if there's enough room for more than one of us. We've had at least one other Jason on the show, I believe. Yeah.

Have we? Yeah, I think so. This was from Jason beach. It's, it's been, we had Jason rice on about 50 episodes ago. Oh,

Jason rice. Yeah. There are other Jason rice yeah there are other jasons there are other jasons oh all right fine i'll just have to live with that yeah okay well we'd love to hear your thoughts about the show you can always reach out to us on facebook twitter or email us at feedback at cpcast.com and don't forget to leave us a review on itunes or subscribe to us on youtube joining us today today is Claire McRae. Claire is an independent consultant helping teams streamline their work with legacy

and hard to test C++ and Qt code. She has worked in software development for over 30 years and in C++ for 20 years. Since 2017, she has used her spare time to work remotely with Lou and Falco on approval test CPP to radically simplify testing of legacy code. She has enjoyed this so much that she recently went independent to focus even more on helping others to work more easily with legacy code. Claire was until recently a principal scientific

software engineer at Cambridge Crystallographic Data Center. She is the original author of their popular 3D crystal structure visualization program. Claire, welcome to the show. Thank you very much indeed. It's a pleasure to be here. Cambridge Crystallographic Data Center. Sorry, I got hung up on those four words right there. You have to tell us something about that. So it's an amazing organization. It's 54 years old now. So a computing organization that has lasted that long is amazing. It's an organisation that collects published and unpublished crystal structures, so the shapes of molecules in 3D. And just before I left,

the Cambridge Structural Database reached a million crystal structure sets. So it's an amazing achievement. The data is really valuable for scientific teaching, for learning, for research, and for designing new chemicals, mainly medicines, some agrichemicals and things like that. So it was a great, great place to work and non-profit, really positive and supportive place to work. So leaving was a big decision, but I'm looking forward to a new phase of my life. That sounds pretty cool. I mean, but as a fan of science fiction,

I just can't help but wonder, like, are any of these crystalline entities becoming self-aware and, you know, destroying space stations or anything? But it's probably unlikely, I guess. There has actually been some crystallographic research done on the International Space Station in space where we find how crystals form is really important

for things like the manufacture of medicines. And apparently you can learn a lot by trying to grow crystals in low-gravity environments. So, yeah, there's potential for science fiction links somewhere in the future, I feel. That sounds great. But honestly, I had no idea that the formation of crystals was important to the creation of medicines, that kind of thing. Yeah, that's interesting.

Yeah. Okay, well, Claire, we got a couple news articles to discuss. Feel free to comment on any of these, and we'll start talking more about approval tests and legacy testing. Okay. Great. Okay. So this first one is CMake 3.15 is available for download. And I think we did talk about the 3.15 release a little bit a few weeks ago when we had Robert Maynard on, right, Jason? We did. Yes. Yeah. Anything you wanted to highlight here though? There's a couple of things that stood out to me. One is the ability to use the GNU-like versions of Clang for Windows with CMake on Windows. This sounds separate from the Clang-CL command line version, I believe, which is the one that has the cl.exe parameter support.

That will come up again in two news articles. The other thing is that they added a bunch more message types, which I find interesting. Notice verbose debug and trace. It says our new types. And it made me wonder, does CMake have the ability to actually filter which types, which message types are displayed at configure time, which is something I'd never thought of before? And apparently it does. It has the dash dash log level option when you call CMake from the command line. Okay.

How about you, Claire? Are you a CMake user? Yes. Yeah. I enjoy it a lot. I feel I've got a lot to learn about it. This was the first time I looked really closely at a set of release notes for it. I was really impressed at how much work is going on.

The things that stood out for me, I mean, it was hard to pick out individual things because there's so much listed. But there's a new environment variable that it supports now, cmake underscore generator. So if you've settled on a generator and you just want to carry on using that you can set that environment variable and not have to keep typing minus g on the command line so i thought that could be could be quite convenient and um um the there's another variable i've introduced cmake vs just my code debugging so it's possible you have control of that basically at configure time whether to enable Visual Studio's feature to just step into your own code and debugging. So I thought that was a good one to know about too. Are you a Visual Studio user? Yes. Now I use

Visual Studio when porting my code to Windows and such, but I don't tend to use it. And I'm aware of the just my code feature, but I assumed it was a debugger thing, not a compile time thing. Do you have to recompile the code with that feature enabled? That's a really good question. I didn't look onto that. I don't know. I need to look at that. Yeah, that's a good question.

I always also thought of it as just a debug feature. Yeah. I'm just wondering why it's now a CMake variable. That implies that it's something that happens at build time? Right. That's a really good question. I need to look into that a little more. I'll have to go away and experiment. It seems like it would be much faster if they filtered those things. Maybe at link time, even,

if it wasn't necessarily at debug, at build time, I mean. Well, I have no idea. Okay. Well, since we're talking about Visual Studio, we can move on to the next article, which is that they're adding Clang LLVM support to MSBuild projects. And this follows up from a previous announcement where they introduced Clang LLVM support in their CMake integration. projects you can also use clang as your you know compiler from within visual studio which is a pretty big change uh for them to allow you to use something other than msvc yeah and this is

the one that says clang-cl and one of the questions down at the bottom is uh do you plan on using is it possible to use clang plus plus in addition to clang cl an ms build or a cmake project apparently the answer is as of last week yes very cool if i'm reading the cmake commit correct and correlating it with this guy's uh this person's question correctly as well right okay and then the last article we have is um on reddit cpp and this is about, which is a lightweight error handling library, and it's currently seeking a boost review manager. And as we talked about before on the show, being a boost review manager is a pretty big commitment,

so they can be hard to find. But take a look at this library if it's something you're interested in. Maybe you'd be willing to be the review manager. Did either one of you dig into the library itself? I looked into it a little bit, and I read about the comparison between Boost Outcome, which is another library we've talked about. Right. Yeah.

What were your thoughts on it, Jason? I don't know. I didn't feel like I really, I don't know, actually formulated any solid thoughts. That's why I was asking if either one of you did. I had a look at it. uh skimmed through the documentation it's stunningly beautiful documentation um so it'd be interesting to understand how they produce it um i spent quite some time trying to understand what the purpose of the library was you know and under what circumstances would someone be interested in in using and it wasn't till i got right to the end in the comparison of the library was, you know, and under what circumstances would someone be interested

in using it. And it wasn't until I got right to the end in the comparison with other alternatives that it said useful in low latency environments. So, you know, better than exceptions or the current availability of exceptions. So that was good to understand. Oh, I see. Yeah. That's like, if you have to scroll like 300 pages down this first page you

click on is very large i feel like they probably should put this comparison to other libraries near the top yeah yeah but at the same time as having written my own open source libraries you know sometimes the temptation to want to compare yourself to other things is great and sometimes you really just don't want to make the comparison to other things and let people ask those questions and although i mean i feel like if it's a boost library though and you're trying to get yourself into boost that's going to be one of the first things you ask like is there something similar to this already in boost right yes yeah good point okay so claire um do you want to start off by telling us about the talk you're going to be giving at CppCon 2019? Yes, I'm going to be giving a talk called Quickly Testing Legacy C++ Code with Approval Tests.

And I just learned today that it's going to be on the Monday afternoon because I see that the CppCon schedule has just been announced. Oh, did they? I didn't even realize they announced that. There might be sort of minor changes and things like that, but they've got the full schedule up now on Shed. I also did not know that. Now I should know when I'm speaking. Okay, so you said it was testing legacy code with approval tests, though, was the name of the talk? That's right, yes.

So I don't think we've talked about this concept of approval test before on the show. Do you want to tell us a little bit about them, how they relate to unit testing? Sure. So I think that approval tests are a lot better known in other languages than C++. So that's become a bit of a mission of mine to try and share some kind of understanding about the situations when they're useful. I think lots of people are familiar with the idea when you write unit tests that you have the pattern of arrange, act and assert so that your test ends up saying the recipe that the test is executing and then a statement about what the right answer is.

So value is going to be 42 or in a given range, that kind of thing. And writing unit tests as you're writing your code can be a very effective and very satisfying way of working. But if you've inherited a large chunk of code and that code often isn't structured in a way that it's even easy to write, to do the arranging and do the acting without creating lots of objects beforehand and so on. Maybe you don't at this point even know what the right answer is. Approval tests provides an alternative to the sort of fine-grained unit test approach.

With approval tests, you say you're arranging your code, you're doing the setup, and the setup might be run my entire program or virtually all of my program. You provide the inputs, and approval tests provides a range of sophisticated ways to save the output, so save the current output of your program,

as a snapshot of the current behavior. It's very similar to the Golden Master approach, but it's quite a sophisticated implementation of Golden Master. So you don't have to understand the details of the program you've inherited. You run the program, you capture some output, and you save that in your version control system and from that point on as you're refactoring the code you can get quick feedback as to whether you've changed unintentionally changed the behavior so i guess the summary is if you can write unit tests do if you can't or can't easily or you need a starting point um to break out of the i

can't refactor the code because i don't have tests loop then approval test provides a surprisingly powerful surprisingly quick to use mechanism for such a small volume of code and it's been luell and falco created and has popularized the. And it's well established in a dozen or so other languages, but relatively new to C++. This concept is very apropos for me because I've been, as I teach students, it's often people who want to get up to modern features of C++ and they're dealing with some 40-year-old code base or something like that. And I just had my students last week ask me like, well, you know, how do I start refactoring this code? I'm like, well, you can't do anything without testing first.

But what exactly does testing need to look like? Like, I'm not the expert in that. And it sounds like you have a good answer to that question. Yeah, I actually used to. It's not just for legacy code i um one of the things i did over christmas when i'm working on mercury the crystal structure visualization program i was adding a new display style um it's hard to hard to explain in a podcast but um but we have video you know yeah so instead of drawing atoms just as spheres, some chemists in some situations want atoms to be drawn as polyhedra,

so that you can where the corners of your polyhedrons are the neighbouring atoms. So for certain types of chemistry enables much quicker parsing, if you like, interpreting of the crystal structure, the shapes of the molecule. And so I didn't know what the right answer was. So it's basically OpenGL, changing some OpenGL code. And I went through several iterations of the implementation, and I found that I could use approval tests with saving my pictures of crystal structures in png format and then using a really nice differencing tool to run through 500 or 1000 crystal structures and very very quickly be able to inspect the results and then as i was improving the implementation i could make sure that i wasn't changing the image output and then when I saw a problem that I wanted to fix,

I changed the implementation. You know, perhaps there was some visual ugliness on one particular orientation of atoms or something like that. And then I could quickly run through, once I changed the implementation, and only see the structures where the results had changed, the pictures where the results have changed. So I have to say at that point,

I was seeing if I could use approval tests for that because I had a talk coming up and I wanted an example. But actually, it turned out to be really powerful and it was a great way to communicate with the product owner as well and say, okay, here are some images that are problem cases. Do you care about this or not very often it was no that's an unusual structure people won't complain about that don't worry so it was really really satisfying so just make sure i understand what you're saying so when you first

describe approval test i'm imagining some command line application that spits out some text output saves that and then compares it when running the program again. But you're describing like printing out an image and then comparing that against another image to see what the changes are, if any. Yeah, so that's part of the challenge of explaining approval tests is trying to give a sense of the variety of ways in which it can be used. I think the most common way will be if you can have some text representation of the... It can be of the inputs to your tests and to the outputs. And we have a page on GitHub in the documentation that talks about how you might evolve that text representation.

It turns out through experience with using it over years that people found nice ways of structuring the output so that if you're looking at a few hundred lines of data that you can see patterns and make the outputs really glanceable. So if you've got a command line program that has no tests and its implementation is in C++, maybe before you even start changing the code, you might use the Python implementation of approval tests as a wrapper around your entire program. And so then you haven't changed your code at all, as long as you've got some output from your command line program

that you can preserve and version control and see changes. If you don't have that, then you'd be trying to find seams in your program probably initially as close to your main as possible. And you would then have a test program create a test program that instead of directly using google test framework or the cache test framework for example it uses approval tests as a layer on top of that on top of either of those frameworks and your approval test would write out the text some text representation of the outputs that you're wanting to check over time so so it doesn't always provide by the c++ layer you're

going to have to spend a little bit of time trying to find some seam, some layer that you can put tests around. But you can do that so much more easily than needing to refactor your code down to the small units that you can write unit tests for. Does that make it any clearer? Yeah, it does. I have so many questions about so many possibilities right now. Going back to your like your png image my first thought was are you doing an actual png diff yes okay so you can see the actual pixels that have changed yeah so then what approve no sorry go ahead go on so at first when i was doing this okay so maybe if i actually explain what's at a very simple level what approval test is actually doing under the hood it might well it'll certainly make this conversation easier and hopefully it might answer some of your questions so what approval test does is it provides first of all it provides

a naming convention for saving your files so it knows about the test framework that you're using google test or catch and soon doc test so you don't have to spend time thinking about how you're using, Google Tests or Catch and soon DocTest. So you don't have to spend time thinking about how you're naming your output files. It says I see your source file name is this and your test name is that. So I'm going to create files called this.that.received. Okay. So you run your test and it writes out a file called received one one for each test in your test program each approval test and it looks for a file the same base name but called dot approve and the first time you run the approval test you the approved file doesn't

exist it can't magically create that so it shows up in a differencing tool. It looks for differencing tools on your machine. So it's trying to save you. It's trying to go for minimum developer configuration. So if it finds any of a couple of dozen commonly used differencing tools, it will launch the output from this run on the left-hand side, left-hand panel, and the approved output, which is currently empty because you haven't approved any output yet.

And if you like the output, you use the differencing tool to copy the received file over to the other side of your differencing file, to the differencing tool, the approved file. So the first time you run it, it's 100% diff and you accept it, basically. That's right, yeah. of your differencing file to the differencing tool, the approved file. So the first time you run it, it's 100% diff and you accept it, basically. That's right, yes. So approval test, the first run always fails unless you've done something clever

like saving some output from some other mechanism. Pretty much, it always fails. And if it's legacy code and you don't know what the right answer is then you automatically approve it because this is the current output and that's the current behavior and then of course you submit to your version control system the uh the approved files and they become part of your test in future um so it's a key part of using approval tests well is formatting the information that you save so that when you or another developer in future sees a test failure they

get enough information it's easy to interpret was this an intended change in behavior in which case i will update the approved file and that becomes the new golden master or oh no i made a mistake i forgot about this case i'll stop the test fix the code and try again so it's the temptation is to use whatever the output stream operator is for your big class that you're testing that's writing out 50 different things and that's certainly how i started doing it um but that makes it really hard if you get a change you don't you know maybe someone changes the operator and adds a new data item or such to your big thing that you're testing right so so part of of helping people understand this process is kind of sharing experience about it may seem like it's extra work writing your own Lambda to format the text or a little helper function or something like that, but it's definitely worthwhile doing it. And here are some output patterns, text layout patterns that you could use to help with that.

Does that fire away with your next question. Well, my next question is, I have two different ways that I could ask it. But I think it's basically the same question. I'm thinking still about your diff of binary images, but also diff of text files. Like say you're running anti aliasing on these images. Yeah, that inherently might cause a few pixels to be different right yes so do you have the ability to say well a one percent difference that's acceptable yes so first of all um approval test reads the two files and does a character for character comparison okay um it's very opinionated that differences in line endings don't matter. That's its one

opinion. And in a lot of cases, end of line characters, and in a lot of cases, that's fine. And you're getting stable output. What I found when I was doing this image comparison was I was getting different output files depending on whether I was sitting in front of my computer at work, in which case I was getting 32-bit PNGs, or if I was remote desktop in, in which case I was getting 24-bit PNGs. So I was getting failing tests and didn't understand why, and that turned out to be the answer.

Even though the alpha channel was zero or whatever the whole time yeah yeah so we ended up and as um as i used approval test more i found there were some times when when i was logged in directly and getting these 32-bit images where the Qt image writing was giving me one out of 255 differences you know 255 values for red green and blue tiny tiny pixel differences that no user could see the difference and I didn't care about right and so at that point we added a mechanism to the C++ approval test implementation that you could provide per file format a custom comparator so we said so the code I had said if it's a PNG then

use Qt to open because I was writing on Qt code, use Qt to open the image and then loop through the pixels so we hope that most of the time you don't need to customize it, but you can do if you want to. So under the hood, what Approval Tests is doing is providing file names for you,

which is really, really helpful that you don't have to think about the file names. You give your test a sensible name, your file gets a sensible name. It's doing that comparison that i mentioned and then if either the approved file doesn't exist or if it detects a difference if it looks for a differencing tool one or more differencing tools on your system and shows you the visual representation of the differences and And you can customise in all sorts of really powerful ways how it presents that information, including saying automatically approve the difference

if you've made an intentional change or you want to just review the differences in a version control system or whatever. And then finally, once it's used, it calls reporters as its vocabulary for the reporting the difference. And then finally, it throws an exception, one of its own custom exception types. And that's how it integrates into version control systems, all you need into, sorry, into test frameworks. All it needs for a test framework is the ability to provide the name of the test that's running and that when an exception

is thrown that the test framework catches it and says unexpected exception which makes it really easy to extend to other other frameworks in future so it's surprisingly simple but very very powerful yeah it sounds very powerful so it sounds like uh and the other branch of my question was, if you've got timestamps in your output files, then it sounds like it is up to you then to write a custom filter that would filter those out and be able to diff them. So that's a really powerful question. I'm really glad you asked that. So the standard answer for things like timestamps is refactor your code

so that you can pass in a known date or something so that your production code uses the actual timestamp and your testing code um and what approval tests allows is you can hook in after the file say your log file or whatever it is that you're approving, after that's been written out, it provides a way for your test code to then read that file in and modify its contents. So you might imagine a regular expression that is configured to recognize the date and timestamps in the system that you're testing and so you don't need to change your code that you don't like it's maybe it's writing out pointer address objects or something like that so you read your log file you approve the file that you're being approved

that you're approving your test code tidies up and normalizes anything that would be changing data. And then ApprovalTest sees that as the file the system wrote and that's stable over successive runs. Okay. It's a lovely idea and it works really, really well. Does that make sense? Yes. I wanted to interrupt the discussion for just a moment to talk about the sponsor of this episode of CppCast,

the PVS Studio team. The team promotes the practice of writing high-quality code, as well as the methodology of static code analysis. In their blog, you'll find many articles on programming, code security, checks of open-source projects, and much more. For example, they've recently posted an article which demonstrates not in theory, but in practice, that many pull requests on GitHub related to bug fixing

could have been avoided if code authors regularly use static code analysis. Speaking of which, another recent article shows how to set up regular runs of the PVS Studio static code analyzer on Travis CI. Links to these publications are in the show notes for this episode. Try PVS Studio. The tool will help you find bugs and potential vulnerabilities in the code of programs written in C, C++, C Sharp, and Java. So I think you said approval tests can integrate with other unit testing libraries just by being able to throw an exception if it recognizes a difference?

That's right, yes. So does it work with any unit testing library or does it have specific integrations? There are specific integrations and and it would need specific integrations. So it supports both versions of Catch. Though ApprovalTest, the library, uses C++11 constructs, so I think there's no reason to use Catch 1. You might as well use Catch 2. And Google Test Framework, it supports well, too.

There's so much that we want to do with it. We are mostly being driven by requests from users, and we recently had a request for DocTest, which I see was covered not too long ago on this podcast. And so we're partway through implementing that, and I think that will be implemented in the next few days. So please do get in touch with us if there's any other frameworks you'd like to see added.

Okay. Do you have any good success stories about using approval tests to help modernize legacy code? Yeah, so I gave an earlier version of the talk I'm going to be giving at CppCon at C++ on C in February. And somebody saw the video of that and saw that it would be valuable in testing some Python code

that was really, really important code, many, many paths through the code and no conceivable way of writing tests for it and with just a very short conversation this was something that came through the include c++ discord a request for help that I was able to explain so a common challenge is approval test is designed to write one output file per test case that you've got but sometimes what you're testing is going to write out several files and you want to approve that so there's a bit of a trick to do that and with that trick within 24 hours of the video being watched I got feedback that this really really years-old code was totally tested,

had been refactored, the approval tests had caught several small mistakes in the refactoring, and now that code could be maintained. So that was really obviously very satisfying. And I think that's an important thing, although I'm mainly speaking at c++ conferences about approval test because it's implemented in a dozen or so languages and and it uses um the same concepts the same vocabulary in all the different implementations but tries to fit with the language as well as possible if anyone's listening to this and they've got legacy code in other languages, then have a look.

Chances are you'll find what you're looking for. You already mentioned Python. What are some of the other languages that have the approval test implementations? So from memory, there's a range of different.NET versions. So I don't know the details, but there's sort of core C-sharp implementation, and then there are various layers on top of that for different aspects. So one for testing GUI environments and things like that. I must admit I haven't looked too closely at what the different ones are.

Python I've used for some of my own computing projects at home, so I'm familiar with that. I applied to speak at Meeting C++ this year, and I decided I wanted to branch out a bit. And so I'm going to be talking about testing cute code quickly testing legacy cute code or cute desktop applications so sometime between now and november there'll be an additional c++ approval test library that will depend on the cute framework and provide some ways of using approval tests there. That's something I did use at work just before I left with a dialog whose implementation code

was mixed in with the Qt custom widget code, and I wanted to separate that out. And as it happened, it was for a widget that had an easy it was a table widget so it had an easy text representation so it was very comfortable um very easy to to fit that in but um with approval tests but we i don't think we want to make the core approval test library depend on cute so it will need to be a separate layer on top of the basic c++ approval test. Yeah, that would almost certainly slow down adoption from people if they had to pull in Qt on their project with no current dependencies. I think so, yes. But so now, okay, you're dumping some sort of textual representation of what's in a

dialogue and using that as the thing that you're diffing for your approval test with the Qt testing? That's right. In this case, it was a dialog that had two tables, basically. So a table of scientific data in the main dialog, and then an extra kind of info dialog that it popped up. So in that particular case, the text representation was easy to see and easy to create. At first, when you first started mentioning it, I'm like, what is the idea to dump like a PNG of the dialog box? Which seems like that would be not a great way to go about that, but I don't know, right?

So for Qt in particular, there is a separate testing framework called squish and um i know from experience at work that you really really want to try and avoid using screenshots for as much as possible in testing um as soon as you want to run something on a different environment you end up with so much pain and looking for differences and things like that so um the example i used earlier i was only saving images because you know it was opengl code opengl code exactly what i wanted to save was the image and right and see see the changes and that code those tests are disabled in the code that i i learned very quickly that partly because of the volume

of output and the subtle changes running in different environments it was a test that was useful on a given machine when that code was being maintained so but but not and but it gave me the information that then i could write unit tests for the particular mathematical challenges i discovered through being able to throw large numbers of crystal structures through the approval test image mechanism. So I don't always see approval tests as the end result and necessarily something that you would save forever. It can be a means to an end for a short-term thing,

where you're stuck on writing unit tests and it may be that if the setup is expensive and time consuming of the tests that you soon want to get to the stage of having unit tests and unit tests for longer term maintenance I'm sure are going to be the answer in many cases but there are also times when the approval tests are quick enough

and sufficient for the needs that they may live for a long time. It really depends. You have to form your own opinion based on whatever the case that you're testing. But it's really, I come back, the goal of it is to unblock in scenarios when you can't see, it's not feasible for you in your situation to start with writing small tests. It makes a lot of sense. And I can totally see it in this scenario. And you kind of answered the next question I was going to ask is, like, if you already have one particular organization

I'm thinking of has mostly just behavior-driven tests. They don't care about unit testing. They just want to know that the behavior of the system stays the same. It kind of sounds like they kind of already have what they need from an approval test. I think so, yes. Okay. We've been talking about legacy tests a lot, but approval tests provides a really nice mechanism for testing, for acting on containers of values. So and containers are sort of a combination, multiple nested containers.

How to describe it so you may have some new code that you don't regard as legacy code but you want to generate a large number of test cases for it and to write individual unit tests one line at a time for thousands of test cases would be prohibitively expensive and so approval tests is really valuable in that situation as well um where you might say um i don't know say it's a chess board or drafts board or something like that you've got eight cells by eight cells and you want to test large numbers of different moves um with inputs and outputs you can have a little text representation of the board before and after the move, and then have containers with large numbers of moves in and provide a nice visual representation

that's easy to understand and easy to see changes. So I think there are specific circumstances when you're working on new code where the fact that you can generate large numbers of test cases very easily and manage them in an easy to understand text form. That's the other main area where I think it has a lot of benefits. Now, earlier you said the last version of this talk you gave, you got like a 24-hour turnaround from someone saying this was amazing, it solved all of our problems. I'm assuming not everyone

should expect that within 24 hours they've solved all of their problems. But do you have any kind of rule of thumb guidelines for about how long it takes to set this kind of thing up for the new newbie? I think that understanding approval tests is a small part of the process. Okay. I think how long it takes to actually get going is really going to depend on how malleable at all the program is that you're working on. So if it's a command line program and it's text input and text output, then that's going to be a much quicker process than if it's the guts of some complicated dialogue or something like that. So I hate to quote numbers and then sort of mislead people.

I will say that the dialogue example that I mentioned earlier at work, that probably took me somewhere between half a day and a day or something like that. But that was code where I didn't really understand the code when I started and I needed to understand the path through the widgets and things like that. But to write individual unit tests for that and get the same degree of coverage that I got with approval tests would have taken me considerably longer. I guess I'd love to throw it open. If anyone tries it out and gets stuck, ping me on Twitter, ask questions.

You know, my life now is helping people to test code that's hard to test. And so the more examples I get to see, the more I get to help people and then have that feed into helping other people, the more fun I get to have and hopefully a few other people get to benefit from it along the way. And you said, did I hear you correctly that

you're going independent doing this? That's right. Yes. Okay. Well, I mean, it sounds like so far, you've made a great sales pitch. Honestly, because I know there's a lot of companies and, and even individuals at companies that are just like, well, I was just given this project that's 20 years old and it doesn't currently have any unit test. What do I do? Well, it sounds like you have some answers. This is a great way to start. Going back to that 24-hour success story you were talking about,

you said that someone reached out to you over the Include C++ Discord. Are you pretty active in that community? I know they just had like a two-year anniversary, I think, right? Yes, I was really lucky with the timing. So I've been in, I wouldn't say I'm absolutely one of the core people, but I've been very supportive throughout that entire time. And when Guy Davidson spoke at Meeting C++, I think probably 18 months ago, I was on the Code of Conduct group

for that conference. I try to support where I can. Okay. What are the goals for CppCon this year with the Include group? We're doing an amazing amount of things at CppCon. On the Wednesday evening, there's an

Include dinner. dinner include c++ dinner um and after the dinner so the idea of the dinner is to come along and network with other people who are already helping to make c++ a more welcoming and supportive and inclusive community and afterwards kate greg Gregory is leading a discussion panel. So I think that's going to be a really, really interesting evening. You can sign up to that when you register for the conference. And there's two different prices as well. Thanks to the foundation for offering a discounted rate

to subsidise dinner for those for whom it's too expensive. We're going to have a table there where we'll be selling t-shirts and giving away stickers and things like that and our table at conferences turns out to be a really great place to meet other friendly like-minded people so that's become a bit of a magnet um on discord our discord discord server um there'll be a channel where if you want to meet people for find people to go and have a meal with maybe have dinner things like that you don't have to worry about if you don't know anyone at the conference it's a great way to to start to make acquaintances

and friendships and i think the most significant thing is that we have been running for a few weeks now a funding initiative to run scholarships to send a few applying to the conference student scheme, or maybe employers won't send them. So it'd be great if you could include the application, the scholarships application link in the show notes at the end. And that also has a link to the funding. So if you could donate to that scholarship fund we'd really appreciate that as well so far and in our two years probably pretty much over the last year we've sent about a dozen people to conferences most of them fully funded with transport as well and we've had help from the conferences as well to to support that and um i haven't been aware of who the individuals are that have been going, but someone else pointed out that some of the people that we have previously sent to conferences

are now contributing to the funding pool. And in some cases are already speaking at conferences and already supporting other people and pulling them up the ladder behind them. So it's very exciting and very satisfying. Yeah, I know we met at least one of the attendees who went with the Include C++ sponsorship last year, and we had her on our show of different lightning talks that we did during the conference. Yeah, and definitely seems like a great program. We'll definitely put a link to that in our show notes. I also want to mention the t-shirts were a really hot commodity last year. I wanted to get one and I made the mistake of not picking one up on the first day and you couldn't get them later in the week because they were selling out so quickly.

And it's been great seeing photos of other conferences and even photos of standards committee meetings and seeing people wearing the Include t-shirts. So I think they make quite a powerful statement that there are lots of people out there, including some very well-known people who are supporting this initiative. I also wanted to say for anyone who maybe hasn't noticed that CPPCon themselves have made a couple of great announcements recently. So they've announced that there's going to be not one but two quiet rooms at the conference this year. So some people just need to get away from noise sometimes and just have a break. And it can be really useful to just go and take a break

because these conferences are, I was going to say they can be, but they are very intense environments. They are, yeah. And just yesterday the conference announced that there's going to be funded childcare. Oh, yes. I said for a limited number of places. So I guess I last went to CPPCon in 2014, and then there were about,

I think it was the very first one, there were about 600 attendees then and maybe 10 or 20 women, something like that. I mean, it was a welcoming and friendly environment. I don't want to suggest otherwise, but I very much felt like an outsider. And the C++ community has changed and evolved so much in that time. So I'd quite like to say for anyone who isn't sure about whether to give CPPCon a go this year, I'd say you're fine for many different reasons. It's a really, really welcoming environment.

I'd definitely encourage you to give it a go. And as I said, if money's a problem, then definitely look at the scholarships. I kind of, if you don't mind, want to rewind just a little bit to when you said that the Include table is a great place to meet people for dinner. I believe that's what you said. So there's two different things going on. The Include table is great for coffee breaks and things like that, maybe going to get lunch with people and things like that. Or get lunch, right.

But on Discord, at all of the C++ conferences, we have a channel specifically for people at that conference so often in the evenings there'll be there's a group of us meeting up to go out to dinner at such and such a place we're going to meet in the lobby at this time everyone's welcome to join us and i think i just wanted to highlight that as as being like an important good thing because as you said, the conferences can be overwhelming. Sometimes you don't know anyone there, you're just standing there looking around like, well, I guess I'll start wandering in a random direction and see if I find a McDonald's when really, like if you took the time to maybe wander by the include table or go on Discord, you can find someone to actually communicate with for, you know, meal time. Unless you just want to be alone, that's fine, too. Yes, yeah. Okay. Is there anything else you wanted to go over,

Claire, before we let you go? No, that's great. It's been a really interesting conversation. Thank you very much indeed. Yeah, thanks for coming on the show today. Pleasure. Thanks for joining us. Thanks so much for listening in as we chat about C++. We'd love to hear what you think of the podcast. Please let us know if we're discussing the stuff you're interested in, or if you have a suggestion for a topic, we'd love to hear about that too.

You can email all your thoughts to feedback at cppcast.com. We'd also appreciate if you can like CppCast on Facebook and follow CppCast on Twitter. You can also follow me at Rob W. Irving and Jason at Lefticus on Twitter. We'd also like to thank all our patrons who help support the show through Patreon. If you'd like to support us on Patreon, you can do so at patreon.com slash cppcast. And of course you can find all that

info and the show notes on the podcast website at cppcast.com Theme music for this episode was provided by