CppCast - Backtrace

Episode Date: November 30, 2016

Rob and Jason are joined by Abel Mathew, Co-Founder and CEO of Backtrace I/O, to talk about the debugging platform and its features for C++ developers. Abel Mathew is the co-founder and CEO of Backtrace I/O. Prior to Backtrace, Abel was a Head of Engineering at AppNexus where he led a team of developers to improve ad optimization and reduce platform-wide costs. He spent multiple years as a developer and a team lead on AppNexus’ Adserver Team where he helped design and implement their low-latency advertising platform. Before AppNexus, Abel was a kernel module and tools developer at IBM and a server room monkey at AMD. News Give Visual C++ a Switch to Standard Conformance Zapcc: a faster C++ compiler Better, stronger, faster … there is zapcc Conan Joins JFrog What do YOU use C++ for Abel Mathew @nullisnt0 Abel Mathew on GitHub Links Backtrace Backtrace Blog Minidump Free Beta Surge 2016 - Abel Mathew - Post-mortem Debugging: could you be the one? Bazel Sponsor Backtrace

This episode of CppCast is sponsored by Backtrace, the turnkey debugging platform that helps you spend less time debugging and more time building. Get to the root cause quickly with detailed information at your fingertips. Start your free trial at backtrace.io.cppcast. And by Meeting C++, the leading European C++ event for everyone in the programming community. Meeting C++ offers five tracks with seven sessions and two great

keynotes. This year, the conference is on the 18th and 19th of November in Berlin. Episode 80 of CPP cast with guest Abel Matthew recorded November 30th, 2016. In this episode, we discuss a new C++ compiler. And we talk to Abel Matthew, co-founder and CEO of Backtrace. Abel tells us about the debugging platform and its features for C++ developers by C++ developers. I'm your host, Rob Irving, joined by my co-host, Jason Turner. Jason, how are you doing today? Doing good, Rob. You have a good Thanksgiving? I did. Took it easy this year.

Last year, we wound up traveling back up to New Jersey for both Thanksgiving and Christmas, and we just decided we didn't want to do that again, so we did Thanksgiving local with some friends here this year. Yeah, that'd be quite the drive up and back down and up and back down again. It's not fun. It was like an 11-hour drive back from Jersey to North Carolina last year. And how long would you expect it to be? You can do it in about seven. Oh, goodness.

Yeah, that's not worth it. No, not at all. Well, top of every episode, I'd like to read a piece of feedback. Last episode, Vipple wrote in a comment on the website, formal specification modeling verification like ACM Turing award recipient leslie lamport tla's work at ms research according to acm.org it's been used to find a major error in the coherence protocol using the hardware for xbox 360 prior to its release in 2005 and at intel for the analysis of catch coherence protocol of the intel quick path interconnect as implemented in the Halem core processor.

That's pretty interesting. We're talking about the formal verification of code, I guess, last episode, right? Yeah, it did come up. I don't recall the context at the moment, but we definitely mentioned it. Yeah, it definitely sounds like an interesting topic and must be pretty powerful. it's able to catch problems prior to release of hardware so definitely something we'll have to try to find a guest for yeah we need to do that yeah i think so well we'd love to hear your thoughts about the show as well you can always reach out to us on facebook twitter or email us at feedback at cpcast.com and don't forget to leave us reviews on itunes joining us today is abel

matthew abel is the co-founder and ceo of backtrace io prior to backtrace abel was a head of engineering at app nexus where he led a team of developers to improve ad optimization and reduce platform-wide costs he spent multiple years as a developer and a team lead on app nexus's ad server team where he helped design and implement their low-latency advertising platform. Before AppNexus, Abel was a kernel module and tools developer at IBM and a server room monkey at AMD. Abel, welcome to the show. Thanks, guys. Thanks for having me.

You know, I always thought it would be fun to work on kernel stuff. I'm curious, what operating system were you working on at IBM? Linux kernel in specific. So we were working on drivers and kernel modules for an MPP processor for network backend inspection. This was my first job out of college. It was an interesting one to say the least. That's a pretty cool start out of college. Yeah, it was really interesting. During college, I was electrical engineering, and there was this hybrid program where you did some software engineering and some electrical engineering. So I started off with embedded systems. And so it was kind of a natural fit, low-level systems, assembly languages to work on the Linux kernel. And that's how I found my start in the world, I guess. Very cool. So we have a couple news articles to go through, Abel, and then we'll start talking to you about backtrace, okay?

Sounds good. Okay, so this first one, there is a Visual C++ switch now to enable standard conformance. And this is one of many blog posts that went up about two weeks ago after the announcement of the Visual Studio 2017 RC but this one's definitely of great importance to the

C++ community that uses the Microsoft compiler they now have this switch which will you know turn on better conformance for C++ developers, right, Jason? Yeah, yeah, and this is something I totally love. Standard conformance from a compiler is a great thing, in my opinion. Yeah, go on. It's still missing a few things, though. They say it's certainly not complete and changing all of the standard conformance things, but it does a lot.

Right. The switch itself is slash permissive minus. And one of the things I do point out, it's still missing a two-phase name lookup and a couple other features, but they plan to continue working on it during the 2017 update and release cycle. So it sounds like we might be getting two-phase name lookup

maybe sometime next year. That's what they're hoping for. And my personal pet peeve of Visual C++ allowing you to take non-const references to temporary values is still not fixed

and is still allowed by this permissive minus flag, as far as I can tell. Oh, that's not good. But, anyhow. Well, it's definitely a good feature either way. And there are a bunch of other articles. Maybe we'll go into these in a little more detail in a later episode. One of the ones I think we've talked about before is CMake support.

It is now there in preview form. And open any folder with C++ sources, so you don't need to have a Visual Studio solution file. It's kind of cool. Yeah, that would definitely be cool. Yeah. This next one, which was getting a bit of attention over the past few days, is ZapCC, which is a faster C++ compiler based on Clang.

Yeah. And it looks like it does this by um by using heavy caching it's got a built-in cache server and we've got two articles here that are doing some performance comparisons between clang gc gcc um and zap cc and it's up to two or three times faster when in debug, and between 1.1 and 1.3 times faster in release builds. And this is just an early preview, so it might get even faster. Yeah, I'm very curious about this. Since it is based on Clang, well, they're making it commercial, if you didn't notice that. Yes.

So it will be for cost by the time it goes into public release but i also don't really understand how it's getting these speed ups if like one of the articles says that they cleared the cache between each test and was still seeing the speed ups yeah i don't quite follow that either you would think it does a good speed up in like your incremental build process when you when you hear the word caching right have you guys ever tried uh basil out of google their build system i've definitely heard of it i haven't tried it yeah when i read this article uh i thought immediately thought of that system and was wondering because basil also has a caching component but it may be on a different level

because it sounds like ZapCC caches internal compiler data structures and data itself with this cache server, whereas Bazel, I think, works on an object and CU level. But curious if any of the community out there has compared the two. That's a good question. Yeah, it kind of sounds like from what you said

that Bazel works maybe more like distcc works or maybe Incredibuild or something. Yeah, I think so. I think you're right there. I thought Incredibuild was more about just paralyzing your compilation. I believe they can also cache object files.

I think so. Sorry, I didn't mean to interrupt. Bazel also paralyzes the build across multiple machines. Oh, that would be handy. I didn't know that. I've only used it offhand for a couple of projects, but my experience so far with Bazel has been good for C++ projects. And Bazel is open source from Google, right?

Yeah. Yeah. Yeah. I'm curious how well a for-cost solution is going to go with the community that's currently using Clang or GCC. Yeah, I kind of personally feel like I've just, in the past few years, reached the world where I can count on everyone having access to high quality compilers for free on all their platforms. And I'm like, I don't want to pay for one again. But that's my own personal commentary, I guess. Well, I'm sure if it's something like a financial company that's

using C++ code to run their servers or whatnot, they'd probably be interested in paying a bit of money to decrease compilation time. And maybe they'll do some free they'd probably be interested in paying a bit of money to decrease compilation time. Yeah. And maybe they'll do some free option for the community. That might be okay. Yeah, that could be cool.

Yeah. Okay, next article is Conan Joins JFrog. And this is obviously a good step for Conan to be getting some adoption from these enterprise products that make use of package managers. I wasn't terribly familiar with JFrog before reading this article. Have you heard of them, Jason? No, I do not believe I'd heard of them. How about you, Abel?

Yeah, I've actually heard of them. There are recent stuff I'm not too familiar with, but I knew them back when I was playing around with Java back in the day. I think what they do now, like you said, package management, also container orchestration and registry type stuff. But for me, it's exciting news because it kind of validates C++ demand and the growing community there. Yeah, it's one of those things where it kind of will cement

Conan as maybe the official C++ package manager to get adopted in a system like this. That'd be pretty cool. Have you guys yet actually worked with Conan? I have not yet. Not enough to say anything educated about it. Okay. I mean, we've

had the developers on here before, and I haven't actually played with it yet myself. I mean, we've had the developers on here before, and I haven't actually played with it yet myself. I need to do that also. But just a little bit more about the JFrog Artifactory, which I did read a little bit about when looking at this article. It shows that they work with all these other package managers like Nuget, Bower, Maven, and NPM. I guess the point of it is to ensure that every developer amongst the team is using the same version of a certain package to ensure that you're all basically building with the same code. So now they'll be doing that for Conan as well.

That's pretty cool. Yeah. So this next one isn't really an article. It's just an interesting question that got a bunch of response from the Reddit C++ community. And the question was, what do you use C++ for? And I just like this because it really

shows kind of the diversity of C++ developers and what type of work they do. There's people saying they work on computer vision and machine learning or game development, music synthesizers, just across the board, anything you can think of,

there's people saying they do it with C++ in here, which is pretty cool. And this is totally appropriate for me because I've just had three different people in the past couple months say, so why would you program in C++? Or is it actually used for anything? And I'm like, well, do you use a web browser?

It was probably written in C++. Yeah, it actually used for anything? And I'm like, well, do you use a web browser? It was probably written in C++. Yeah, it's used in everything. But it's cool to have this whole list of things here. Yeah, definitely. Any other specific ones you wanted to call out? Nothing specific for me. Okay.

I found it interesting that the one interesting use case was platform-independent mobile code. I think someone had said something about the network stack being written in C++ across Android and iOS phones. That wasn't something I expected, just because I'm not familiar with that world. And it was pretty cool to hear that C++ is being used in that case. Yeah, that's actually something I'm pretty familiar with. There was a lot of talk about it I think two years ago at CppCon

when Dropbox did some presentations. They do their shared network and file storage stack with C++ and it's used across all their mobile applications. I forgot about that. And that's what I used

to work on as well, some cross-platform Open source stuff for it. Yeah. And that's what I used to work on as well, some cross-platform C++ code for mobile. Okay. Well, let's start talking to you, Abel. We've been working with you guys for a little bit. You've been a sponsor of the show, which we really appreciate. But let's go into some more detail. Can you give us a little intro on what exactly Backtrace is?

Sure. Just before I start, is this the first time that you had both the sponsor and the guest as the same, I guess, entity or person? We've talked to people from JetBrains before who sponsored the show, and we've had a guest from Incredibuild. So it's not the first time. I thought I was going to be the first. Okay. That's perfectly fine.

Cool. was going to be the first okay it's perfectly fine um cool i think given the audience the best way i can describe backtrace is it's a way it's a product or a project that um tries to help improve both the act and the process of debugging um sometimes we'll say it's a turnkey debugging platform but to really appreciate what that means I like to go through like a simple like thought experiment about what you do when you try to debug. And debugging not only happens in your development environment, it happens in quality assurance, happens in production. And in each of these environments, different stages of the process happen. You know, first we, you know, you try to capture or observe what the erroneous behavior is, and you're trying to capture application state. That might be through logging,

that might be through a breakpad or a mini dump, that might be through a core dump. But the goal is to capture as much information as possible so that you can kind of play Sherlock Holmes and investigate what's going on. In some cases, you won't be able to immediately reduce the problem so that you can resolve it. So you might try to reproduce the problem. And in that case, you continue on the investigation stage. You try to potentially triage it to the proper code owner or subsystem owner, and then you try to resolve the error. So it's clear, you know, in my experience and those of my colleagues that debugging is not just a simple thing. It takes a lot of time and it's a full process. So Backtrace was a project that we started to really improve every stage of that process. Everything from capturing data to providing

an error database where you can upload all of these postmortem assets to be able to coalesce and aggregate them, investigate them at large, and then triage them and integrate them into systems that you use to go through your workflow. So whether it be a ticket management system like JIRA, a messaging system like IRC or Slack, you can plug in this data automatically so that you can improve the holistic process of debugging. There are a lot of benefits, I guess, to a system like this. I actually gave an open content talk at CPPCon this last year about a similar system that was deployed at Microsoft. They described it in a paper called Debugging in the Very Large. And it was pretty cool because first, it kind of proposed this idea that you can treat debugging

as a data analysis problem and not simply just like a forensic or investigative problem. And that was through this idea of having a single point, single system to aggregate and analyze all of the error information from Windows systems, from Microsoft Office, etc. So the talk itself talked about the three kind of benefits, which was automated error diagnosis, this idea of statistics-based debugging, as well as progressive data collection. And that's what we're doing at Backtrace. We were inspired by works like this. We were inspired by similar systems that we built in the past.

So we decided to build this platform to be able to provide this functionality to the community at large across platforms, whether it be Windows, Mac OS X, or Linux, as well as languages. So we support C and C++. And we've had a lot of good experience with the C++ community and people using the platform there. So I'm kind of curious, like, the actual process for how backtrace is used. Like, so you have testers working on your product

and it crashes, say, then what happens? Yeah, that's a great question. So at the time of crash, let's say, you have testers, that information will be captured. How it's captured is dependent on the integration, which I could talk about in a second. And that integration also dictates how much data we're able to collect.

That data is then automatically sent to an error database. This database can be hosted on premise, or you can use our hosted service. But all of these crashes will be in a single database. And what you can then do is have it automatically notify you when a crash happens. You can also log into our interfaces via the terminal or a web UI and view your crash information and investigate it from your web browser. So instead of having to SSH into the machine to look at a core dump or to retrieve your breakpad report from the symbolification server, you can do it in your web browser and investigate this stuff. You can collaborate with your other teammates by sharing links, commenting on the issue, and then you can also have the nice things

of integrating it into your ticket management system, so your bug tracker. So talking about the integration, I mentioned Breakpad before. We also provide a set of tools on Linux, FreeBSD, and Solara Space Systems, where we capture what we call a snapshot of the application. You can think of the snapshot as kind of a minified core dump. What it captures is the stack trace across all threads, local and automatic variables. And then using this data, we actually perform what we call automated debugging. It sounds like oil, but the idea is you perform a set of algorithms to

try to highlight very basic and important signals. So if it's something as simple as like a NullD reference, we can immediately flag that from the faulting instruction as well as the context of the crash. It may be something a little bit more complicated, like, let's say, a double free or a use after free. And if you're using a general purpose allocator, let's say like PT Malik or J.E. Malik, we have the analysis modules in place to be able to leverage the metadata from these allocators to detect those invalid behaviors. So double free, use after free, etc. And this actually was inspired a lot by the work at Microsoft on Windows Air Reporting or Dr. Watson, where they would upload all of this data and their backend servers would perform a lot of this automated analysis. So that when the developers actually began looking at the crash report, they had a jumpstart as to what was going on

from this kind of data analysis that was performed by the collection server. So talking about the integrations that you have, I guess, you mentioned what J.E. Malik and stuff similar. What about the sanitizers that GCC and Clang have now? Can you also process that kind of data? So you can submit that data to the database. However, we don't have a native integration, let's say, with like ASAN or TSAN. The idea with the database is that it's supposed to be a generic powerful database where you can upload auxiliary

data points and metadata points with your data that you can then search across. So whether that be the output from static analysis tools or sanitizers or even log information or, you know, let's say configuration files, all of that can be bundled up and sent and associated with each of these crash reports. That way, when you're looking at something, let's say a week after, you can inspect this stuff and get as much context as possible while you're investigating it. Okay. Does the crash reporting work just during like the development and QA stages, or is this going to be a part part of a released version of an app? It's part of all stages.

So testing, development, QA, and production. Obviously, with a lot of the C++ world, a production build won't have, let's say, debug symbols, etc. And so in those cases, you have a limited amount of information that can be extracted. If you're using something like Breakpad, or if, let's say, you're on Windows and using Minidump, we can leverage your separated symbols file and then do symbolification after the fact. If you're running on, let's say, Linux or FreeBSD, you can also use some of our own tools to be able to extract this data that leveraged debug symbols before it's submitted to the database. So with the, I'm still, you know, I'm just trying to get my mind around

exactly what all your database can do with its correlations and stuff. If, for example, you had multiple testers all submit a crash that was the same, does the database engine, your backend, somehow recognize that these are all the same crash? Yes, so it will recognize that it's the same crash based on the type of crash or the type of fault. And so if it's, let's say, an outright, let's say, 6xv, that could be in the context of the faulting thread after normalization, et cetera. If it's something like, say, a deadlock,

and then you invoke our available tools, then we would use context as to the resources that were in contention to be able to aggregate the crash reports. So based on the type of fault, we'll aggregate their error reports that are submitted. And that aggregation will be able to tell you which ones are unique, and we'll be able to give you time information as to when things happen. You know, a good example is we have users who will notice a crash at some point, and then they don't notice it again for several weeks or months. And then all of a sudden it starts happening even more. And it's usually some sort of memory corruption bug. But they have the system archived.

Sorry, they have the data archived in the system. And so they could easily say, oh, hey, this was the first time we saw it in QA before we released it to production, etc. What are some of the intended users of Backtrace? Are you targeting specific types of developers or specific types of industries? We initially focused on natively compiled languages, so C++ and C. That said, the technology itself

can support a wide array of software. For example, we're releasing support for other higher-level languages like Python, Node.js. But we really focused on high performance, low latency workloads initially because we found that that community was underserved by any sort of new solution out there. There's a lot of niceties that come

when you work in other languages, like on the mobile phone or, you know, these higher level managed languages. And so we really wanted to see that come to C++ as well. And so that's why we initially focused there. But our intended users right now are kind of a wide gambit of people, whether they be hardware appliances, high performance web servers, companies that produce like CDNs, et cetera, to even video game developers and folks that, you know, leverage Breakpad. You've actually mentioned Breakpad several times, and I'm not familiar with it. Can you tell me what that is?

Sure. So Breakpad, it was an open source project out of Google that leveraged the same format as Minidump, which is something that Windows also leverages. And it's just a specific format where you can use the Breakpad library to capture application state in a faulted context. So when you've actually seen a crash or in your signal handler, that data is then sent to a breakpad symbol server, and it's symbolicated. So it translates the raw memory into the stack trace, as well as other information on the system. So things like OS version, etc. And then now you have a human readable report. And this works across your production build. So even if you don't have debunksables

that you're shipping out or you're doing an optimized build, you can still use Breakpad's libraries to extract this information, then have your users submit it to your own servers where it's then translated in something that's readable. There's a lot of projects out there that do ship independent software or remote software that use it.

And that's definitely something heavily used to be able to do this notion of like crash reporting. That's cool. I didn't know anything about that. Yeah, a lot of, I guess, database vendors, I guess, independent software vendors like desktop application makers I've seen use it. It's supported across operating systems. I know it has Mac OS X and Windows and Linux support. It also has pretty good support across processor architectures. I believe it has ARM 64, x86-64, x86, etc. Wow. Let's talk a little bit more about the integration.

If you're writing some sort of C++ application and you're interested in trying backtrace, what type of work would the developer need to do in order to integrate the code? Sure. So not to mention Breakpad again, but if you're using Breakpad, you simply just point the Breakpad library to our servers. So it's a simple change. If you're not using Breakpad,

and let's say you're running on one of the systems that we support with our snapshot tool, the thing that generates a smitified core dump, the tool itself actually performs very much like an outer process debugger, meaning it performs very much like GDB or LDB in the sense that it attaches to this process, it extracts the information. It's not interactive. So unlike GDB, you're not trying to print out this information via interactive terminal. But what you're doing is you're running the tool on it, and it extracts this information and then uploads it.

So if you have, let's say, a watchdog system, you could easily integrate this snapshot tool to be able to capture the snapshot when your watchdog system detects that something wrong has happened. We also provide a library that is C99 compliant so that you can integrate it into your C++ projects so that you can call this out-of-process debugger anywhere in your code, whether it be a functional error or within a signal handler, etc. The last piece is we also can process core dumps. So if you're on a Linux-based system, there's something known as core pattern support, where the Linux kernel will actually

send the core as it's being generated to a separate process. And that process could be Backtrace's core processing tool, which we call CoreSnap. And it will generate this information and send it up to the database. That's also something I was not aware of. Yeah, there's a huge amount of stuff out there in the debugging ecosystem that I guess I only familiarize myself with through experience or through learning about the different ways people debug their systems here at Backtrace. So if in your set of integrations, do you integrate with the source control at all to like somehow tie these bug reports like, hey, they showed up at a particular check-in or something like that? Our product isn't that advanced yet.

We do integrate to GitHub. And so we can, at least within your single source control manager, we can be able to indicate when crashes happen. And we'll also have the source, the function, as well as the line number, so you can tie it together. Now, I think the example that you said was this notion of bisecting commits to be able to pinpoint the actual commit. That sounded like it could be cool. Yeah, it's definitely on our radar.

It's something we want to do. And it's on what we're thinking about later down the road. And this idea of source control integration. So not even working on the level of commits, but also working on just a level of changes as well. So if a file, like let's say your interaction with the memory allocator has changed in some way, and we're noticing a memory corruption bug based on the analysis that the system has done, then can we even pinpoint the change in that way, as opposed to simply just based on time? Cool. Are there any limitations to the platforms or compilers that you're able to work with?

Yeah, so I am sure there's esoteric systems out there that the patch race doesn't work in. I don't want to say that it works with everything okay uh but uh with by leveraging you know open source work like breakpad or uh some of our other stuff we do have pretty good support across all of the major unix based systems as well as uh windows um in terms of architectures, most of the folks that we interact with are on x86-64. But with our support, we also support things like ARM64 with Breakpad, as well as just regular 32-bit x86 with Breakpad as well. Curious, what led to your co-founding this company, this project? Yeah, so it's an interesting story.

First, even though I'm working with Backtrace now as one of the co-founders, my history is in software development. So like most software developers, I know the majority of our time is spent actually debugging code as opposed to writing it. And so there's a lot of frustration just with the process of debugging. I was very used to either debugging the kernel via log statements, because that's the best that I had back then, or, you know,

debugging user space programs via core dumps, or as well as log statements. And there was no system out there that kind of really helped automate the overall process. The actually is the particular event where my co-founder was playing Counter-Strike and it crashed on us. And when it crashed on my co-founder, there was a small report and it asked us if we wanted to submit it to the, I guess, the software vendor. And so we thought, you know, hey, this is actually a much bigger problem than our own little world. And we tried to investigate systems out there that

really replicated a lot of what Microsoft, Google and Facebook were doing with error data, sorry, forensic error data analysis, like Dr. Watson, etc. And we didn't find anything out there that seemed compelling. And so we asked ourselves, can we try to do this and try to provide, I guess, a larger software community, something that brings these benefits to normal software development? And that's how Backtrace kind of started at least initially okay so it sounds like you kind of built uh your own debugger as part of developing backtrace um are there any things you learned while while doing that that might be of interest to c++ developers who as you said use a debugger

every day as a majority of our time sure um i think there there's a lot of good stuff. We did build our own debugger. I gave a lightning talk at CppCon as well. Something that's interesting is when Dwarf is a highly expressive language and specification, even Dwarf 2, it's actually incredible the kind of things and information that you can back out of Dwarf, even if it's a fully optimized build. So things like local automatic variables, et cetera, data structures, those things are still retrievable when you have an optimized build. It's not guaranteed that all your variables are retrievable, but a lot of them are. And so the lightning talk was actually about the differences between the debug information, specifically dwarf generated by CLang and GCC. And when we initially tested it,

we saw a pretty big discrepancy between the two compilers. But with CLang producing a lot less conformant and a lot less verbose dwarf information. But recent versions of CLangs have actually brought it back up to parity. So interesting thing, first thing that we learned was is that compilers will generate different debug information. And it's not something we typically thought about when, you know, we were choosing a compiler or writing a software project. I think the second thing is just how expressive Dwarf is. If I remember correctly, the Dwarf 3 specification was over 200 pages long.

So it's definitely a complicated piece of technology. In fact, I think it's Turing complete as well. So that's a very interesting tidbit as well. I think the last thing is just people using the product and the project itself. There's a lot of low-hanging fruit, I guess, in the debugging process.

And the only time people typically think about it is when they're actually in the act debugging. But a lot of people have made investments at larger companies that people just don't see the benefits of, but until you have those resources. And so something we learned about building that debugging platform is that even the small things like automatically filing tickets can really improve the overall process of resolving bugs and debugging itself. So this whole time, we've pretty much been talking about crash bugs, it sounds like. Do you have any support for like, not a crash bug, but like, hey, something really weird is going on right now, and I'd like to send a status report back? Yeah, so our two things, the debugger itself, the thing that takes the snapshot,

you can run at any point in time during the application's lifetime. So it could be during normal steady state behavior that you run this. In addition, we provide a library that lets you invoke this within the program. So if you're able to detect, let's say, via a watermark or some sort of anomalous behavior in the system itself, you can actually programmatically invoke a snapshot, and that would be sent up to the database. And it's in the database. You can search against it. You can retrieve that information and view it via our interfaces. And so people have used this to not only debug, let's say, functional bugs, but also, let's say, like behavior that

needs to be observed over time. So it's kind of the classic way of using core dumps, or people used to use core dumps to be able to debug memory leaks by taking the core dump and traversing the object graph to be able to see where things are consuming more memory. And in that same way, Backtrace would be able to capture this information and you'd be able to inspect it to be able to do this kind of analysis as well. I imagine you could also maybe use it for performance analysis if something is taking longer than it would be expected, maybe find out where the code is spending time right now using the debugger.

That's definitely a use case we're thinking about, this idea of kind of a sampling-based profiler. Definitely, we're not the first people to do it. There's a lot of profilers out there that mimic a similar approach. But right now, we're focused on this idea of whether it be functional or fatal error reporting and being able to dig deep into those instances of time as well. Okay. Is there anything else you wanted to share? I guess if developers are interested in Backtrace, where should they go to get started or learn a little more information?

Sure. You can go to backtrace.io.cppcast and we've got some information there. We have a set of documentation. There's also a form you can fill out to just try out Backtrace for free.

We give out free trials and if you're we're actually just now releasing support um for a lot of different things so if you're on one of these other platforms let's say using breakpad um then more than happy to uh give you access to the free beta as well which is just an extended trial where you can try it out um at your leisure while we're in the beta phase of the product itself. Okay. And where can people find more information about you online, Abel? My Twitter

account is nullisntzero. And most of my content that's out there is under the Backtrace name. So if it isn't null isn't zero, then it's just normal Abel Matthew or it's just backtrace. That's usually where you'll find me.

Okay. Well, thank you so much for your time today. Yeah, likewise. Thanks for having me. Thanks for the great podcast as well. Thanks for joining us. Thank you. Thanks so much for listening in as we chat about C++. I'd love to hear what you think of the podcast.

Please let me know if we're discussing the stuff you're interested in. Or if you have a suggestion for a topic, I'd love to hear about that too. You can email all your thoughts to feedback at cppcast.com. I'd also appreciate if you like CppCast on Facebook and follow CppCast on Twitter. You can also follow me at Rob W. Ir and jason at left kiss on twitter and of course you can find all that info and the show notes on the podcast website