Software Huddle - No More Broken Docs with Manny Silva

Episode Date: July 30, 2024

Today, we have Manny Silva, Head of Docs at Skyflow, on the show to talk about two open source projects he created, Docs as Tests and Doc Detective. Docs as Tests is a framework to make sure that your docs are in sync with your product. It's essentially a way to test your docs just like engineers test their code, and Doc Detective is Manny's implementation of that framework. We discuss the history and motivation behind the projects, what they enable, and how people are using it today. Timestamps 01:57 Intro 06:51 Testing Documentation 09:26 Competing Against 11:26 Docs as Tests & Doc Detective 13:32 How does one apply these ideas? 16:49 How does test writing work? 19:26 Out of the box checks 23:15 Configurations Structure to create tests 24:28 Integration with the normal flow 28:20 Freshness 29:13 Tools used to build it 32:02 Open Source 33:27 Limitations 35:31 MongoDB's version of Docs as tests 36:42 Innovation Engine by Azure 37:27 Teams using Doc Detective 38:12 At Skyflow 40:52 Future 41:41 How to get started 45:18 Rapid Fire Links Docs as Tests: https://www.docsastests.com/ Doc Detective: https://doc-detective.com/

There is a concept in the tech doc space of freshness check, where, you know, a lot of companies have time set aside every year to go through and make sure that absolutely all of their procedures are still accurate. Well, I'll let you in on a dirty little secret. Freshness testing almost never happens because something comes up and people decide that their time is needed elsewhere. When it comes to documentation, your actual direct competitor is what is the best in class documentation that exists out there? Because the same people that

are probably integrating your product have had those experiences where they've integrated, you know, Stripe or something like that, where they've had, you know, a truly magical experience. So that's really what the baseline of what they're comparing you against, not necessarily your direct competitor, which puts a lot of pressure on or additional pressure on organizations to really try to invest in high quality documentation far more than maybe what we saw, you know, certainly a decade ago. 100% everyone says that they invest in good developer experience. Few people actually do.

What was involved with actually developing Docs and stuff? What were some of the tools and frameworks involved and technologies that you used to build it? Hey everyone, Sean here, and today I have Manny Silva, head of Docs at Skyflow, on the show to talk about two open source projects he created, Docs as Test and Doc Detective. Docs as Test is a framework to make sure that your docs are in sync with your product. It's essentially a way to test your docs just like engineers test their code. And Doc Detective is Manny's implementation of that framework. We discuss the history and

motivation behind the project, what they enable and how people are using it today. Manny's a really smart guy. We worked together previously at Google. I've always said that Manny knows more about Google documentation than most people know of anything. He's a really smart guy. We worked together previously at Google. I've always said that Manny knows more about Google documentation than most people know of anything. He's an absolute expert. And as always, if you have suggestions for the show, please reach out to me or Alex. And with that said, let's get over to my conversation with Manny. Manny, welcome to Software Huddle. Hey, Sean. Thanks for having me.

Yeah, thanks so much for being here. So we're talking like all things docs, documentation today, in particular testing docs. And, you know, before we get into testing documentation, I wanted to get your thoughts on, I think, this like transformation that's kind of happened with technical documentation. I would say I might be getting the timeline on this kind of wrong, but maybe like the last decade or so. At one time, I think, I think what documentation was, I mean, this is like 20 years ago, but it was like a printout manual that company used to distribute. And really like, really the nerdiest of the nerdy spent time out reading those. And, you know, from there, I think companies started to really invest in documentation and kind of like turn it into a product, a source of pride. We gen community love. Companies like Stripe stripe i think really blazed a trail here twilio to some extent what are your thoughts on that transformation and in the impact and has it had an impact on the

technical writing profession oh it's had a tremendous impact on the technical writing profession i mean going back to the paper manuals that you mentioned, yeah, that's where it all started. And those were in the domain of desktop publishing. And that transformed into in-application help. Windows users pressing F1 might be very familiar to some of you. And then we moved into online help, but it still used those same basic systems. All the same, technical writers were learning new skills and softwares along the way. IBM introduced DITA a good long while back now for an XML-based documentation management platform.

But there hadn't been a big shakeup in that domain for a long time either. But when web technology started democratizing access to web development for everyone that also applied to documentation. And so we started seeing online help. We started seeing web developers, engineers authoring their own help systems instead of getting into the technical writing domain. And that's only increased with the advent of docs as code using markdown-based documentation, what a lot of folks are familiar with doing on GitHub and other code repositories. And even into the most modern flavors of this

with tools like DocuSource and MDX that are able to embed React components directly in documentation for interactive experiences. It's constantly changing, and that's one of the things I love about the job. Yeah, it's kind of, if you look at the early web, a lot of the early web was taking things that existed sort of in the paper world and like, oh, we're just going to create like a website version of this thing, suddenly. And it took a while for it to go, you know, certainly to get from the place where it is now we're rebuilding like really,

really powerful, whole scale applications like this online. In some respects, like documentation started there as well. And then it's maybe lagged a little bit behind the transformation that happened. But now, we're caught up where docs can be this like interactive, living, breathing product experience for people. And of course, with such an, uh, so many companies that are sort of developer first developer tooling companies too, that, um, has, uh, increased essentially the audience that is available to consume these types of this type of information too.

Oh, completely. And the types of experiences that documentation is able to offer are dramatically improved. I mean, even looking back at just at API records, I mean, heck, APIs have grown up alongside documentation. And how we communicate information about those APIs has changed. We go from people typing out a couple of API endpoints and just saying, here, go for it, best of luck, to standardized API specifications and then having those interact with documentation sets to have interactive API references and being able to dynamically generate, oh, hey, what would this look like in Python?

What would this look like in whatever other language you're interested in? Yeah, so I wanted to also talk about, I think we're going to dip into a lot of stuff related to testing, documentation. There's been a long history, though, with testing software. There's tons of different types of tests,

like unit tests, integration tests, security, performance, the list goes on and on, end-to-end tests. I don't think... Obviously, not an experience that you have, but it doesn't feel like there's been as much effort and thought process

put into testing documentation as there has been to testing software beyond someone maybe doing like a peer review of like an MD file that someone's, you know, publishing before it goes public. So how much does, so first part, I guess, like is how much does incorrect documentation impact essentially or negatively impact like the company and product? And then as a follow-up to that, what are some of the historical methods for actually testing documentation?

Sure. So I'll actually tackle the second one first. The historical method of testing documentation is somebody sitting down and stepping through the procedure. And that is incredibly error-prone. Because if it's the person who wrote the documentation, then they're too close to it. They know what they intended to write

rather than what they actually wrote. If it they know what they intended to write rather than what they actually wrote if it's someone who is doing a peer review like you mentioned earlier it's uh you are hoping that they actually go through and do it and they're also not too familiar with the product uh or procedure you have user testing which ends which takes a lot of time. It takes a lot of user effort to set up an interview to do that in the first place. And it's difficult and cumbersome, and it's better for research rather than content validation. But all of this is really important because getting to the costs of it, you have both direct and indirect costs. The indirect cost is, let's say you have a prospect who's reviewing your project product to see if they want to adopt it. Well, if they see a typo, if they see that a

string for a button is wrong in the docs compared to what they see in the UI, if they see outdated screenshots, that doesn't exactly inspire confidence in your project and they are less likely to adopt it if you have existing customers who are running into this it undermines the trust that you've built thus far and if they keep into running into these things then that is a substantial cause of customer churn which nobody like but you also have the direct cost uh so let's say that you do have a customer who runs into an issue and they decide to record it.

Most people won't, but this one did. They report it to customer service or customer experience. Customer experience then has to ask questions of the customer to make sure they got it right. Then they reach out to the documentation team. Documentation team has to ask more questions of customer experience

and reach out to people internally, whether it's UX or engineering or whoever else. And there's this cascading effect of wasted time when you could have caught this issue earlier if you had done some content validation test. And so whether it's from outright lost prospects, outright lost customers or wasted time internally, broken docs cost a lot of money. And I think also, one of the things that I think companies sort of neglect sometimes or not think about is, like, who are they competing against when it comes to documentation? Like, a lot of times when we think about competitors, we're thinking about who's our sort of direct competitors,

it's going after the same sort of customers as us. But when it comes to the documentation, and sort of developer experience, your actual direct competitor is what is the best in class documentation that exists out there, because the same people that are probably integrating your product have had those experiences where they've integrated, you know, stripe or something like that, where they've had, you know, a truly magical experience. So

that's really what the like, the baseline of what they're comparing you against not necessarily your direct competitor, which puts a lot of pressure on or additional pressure on organizations to really try to invest in high quality documentation far more than maybe what we saw, you know, certainly a decade ago. David Gellman, 100% everyone says that they invest in good than maybe what we saw, you know, certainly a decade ago. 100%. Everyone says that they invest in good developer experience.

Few people actually do. They grab the newest documentation framework that has the most excitement surrounding it. They apply it and then they say, great, we're done. And they wash their hands of it. That's far from it. There's a lot more to developer experience, even just a documentation than that, as far as what it contributes to the overall experience.

And yeah, Stripe, Twilio, these incumbents of excellent DX have also been pushing, continuously pushing the forefront of what is good documentation. But there are a lot of people who are starting to challenge that in various different ways, whether it's in interactivity, whether it's in content validation, whether it is in the ways in which they're able to surface novel pieces of information. A lot of people are trying to, you know, make a name for themselves. And I think doing a rather admirable, admirable job of it. You're the creator of a project called Docs as Tests and also the doc testing tool, Doc Detective.

I want to get into the details on these, but can you start by giving a bit of an overview of both projects? Like how do they originate and what's the relationship between the two? Sure. So, uh, they originated out of my frustration because I am a technical writer of, I'm not going to think of how many years anymore, but I always got frustrated Oh, hey, that UI or that screenshot is out of date. And I go and do some digging and somebody changed a very nuanced behavior of an API call or somebody updated this UI and didn't bother to tell me

even when I have very good communication channels. Something always breaks. Something always slips through the cracks. And I was frustrated. And I talked to my engineering friends and I said, hey, you've got all these really cool tools for validating your code on going. What is there for docs? And they laughed and said, you, Manny, that's why we have you around. I said, oh, no, it isn't.

And so I took some time to try to solve this solution. And what I ended up with was this two-piece solution of Docs as Tests and DocDetective. To break down what each of these are a little bit. Docs as Tests is a tool agnostic strategy for treating your documentation as testable assertions. Meaning using whatever tools you are most comfortable with. You can validate that your documentation procedures are accurate by running those procedures against the product. DocDetective is my tool for implementing docs as tests.

It is open source, and its target audience is tech writers, such that the least technical tech writer can be confident in their content. I get sort of in principle, like the idea behind it, but can you like walk me through how do I actually, what is the instantiation of this framework? Or what is this kind of like breakdown for somebody who wanted to actually apply this framework to the docs that they're working on? So there are a few ten-pole ideas that pin up docs as tests. First is that docs really are tests. Documentation, especially procedures, are repeatable. You can validate that, yes, this procedure works as written. If you can validate it, it is therefore testable. If A equals B and B equals C, A equals C. Docs

are tense. And in addition, these procedures need to be repeatable. There's a reason we document them. It's a way of deferring support costs because this applies to everybody. And so these tests need to be repeatable in whatever environments are applicable. Now, taking those two ideas and a third idea is that these doc-based tests complement. They don't replace. They are not trying to test your code. That's what all the engineering tests are for. The engineering tests do not test the documentation itself. They test the behaviors underpinning the documentation. And so by having this as one more kind of test, you can validate that, yes, indeed, you have a consistent UX between your product and your documentation, whatever you're telling your users. Now, what it takes to apply docs as tests is to, first, you have to identify what sort of tools you're most familiar with and what is most appropriate for the product surface area you're

trying to test? Is it a UI? Is it a CLI? Is it an API? Is it an SDK? What are the various tools available to test each of those different surfaces? So if it's a UI, it might be Cypress or PlayRunt. If it's an API, it might be Postman, I'm not going to get into all these tools, but you need to identify what is most applicable to you. Then the tricky part, you need to start writing tests that are parallel to your documentation procedures. And the reason I say this is the tricky part is because you're going to have a very hard time convincing engineers to take time to write tests for their documentation. This really needs to be the domain of a tech writer because a tech writer cares about the

capitalization of the button string, not just that the button string is there. You need to be able to get down to that level of nuance, but also understand where you need to take a screenshot and where you don't, and where you need to embed a GIF and where you don't. And so you need to have a tool that taps into your documentation and is usable by your tech writers. So once you can identify those,

then you can really start getting test writing going. Okay, so if we take a specific example, like let's say I have some sort of guide that has, you know, five steps in it, essentially, in order to perform some sort of action. Maybe it's like, you know, log in here to this management UI, click this

button, enter this thing into a form, click another button. So how does essentially the test writing for that guide work? That depends entirely on your tool. If you're using something like Cypress, then you would write a test definition

like you would for any other JavaScript or TypeScript test. And then you would say, oh, hey, go to this URL and validate or assert that, yes, you were able to go there. And then for each and every step of that procedure, you would write an assertion or an action and then assertion to validate that the action was performed correctly. With something like DocDetective, it simplifies it pretty substantially, such that you don't have to write test assertions. You don't have to know JavaScript or TypeScript or how any of that work. It's just a JSON object at heart. You write a single test in JSON, and then each and every step that you tell it or action that you tell it to perform, it will validate if it performed correctly. It can validate if an element on the page exists with the string you expect it

to have. You can take a screenshot and compare it to an existing screenshot to see if the UI has changed more than an acceptable threshold. And you don't have to worry about writing any sort of code. But the best part about Doc Detective is that it directly integrates with your documentation source itself. So if you have content written in Markdown, then you can directly declare the test actions that you need to do within the Markdown copy so that if a tech writer is going in and updating one of those five steps, or if they add a sixth, then they don't have to remember, oh, hey, I need to go and update the test file. They can update the tests directly within the Markdown.

Or even better, you can also set up configuration files such that it reads the markdown from your source content and it programmatically detects and generates tests so that you never have to say, oh, hey, for this hyperlink, I want to do a check link action.

It just says, I see a hyperlink. I'm going to generate and execute a test, a check link action for you. Okay, so I want to get into the details of the doc detectives a little bit. So essentially, it sounds like that with doc detective, there's sort of some out-of-the-box, essentially, checks in place,

like checking that a link is valid, maybe checking that an image is a valid URL, this type of thing. What are some of the other sort of out-of-the-box checks? So DocDetective has a set of actions that you can perform to take a step back from those in particular. DocDetective can test web-based UIs.

It can test APIs, and it can test CLIs. And if you have a unit test script, it can test SDKs as well. The as far as the actions that it supports focusing on the UI elements so far you can tell to navigate to

a specific URL. You can have it find an element on the page and capture some of the substring of whatever text the element is displaying to as an environment variable to use later on, you can tell it to type keys. I mean, this is really at its heart, simple UI automation type things just simplified such that you don't need to know how to code to do it. But there are a few other things, one of which I mentioned,

you can capture screenshots. And if you've already run this test before and you've captured this screenshot before, then it can do pixel diffs and make sure that, oh, hey, this didn't change too much. But what that really means, if you stop and think about it, is you can programmatically capture all of the screenshots for your documentation. You can take these captured screenshots and service them to your users. And then on an ongoing basis, DocDetective can let you know if something needs to change, and it can capture the new screenshots for you. So you don't have to spend all that time figuring out how to replicate that screenshot similarly uh it can also do uh recordings so if there's a particular procedure or a snippet of a procedure you want to

have a gif for doc detective can do that it can record that test execution output a gif that way you can have uh motion and video elements in your documentation as well. And that's rare. But additionally, the, you know, running arbitrary HTTP requests. So you can test the API, whatever APIs you want to, and making it a little simpler, doing check links. So to make sure that, yes, like you mentioned, a hyperlink or an image reference is valid. For the API request, the HTTP requests,

you can also validate that, oh, hey, the response payload included at least or exactly this field invalid to make sure that, yes, it not only returned a 200 OK, but it responded with what I expected it to. And also capture elements of the responses to do, uh, as environment variables to use in subsequent steps. So if you create, if you're testing a CRUD API and you create a resource

and capture that ID to delete it or update it or whatever else later, as far as, uh, CLI goes, it can run shell commands so that whatever CLI commands you might want to run, you can validate, oh yeah, the output did include what I expected it to. And for both API

tests and CLI tests, you can save the output to file. So much like the screenshot diffing, you can diff the responses so that if, oh, hey, this API returns something slightly different than expected, the tests can flag that for you

so that you don't have to discover it yourself. And as a user, I'm defining this all within a JSON file or am I creating essentially a different JSON file for each test? What's the configuration structure to create some of these tests? So, uh, doc detective has a global configuration object that it uses, and that is where you define the patterns for automatically detecting tests.

So for example, if you wanted to make sure that every single hyperlink had a check link action, that's where you would define. As far as the individual tests are concerned, that you could define either in a syntax, but define it inline in comments in whatever your documentation source is, either are perfectly acceptable. And what doc detective can then do is you just point it to an input file or input directory, and it will crawl the directory and the file, parse through everything that it finds, and dynamically build whatever, compile whatever test it detects and then go what's your expectation i guess around like how this gets integrated integrated into the

normal uh like flow of essentially like commit cycle to eventually like publishing documentation so funny thing about technical documentation is there is no standard procedure. Everyone does it a little bit differently. And DocDetective was planned to fit into as many people's workflows as possible. So DocDetective can run on whatever development device you have. It can run locally. Or you can run it in CI. There are a few different deployment methods. Today, we have a NPX based command. So as long as you can run Node, you're good. And that's what is easiest to run as a general recommendation. It's easiest for debugging as well. It does run on Mac, Windows, and Linux.

As far as CI is concerned, we have a GitHub action that's currently in Release Candidate and open to BGA too. And if that doesn't work for you for whatever reason, or you do have dependency issues with Node, there is a Docker image that is currently an open album.

Uh, so don't have to worry about dependencies there, but as far as where it fits in from an engineering perspective in CI, I would, I generally recommend you write tests. When you write your docs, you vet your tests on device as you're writing your docs so that before it ever gets merged, before it ever gets into a PR, you know that the tests are good. Then when it gets in, then for every PR, you run your doc tests against it to see if anything breaks. Now, this could be if you're feeling really adventurous in your development environment. This could be just against your docs if they have their own repo. That's up to you.

But I generally recommend every PR run your doc tests. Additionally, every time you push into a staging environment before release and you're doing all of your engineering testing, you should be doing your doc testing too. Because if something changed, you should catch it before it goes out to production and users might ever see absolute worst you should be testing every time you have a production release um that way you know at least what's live you can make sure is here um and if you are feeling adventurous you can also test just on a schedule on a cron so that uh you can help monitor for production issues uh you know the again these tests complement they do not replace there are other tools that do this but an extra safety net never hurts um and i have heard of people who have got issues

like this before so yeah you kind of with documentation there is uh sort of separate systems in some ways that can actually impact whether docs work or not like obviously you can be writing documentation and you could break something because you make a mistake but then the actual thing that you're documenting which is probably like a completely different separate system that is run by engineering and product is evolving independently and they can break your dots as well and also where i mean to your point essentially like, like integrating docs, like the testing loop there is important as well, so that most likely it's the evolution of the product

itself that's going to break the documentation, because presumably the documentation will correct at some point. I would hope so, yes. And that's why it's important to test frequently. It's important to test every time you're making a doc update. And to my point earlier, that three, six, you know, a lot of companies have time set aside every year to go through and make sure

that absolutely all of their procedures are still accurate. Well, I'll let you in on a dirty little secret. Freshness testing almost never happens because something comes up and people decide that their time is needed elsewhere. And what Doc Detective, what Docs as Tests lets you do

is automate your freshness checking so that, well, the computer can do it for you, and it can do it on a daily, hourly, if you really want it to, basis. What's involved with actually developing DocSysTests? Like, what were some of the tools and frameworks involved and technologies that you used to build it? So I've gone through a couple of iterations of doc detective at this point.

Uh, but today we use, it is a node JS project. Uh, it is written in JavaScript. Um, the is deployed to NPM. So you can do a global install or run it from NPX directly, but I'm using a stack of a few different layers first i have the test that so we can start from the tests perspective since that's what we were talking about um when i Or when you run doc detective, the tool goes and parses the files. So let's just say markdown for the sake of conversation. And that, nothing terribly special happens.

It iterates through line by line, detecting tests and pattern matching to generate tests. It then compiles all of that into a gigantic JSON object or a set of JSON objects of tests to run. At that point, it uses Appium, which is a Linux foundation open source project for automation that's most known for iOS and Android testing, but it also handles web automation and even native automation testing as well. And so I spin that up on the packet. Appium communicates via Selenium tests as an intermediary.

And so as a client, I use WebDriverIO, and I translate the doc detective tests to WebDriverIO tests, which then speak Selenium to Appium, and Appium is what actually spins up the web browsers and performs the tests themselves. The DocDetective manages internal Firefox and Chrome binaries. It downloads them at installation time from the approved known sources. And that way you don't have to mess with your local install at all. Now, it does also support Edge and Safari browser testing. If you need to target those, um, there is also a web kit based Linux

browser that I'm working to enable automation for, and that would be separate from app. Uh, but those are the most common tools in my stack. And then this is a Microsoft project? Yes. DocDetective is 100% open source. Every aspect of the project is. The way that it's actually structured is there is

DocDetective Common, which is where we house all of our JSON schemas and where we have our validation logic for those JSON schemas because that's used across a variety of other cell repos. There's DocDetectiveCore, which is an NPM package that you can integrate directly if you want.

And that's what contains all of the testing logic for DocDetective. The main DocDetective repo, also an NPM package, this is what gets run via the NPX command. Handles the I.O. of it, looking for the config and the source files and that sort of thing. And this is the main way that people are going to interact with it. But there's also the GitHub action and the Docker image that I mentioned earlier. Under the hood, they just run the NPX command for doc detective, but in their more limited environments. And there's also a browser extension because finding CSS selectors is hard. And so that was my first attempt at helping people

identify the correct, the shortest, unique, valid CSS selector for an element that they can use in their UI testing. What are some of the limitations right now? Right now, the biggest limitation is that it doesn't support native app testing. You can run CLIs, that's fine, but you can't do UI testing for native apps. That's honestly part of why I went with Appium, because it has that capability.

I haven't built it out yet for DocDetective, but I fully plan to one day have DocDetective support in addition to web-based UI tests, native Windows, native macOS, native Android, native iOS, as well as CLI's APIs. I also want to have better auto detection of bash commands and of API-based tests from Markdown files and other different kinds of documentation source files.

You can do it today, but right now, frankly, writing those tests is the hardest part. Same with the CSS selectors that I just mentioned earlier. You have to know what you're looking for. You have to know how CSS selectors operate. And I have a few POCs in the work that will help with that, but those are the current pain points. And I guess like around that, like know what to look for with the CSS, that becomes somewhat of a challenge for the, you know, technical writer, there's maybe less technical that maybe doesn't know all the ins and outs of CSS.

Yes, I mean, I'm a software tech, I have been since I started writing, but there are a lot of writers out there who aren't, who are dealing with native app, native OS application, tax tools, and other such things who aren't as neck deep in it as I am. Uh, and I want doc detective to eventually support them too. They are my target audience. Uh, Oh, I forgot one other pretty glaring, uh, drawback, especially considering the target audience. Right now, Doc Detective is a CLI-only tool. You have to run the NDPX command.

It doesn't have a GUI. Besides, Doc Detective is your sort of instantiation of the Docs as Tests framework. Has there been anybody else that have implemented their own version of Docs as Tests? Yes, yes. And I am so happy to have found some of these people. So I put a name to Docs as Tests? Yes, yes. And I am so happy to have found some of these people. So I put a name to Docs as Tests. But there have been people who have been doing flavors of it for a while.

Most notably is MongoDB. They have a tool called Bluehawk, and they were talking about unit testing your documentation a few years back. It is Docs as Tests in everything but name. And documentation a few years back. It is docs and tests, docs as tests in everything but name.

And their strategy is for any sort of procedure that you should be writing unit tests for. You should be, you know, executing the code you are asking people to execute and, you know, running it as a unit test anytime you are pushing your docs. And that is wonderful for any procedure that includes writing code or running code. But it doesn't help people who are making API calls. It doesn't help people who are making or have UI based procedures procedures. Similarly, Microsoft Azure, just earlier this year, released a tool called Innovation Engine

that can take a markdown source file and detect commands from bash code blocks and run them and validate the outputs. But again, it runs into the same issues as the other style of unit testing. It's very restrictive. The tool that has been around for a while is Runme at runme.dev, but they're more positioned as cloud-native workbooks

that are not restricted to Python. The workbooks or notebooks are great, but they, once again, don't quite fit the bill for technical documenting. And in terms of usage of DocDetective, do you know of teams that are actively using that project? Yes. So we are. Skyflow. I use it in my day job. Surprise. And we have a lot of tests under the hood that are actively being validated.

There are a few other folks who are out there using it, like Red Panda is using it in some of their labs that are out there. And I have a number of other companies who are interested in doing POCs right now. But frankly, I don't know the full scope of it because this is an open source project and it is out there for whoever wants to use it to use it. Yeah, it can be for everyone to hear about. There's 10 other people that are using it that have never told you.

In terms of when you turn this on at Skyflow, how many mistakes did you catch? In our docs? Yeah. Do you have a number? I mean, what was interesting about it is that the mistakes... In our docs? Yeah. Do you have a number? I mean, what was interesting about it is that the mistakes weren't, like, there were misunderstandings. Like, I thought, oh, hey, this feature worked in XYZ way. And so I documented it in XYZ way, but actually it behaved in X, Y, A way.

And so being able to go back, have these discussions and say, oh, hey, is this the best developer experience that we could offer right now, both from a documentation perspective and from a product perspective and being able to use that as a source of conversation. Um, but it has definitely refined our documentation procedures. There's absolutely no doubt of it. And we are not at 100% coverage yet, but I hope to get there. What's the reception there? In particular, I think there's people like yourself that have worked as a technical writer, but sort of exclusively on technical technical like software projects there's a whole class of technical writers probably the majority of technical writers to be frank that have worked on more

sort of like business applications like do you get a different reception depending on like which camp people fall into oh 100 yes uh to i i have to heavily tailor my messaging depending on who i'm talking to uh and so if i'm talking to more software-focused folks, then they get. And the biggest complaint that I get is that there isn't a GUI because even if people can use a CLI, you always usually make things easier for this sort of task.

Or the people who are more used to the business application stack sort of thing. It takes them a while to understand are more used to the business application stack sort of thing. It takes them a while to understand why and how you would test your docs because they're not used to engineering testing or what goes into that. They don't even have a concept of it. So I have to take a few steps back and say, okay, so you've at least heard of docs as code. This is somewhat parallel.

This is what engineers do to test their code. This is why Docs as Tests and Doc Detective is important to test your code. And now we can finally get into how do you actually do this. It takes usually about twice as long. But everyone, by the time I get there, is very invested and very interested in testing their content. But again, especially from the latter camp, the cries of give me a GUI, please, are louder. In terms of Docs as Test Framework, what do you see as sort of the future of that? What are you trying to evolve that into? I want everyone, every technical writer, everyone who fills a technical writer role, regardless

what their title is, to be able to have confidence in the accuracy of their docs. And I will push that forward myself with Doc Detective, however much I might. But I genuinely want to see other tools come out to solve other aspects of this larger issue. And again, there are all of these wonderful, wonderful engineering targeting tools that can do this automation, but there is very little

that targets a non-technical audience that's also comprehensive enough to do everything a tech writer would need to do. Yeah, I get that. And then, for people who want to get started with using, you know, Docs as Tests or potentially Doc Detective, like, what do you recommend in terms of, you know, where to jump in? So there are a few resources I could point you to. There's Docs as Tests dot com, which is a blog that I manage that details different pools you might use for

docs as tests and different ways to test different products, different use cases. We have a very popular post about how to run Doc Detective in CI. But there's also docdetective.com or doc-detective.com, which is the main documentation site for my tool. We also have a public Discord that you are more than welcome to join to ask questions, whether about Doc Detective or Docs as Tests more generally. As far as how to actually get started, find one procedure, something possibly part of your happy path for whatever your product is and try testing that start small, start as small as you reasonably can. Uh, and as easy as you can to get your feet wet and then build up from there. Progressive. Uh, once you have something that you can show, show the rest of your team. If you're a software engineer, show your tech right. Uh, and see and say, hey, look, this is how simple it is to actually get going.

And here's how you write text. If you're a tech writer, bring, you know, build out just a few tests and then take it to your engineering team, your quality assurance team and say, hey, look, this is a different way we can validate our content. Is this something you'd be interested in partnering on? You know, this is an engineering is a team effort and documentation is a part of engineering. So show each other the value, lean on each other, and you'll have, you'll find success. Do you see a way for this to work?

I just thought of this, I don't know if it makes sense or not, but like comments in code bases are also a form of documentation usually. And typically like engineering came through that, but that's also another area where things get out of, out of date. Do you, could you see essentially an application there as well? Oh, 100%. And there are a few applications there as well. I mean, Rust has the Rust doc command. Python has the doc test command. There are versions of this command, whether native or

third party for most different languages. And what you can do as an engineer is make sure that you are actively running these commands as part of your CI. So such that you can validate that, oh, hey, this snippet that's eventually going to get exported somewhere else actually work. It works as written. That way, whether the documentation is coming from a code, whether the documentation is coming from an open API, whether the documentation is being written in Markdown, it doesn't matter where the source is.

There are ways of validating it. And speaking of open API specs, there are lots of tools out there, like Redocly, that can help you run linters over your OpenAPI specs, both for the formatting and for the grammar, to check for style influence. But yes, yes, yes.

Content that originates in code is still content, it is still documentation, it still needs to be tested. So once again, this is a team sport. And if you need help writing content that lives in your code, great. Go find your tech writer, bring them into the code, show them what needs to be edited. Then we'll be more than eager to. All right. So I have some, as we start to get to close here, I have some quickfire questions for you.

Okay. If you can master one skill that you don't have right now, what would it be? Oh my. Being an airplane pilot. Being an airplane pilot. I like the idea of being able to fly. And you know, airplanes are the closest I'm going to get. Waste the most time in your day.

Sorting through slack. Sorting through slack. I'm going to get waste the most time in your day sorting through Slack sorting through Slack if you could invest in one company that's not the company you work for who would it

be today it would oh that's a very difficult question uh crash crash oh no thank you

uh I would say maybe speakeasy to keep it in the DevEx space. What tool or technology could you not live without? Obsidian. I could not live without my note-taking app. Or VS Code. I edit everything in it, whether I'm writing content or code. Which person has influenced you the most in your career?

My very first manager as a tech writer. She helped really inform the fact that documentation is important. It is legitimately very important, but it is still just words. Just like for an engineer, it's still just code and the world will continue to go around. And that has helped me keep my sanity

more times than I can count. All right, last one. Five years from now, will there be more people writing code or less? I think there will be more people writing worse code. Why do you say that? AI proliferation and more people get, I genuinely say that positively

because I think more people will feel empowered to write code, but they will still be learning and they won't know what good and bad patterns look like. As we wrap up, where can people follow your work, stay up to date? You mentioned some URLs. Do you want us to just run through those again? Sure. To follow me and what I'm doing, feel free to reach out on LinkedIn. I'm a very friendly guy, I'm told. As far as docdetective.com, that website is doc-detective.com for the project's doc site. For Docs as Tests, we have docsastests.com, which is the blog

to follow implementations of Doc Detective and other tools to satisfy the DocSys test framework. All of the DocDetective repos are open source, including the documentation. So, hey, tech writer who's happening to watch this, you can contribute too. Don't think that if you can't write code, you can't contribute. Everyone can do this. And we have that public Discord. So feel free to join and ask questions.

Awesome. Well, Manny, thanks so much for being here. This is really fun. Yeah. Thanks for having me, Sean. Cheers.