PurePerformance - ChatOps: Automate yourself into your next job with Nestor and Zohaib from Citrix

Episode Date: October 28, 2019

ChatOps is not new! But many organizations have not understood nor leverage its full potential. The use cases spread from “What’s on todays cafeteria menu?” to “Deploy my latest Git commit as canary and scale based my SLOs!”.Listen to this podcast and learn from Nestor Zapata and Zohaib Hassan – both working at Citrix – on how they have started their ChatOps journey, how the built trust in the technology and how it helped them transform their organization towards more autonomy thanks to the self-service model enabled through the chat bots they developed. We discuss many of their self-service use cases such as Performance as a Self-Service or even Self-Healing which they implemented through Chat Bots integrated with Slack, Dynatrace, ServiceNow, Jira and other tools.If you want to see their chat ops in action watch our Performance Clinic on Automate Deployment and Site Reliability with Bots, ChatOps and Dynatracehttps://www.youtube.com/watch?v=xE_LMQ9u7l4

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. My name is Brian Wilson and as always, my co-host Andy Grabner. Andy Grabner, how are you doing today? I am pretty good, my friend. I just hope that at some point in the future, my voice will sound as great as yours when you announce the show. Well, you know, when you start using that other mic, you sound a lot better. But I don't think you're, it sounds like you're not at your desk with a microphone right now.

I think we're going to have some, we might have some fun background sounds from you. Where are you, Andy? Well, officially, what you asked me to say, tell the world is that I'm in italy having a cappuccino and smoking a cigar the real story though is i'm sitting in an airport lounge and there's a lot of people in the back ground and uh so there might be some rattling there might be some announcements of people being late at the gate so we'll see what happens but i'll try to mute my phone once uh if the if the background noise is too loud all right so andy as i mentioned earlier today um you hearing your voice just made my day and i i stand by that every every time i hear your voice

like my day just improves just brings a big smile on my face and for the record i do not report to andy he's not my boss in any way shape or form so it just tells me that your day is typically really crappy it seems i do have two kids that's the highlight of your day brian yeah hey wait our guests our guests getting rowdy and speaking before a call you know there's only one person andy there's only one person i know who would have the audacity to speak before introduce. Who's making this noise, Andy? talking about great things that his company is doing, but it's also we know that a lot of the stuff that he's actually selling for what he's been doing is actually done by other people. So he's just a great boss having a great team

and they're doing amazing things, obviously. So the person that you're talking about is Nestor. Is Nestor here? And now you're officially allowed to speak. I am officially allowed to speak now. Wonderful. Hey, Andy. Hey, Brian.

How's it going? Hey, Nestor. Welcome back to, Andy. Hey, Brian. How's it going? Hey, Nestor. Welcome back to the show. Hey, Nestor. Sorry for giving you the hard time in the beginning with the intro. But it's very true. It's very true. I have a great opportunity to work with a wonderful team, wonderful colleagues, and being able to have this platform and share it out. It's a good experience. So, yeah, very privileged to have that team. And Nestor, I know you've been speaking at Perform multiple times, both in Vegas as well as in Barcelona this year. We also had you on a podcast before and the Performance Clinic,

and we also did webinars together. Now, the big topics that we've been kind of promoting over the last couple of months, I would say, is around chat ops, which is also the topic of today, why I invited you. But then there's one big thing that people don't know, or probably don't know, that whenever we talk about chat ops,

I always bring up your chatbot, Ultron, and then you have Megatron, and you have probably even more Trons that I'm not even aware of. Just Tron. What about like the old Disney movie Tron? Yeah, that's true. So what I'm wondering now,

and you're a great team, I'm sure there's somebody that is actually the godfather or the creator of these bots. And wouldn't it be cool if we would actually have that person on the line as well? Wouldn't that be cool if we would actually have that person on the line as well? Wouldn't that be cool?

That would be awesome. That would be fantastic. And we've talked about potentially having that person here and, you know, that would be a great opportunity for them to showcase their skillset and share to the world

the wonderful work being done. So do you think would you be concerned that they would steal the spotlight over the fact that, like, you take credit for all this stuff? I mean, you don't take the credit. Showcasing it. Showcasing it.

Always credit to the team. Yeah, wonderful team there. But yeah, no, no spotlight. I want them to share the spotlight and the limelight and take full advantage of that. So that means what we can say now, creative Ultron, please speak up or be quiet forever. Hey, guys. I'm here.

There you go. Hey, Sohib. How are you? I'm very well, thank you. How are you, Andy, Brian, Esther? It's all good. It's all good. I'm just still not sure if we're speaking to a real person

or if it's just a bot in the back because this topic is all about chatbots and maybe voicebots or whatever else there is out there. I assume you're real, though. I am. I am real. And if you were the creator of Ultron, I would have thought you would have had a much more mischievous voice like, yes yes i am here

who are these mortals talking to me it's great to see that really behind the these machines of destruction well actually this is the machines of creation and unity it's just a normal voice yeah nice sounding person um one thing i want to point out though before we jump into all this right because we're going to be talking about a before we jump into all this, right? Because we're going to be talking about a lot of really cool stuff, Andy, right? I want to just reference back to the first time Nestor was on the podcast, when Nestor and Abir were on, right? You all were in the middle, not the beginning, not at the end, but in the middle of your pipeline conversion, your CSAD journey, right? And that was what, the middle of your pipeline conversion or CSAD journey.

Right. And that was what, maybe two, three years ago or something, somewhere around that range. So just for listeners who might've heard that, or also people who haven't heard that, just thinking like they were in the middle of it all. They got past some hurdles, still had a whole bunch more to get past. And now they're going to be talking about this awesome place they're in now so this is possible you know it's at a large company citrix where you know where you're not dealing with some agile startup that you know you would think you could do anything with a lot of people think the larger companies things move a lot slower or harder to do no this is a true tale success story of you know in a in an enterprise doing some really really cool stuff so just in that context i always wanted to frame that before we dive into it.

So I think it's really awesome. Yeah. And actually, let's dive right into the topic. So I want to talk about the role of JetOps within Citrix or also overall, because I'm pretty sure you have done your research when you started your journey on what is actually possible. I always think as JetOps as an enabler for self-service on the one side for engineers, but also then using it obviously for things like you doing it as well,

it's notifications, it's self-healing, you do a lot of cool things. But Sohaip or Nestor, maybe you can give us a little overview of how you started your journey and what were the use cases initially that brought you into leveraging that type of technology? Yeah, so some of the, just to kick it off, and I'll just do the small part, and then Zoidian can drill down to the really cool stuff with that. And when we started using and leveraging Slack, that's really what opened up the door

for us to be doing a lot of this automation. Big Skype shop, but Slack came on board. We were able to do more things in terms with integrating with Dun & Trace, getting things in there. But we noticed that there was a room for automation and for the Slack bots to be created. So at first, it was just basic things. It was just having the bot go out and check what would be the equivalent to a ping or what would be the equivalent for the bot to check something for us or perhaps to do a task that was being done in Jenkins. And that was all cool. But we started noticing that there was a

room for growth and opportunity. And thankfully, when Zoheb got a little bit more into the weeds of how we operated when he came on board, he was able to take that to the next level. So I'm going to let him handle what the vision we had on the application operations team and what his vision was to create what the bots are today that started off a real small scale to now he'll explain to full end-to-end deployments, what we call zero touch or no human touch intervention. Yeah, thanks, Nestor.

So the vision, how it started, there was a team reshuffle. So we used to manage only production systems. And then we have a change management. And now we manage then all three environments, including test, QA, and production. In there, we noticed that there's a lot of manual stuff, especially in the QA environment going. We saw that opportunity where we keep getting deployment requests.

We can automate, but of course, we wanted to have some sort of automation where no intervention required from the operation side, but also we have full control over it. So we saw that. So what we did, we start looking into the bots and we start evaluating how we can do that. So we developed a Slack bot, Ultron. It was integrated with ServiceNow as well as Jenkins. The ability of the bot was that

instead of a developer ask us to release the code, it actually asked the bot to do that. Basically, they go to our kind of Slack alert channel. They kind of give a command. So with that command, the bot actually opens up the service note ticket. It will go connect to the Jenkins and release that bill number to the environment, and it will run load testing. After everything is complete, it will go ahead and close the ticket and let the developer know it's fully complete. So in this way, we kind of achieve fully automated process. Previously, they rely on us to release the code during working hours, but now they can release at any time, even in the middle of night.

Correct. And just to put some more context into what Zohaib was mentioning there, is when I joined Citrix, we were doing these web application deployments. It was very, well, we consider the waterfall aspect of it. We only deployed on Wednesdays. So if you had to deploy any code, you had to make sure you had to get a QA, UAT, and done ready for deployment Wednesday morning. So we wanted it done Tuesday night, the change request submitted.

We're not using Snow service now at the time. And if you couldn't get to code in, then you had to wait until the next week or create an emergency change request. It was a lot of hurdles, a lot of red tape, VP signatures were required. It was a mess. And then we decided to be a little bit more, what we

thought was agile and deploying twice a week, and that went that route. But we noticed the more technology came on board, the more applications we were creating on the internal side, we could not keep up with it once a week. So we had to go ahead and do this on a more consistent basis. Later on, fast forward a few years, when Zohe was already on board, we now took over the entire QA environment as admins, not just as helping them out. So that QA environment, as Zohin mentioned, was being deployed constantly for not just America's East and America's West, but also for the international folks.

So the on-call would have to spend their time now having to deploy the code, simply going into Jenkins at the time, deploying data and taking 15, 20 minutes. And then they went back and tried to fix something. They would call the on-call about an hour and a half later while they were having dinner or driving home to deploy something else. So that was reduced greatly. So the amount of time an engineer was spending on deploying code versus spending time with their family, having their work-life balance,

or just taking care of other things while in the office was greatly reduced. And as Lohe mentioned, the production side of the house, it would take us up to three and a half to four hours, basically the entire morning that Wednesday, to do a deployment or any other day because we change to doing deployments every day. That now has been greatly reduced.

And if a deployment takes end-to-end from request to completion of 15, 20 minutes, I think it's too much. Zoho was the actual number that we have now end-to-end from when the snow ticket is submitted to when Ultron says, or Megatron says that it's all successfully done. I think it's, you know, way more reduced. And so you can see we get three hours back,

four hours back from our day for multiple folks as well. Yeah. So even last year alone, there was around 15 to 1600 QA requests that we as an ops team did not even look at it. They just come in, code gets deployed, and the bot let the developer know hey it's complete go ahead and validate all those

changes and they go ahead and close the ticket as well so this is something great to have I was just asking so when you initially started the journey was there pushback was there like not trust into

the system that you kind of had to overcome? Because I think that's chatbot, chat technology or some other automation technology. But I think the challenge with automation or with change in general is that do we get the same level of confidence later after we change things to something faster, more automated than we have right now. So did you have to prove to your team, to your organization, that what you're doing is not only faster, but also provides better quality results? This is a very good question, Andy. I just want to start on that. I mean, the first step wasn't kind of releasing the code, wasn't like a business end process.

The first step was very small, where the first use case of our chat ops was where we have one of the apps service where it's running on several different nodes, but it often crashed. And we needed to look into which node was crashed manually. We kind of go ahead and check those nodes, checking our dynatrace, okay, this node is crashed, and then we fix that node. So initial use case was that. What we did is we developed a monitoring code. What it does is it go ahead and validated all the nodes. we develop a monitoring code.

What it does is it go ahead and validate all the nodes. So instead of developer kind of by saying, hey, my app is crashed. But what we're doing is they actually go ahead and query, they go ahead and ask the bot to let them know what node is crashed for that particular service. So the bot actually could go ahead and query each node and see what are the response time and what node is crashed. In this way, we'll quickly troubleshoot. So that was the first step. So what we did,

so they were already using it before we implemented the other business process. I'll let Nestor to explain a bit more on it. Yeah, so exactly what Zoya had mentioned. It was, we knew there was going to be a trust factor there. This was something relatively new, not just to the operational team, but for Citrix itself and as an organization, because there was a couple of hurdles that we had to kind of accomplish. And with the same kind of hurdles that we did with, of accomplish and with the same kind of

hurdles that we did with when we went from Atman to Denitrace is the trust and how much value am I going to provide as an engineer so one of the things that we had an issue with going from Atman to Denitrace is that folks said well can I trust the data can I trust that root cause trusting the data was a huge thing and also well now that I'm not going to be needing to spend an hour and a half to drill down a peer path I might I really that relevant as an engineer? So that was another scare. Same thing with the box.

The box came in here saying, well, now instead of coming to me to check for a service or an app, am I really going to be relevant? Am I really going to be needed to do that? And we started thinking, well, if that's all you're here for, then, of course, that's not good. But that's not what you are here for. You're here to sustain, maintain, and improve. So when we started receiving less and less requests to check for an app and empower the development team and the business users to check that, that alleviated some of that pressure from the operational team

and allowed them to spend more time diving into these flagpots, into automation, learning more about the data trace synthetics and cloud. So we were able to free up some time for the engineers and not have to be so stressed out about everything coming their way. So that was a cultural shift that we had to do and let them know bots are not going to take over and take your job and get you fired. They're going to approve you as an engineer and give you more time for that. And also from a business standpoint, trusting that data,

that when the bot comes back and says, hey, it checked all these resources and these are failed, they trusted it because they knew it was valid, they knew it was going to be a concern if it started spitting back irrelevant data. So we started off very small, and then we knew we wouldn't grow into that. You know, I think the point about the changing roles is a really, really, really important one because when you think about it, a person has a skill set

or a person is used to doing a certain type of job at work. And as you're saying, oh, I'm not going to be seen as relevant or I'm not going to have anything to do. And what always strikes me is when people get that fear, first of all, that's naturally, that's a human reaction, right? You think you're going to be out of a job suddenly. But the reality is if you don't go with these changes, that's what's going to more likely put you out of a job because more and more companies and more and more

organizations are going to this automation phase, which means operations folks, dev folks, they all have new roles, new skills that they need to learn in it. You know, maybe it's not manually releasing the things, but it's maintaining the pipeline, automating the pipeline better, making improvements to it, managing and finding other new things to do, which is if you just stick to, well, I manually release, you know, you're going to have less and less places you can work as time goes on. So it's, it's a fantastic opportunity when these things go on to embrace it and jump right on in and say, I'm going to, I'm going to try to get right on top and lead this effort because this is the new skill set. Your company's giving the opportunity to develop this new skill set you know and it's

to me this kind of stuff is fun right anyway um so just for anyone going into it don't think of it as i'm going to lose my job it's you're going to gain skills to be able to continue working in the i.t field field and stay relevant which was that how big of an issue was that or you know were people really concerned or was it just kind of like? No, there was some concern, yeah, for that because they were concerned that, well, one of the main things when we got hired, a couple of them said was we were responsible all the problems and bubbling up to executives and to the development team, that's one area that I'm not going to be focused on so heavily anymore. Before, I used to sit from 9.30 a.m. to lunchtime deploying something a couple of times a day. Now, I'm only going to be spending about half an hour doing that. Then what am I going to do for the

rest of the time? So there was some concern. I'm like, well, how much bandwidth am i going to do for the rest of the time so there was some concern i'm going to go how much uh bandwidth am i going to have available how much relevancy am i going to have so finding those other opportunities i had a quote i think that was used in perform last year and it was automate yourself out of your current job role into the next and if you were a web admin then go ahead and try to become an automation engineer or a devops engineer if you were a web admin, then go ahead and try to become an automation engineer or a DevOps engineer. If you were a data center systems admin, try to become a cloud engineer or get an automation engineer and try to go that route and improve your skillset because you are very correct, Brian. The world is changing. IT is going to change, perhaps a little bit slower in a huge org enterprise like Citrix. But eventually, Azure is going to catch up to more departments,

more items are going to be offloaded into the cloud. Well, then are you going to be ready for that? Are you going to be ready for when we spin up a server? It does not take you an hour and a half. It takes you five to 10 minutes. Are you going to be ready for that? If you're not ready for the automation, if you're not ready to maintain these pipelines and workflows, you are going to suffer because at that point, two years from now, you're not going to be relevant, not just at Citrix or the IT operations department, but when you go hunting for work elsewhere, they're going to ask you these things. They're going to ask you, what is your automation background?

What experience do you have working with perhaps Box or Cloud? And if you don't have that relevancy or if you don't have that experience, it's much harder to get into that door rather than be in here already. Like you mentioned, your career can grow within the org that's training you up to get that going. So with this bot automation, I think the one thing we need to mention,

the amount of time we're saving, not only for ourselves, but also for the other teams. And previously, when we receive requests to release a code, it will take maybe roughly on average at least an hour, an hour and a half to get respond and release the code, releasing the code another 15, 20 minutes. So maybe in a two-hour window, we basically release the code, releasing the code another 15, 20 minutes. So maybe in a two-hour window, we basically release the code. So with this automation bot, it just happened instantaneously. As the request comes in, the release process gets started straight away. So this was well

appreciated by, especially by the dev team, because it will actually help us to ship a code a lot faster from a QA to production. So, you know, the time saving is huge here. I would assume not only time saving, but I believe this is what we are trying to kind of preach right now. It's about giving autonomy to the actual teams, right? If you say it's appreciated by the development teams, it's because you give them more autonomy. They are less depending on, let's say, special teams to do their job, which is providing new value faster to the business.

And if you give them the tools in a self-service way, they become more autonomous. Because as you said, they don't have to wait for you and Nestor all the time to do the deployments. They don't have to wait for all these approvals to get their job done. They become more autonomous. And I think that's why I just love what you guys are doing. And I fully understand the benefits. And you know what, Andy, this sounds like, so we have coming up, and I think possibly next episode, I think, Gene Kim's coming back on because he's got a new book, The Unicorn Project.

And I've been, I'm not sure if Andy's been doing his homework and reading it, but I've been trying to make sure I get it done on time. But this, what you're describing is very much what this book is about. So the Phoenix project, the original book that was, you know, more about switching over to the newer DevOps type of mindset. This is on this, this book kind of tackles on your side of the house. How do you get all these, how do you get rid of all these terrible old processes and get code moving through the system and stop waiting on tickets and all this kind of stuff so it's it's as you're describing this i'm like this is what i'm reading about right now it's really cool so just a plug for that because that's coming out soon yeah yeah that's pretty fantastic

on that workload like andy mentioned hit that right on the head uh developers who are more empowered they feel happier now that they can go ahead and deploy that. We would come in the morning or just check the Slack channel in the middle of the night sometimes, and we would see that four or five, six deployments were being done. And it was not a developer. They didn't have to sit around and wait for us. They could just improve their code. And by the time their director came in the morning, they're like, hey, this is pretty solid.

We can push this now for uat and uh to the director's eyes they've all they they resolved this overnight because they were able to do that they had that flexibility so improving processes improving the quality of the work all that was much greater improvement which reducing the cost but you know, empowering the folks to do that. Hey, can we, Nestor or Suhaib, can you give the listeners kind of an idea on what use cases you actually automated? I know we talked about deployments. You talked about testing. You talked about in the beginning, you know, writing a bot that is querying status information,

then pulling that back. Maybe can you list a couple of use cases that you automated over the over the years or the months and years so that people get a better idea on what's even possible and then also maybe giving a little outlook on what you have on the roadmap because i'm pretty sure you're probably you still have a bunch of ideas um so the what few use cases, even the most frequent ones we kind of picked because we would keep asking for a certain thing over an hour, we automate through our bot. So, first one was release. So, we do hundreds of releases in a month. That's fully automated. No one asks us, it just relates automatically. The second use case, we implemented cell feeling,

where we have some services where we set through Dynatrace Synthetic, they're monitoring, and based on the problem, we execute our cell feeling script, and that's happening through our bot as well. And another use case, we did the performance as a cell service, where we have owners of services. They keep asking ops team to provide some analytical data, but say, hey, how's their service was performing? That's automated.

They can actually ask the bot, bot actually curate the Dynatrace API to pull those results and give them in a graphical form so they can see it. Nestor, you want to mention? Yeah, so all those are the things that we have been able to successfully do. And for a future state, there's other teams that now we're trying to leverage and trying to get this code out to other folks on my team specifically. One of the challenges that we have on the data center and cloud operations side is pretty trivial, is the shipping. We get a lot of stuff shipped to our data centers for work that needs to be done,

but there's no concise way of tracking all this and doing this without going through multiple forms. So one of the things that we've collaborated with from my team and Zohib as well is going and creating a Slack channel, creating a bot, so much friendlier bot, BB8. And a little bit of a jump there. So on my side, the BB8 bot will be able to go ahead and create the shipping request. So somebody can go into the Slack channel and say, add DC shipping, you know, and put in the information on a little form.

And that form gets fed into a site like ServiceNow and then creates everything. And then we have everything of the ticket, the shipping number, who is going to be contacted. Do we need to resource in the data center for this? So instead of having to fill out two different forms, now we have one concise form. We don't slow down projects because if you're not aware of this, if you ship something to a data center without it being notified, the data center can and does not accept the shipment. So if you have net scales being shipped, routers being shipped, and that gets turned away because you didn't fill out the proper form or get it in on time, that obviously delays the project pretty substantially there as well.

Some of the cool thing is assigning tasks in Jira. So when we go into Slackbot, we say add Ultron, assign tasks to whoever on the team. And they can go ahead and go to your Jira and your Agile, in our case, the Kanban board and assign these tasks to these folks here. So you don't have to go into JIRA, assign your stuff. You can quickly add it for different team members and Ultron or, in this case, BBA will respond back with the update on what's being done there as well. So a lot of use cases in the feature that we have for this.

Also, looking for stuff in the data center in the cloud. We don't know who owns the subscription, who does this not. So we can get BBA to start forwarding that information to the users, not just to IT, but also to business folks as well. So we're trying to expand the use cases and the knowledge of bots as something normal. So we want to get to the point where we showcase our bot for it not to be like, oh, my God, that's amazing. For it to be like, yeah, that's what we expect. So in the coming years, that's what the goal is for us, for it to be all in right now, but in the future for it to be the standard

of how we get the information to the IT and business units. And the one thing actually I'm implementing these days is the quality gates, the kept in quality gates. This is a great service. So right now we have set up that microservice as a standalone application. And we're going to leverage that to use our quality gate. So this is being worked on. Yeah, that's pretty cool.

I want to actually say officially and publicly, thank you for being one of the contributors to our kept in open source project, because that's pretty cool. I want to actually say officially and publicly, thank you for being one of the contributors to our Captain Open Source project, because that's pretty cool. And having you as an expert, as the Ultron that can master the world, having you on the team. Who approved that?

I have no idea what's going on here. What is Captain? Oh, Captain, my Captain. No, but it's really cool that you're collaborating so closely with the core Captain team on the Jet use cases and the quality gates now. So, officially, thank you for that. And what I wanted to say, thanks for the use cases.

What I thought of, you explained a lot of the use cases and some of them have to do with ServiceNow. We create a ticket, a change request, and interact with ServiceNow. So for me, it feels, whether it's ServiceNow or Dynatrace or other products out there, what you're really doing, you're building specialized quote-unquote apps

that make it easy for end users to consume out there, what you're really doing, you're building specialized quote-unquote apps that make it easy for end users to consume or interact with a certain use case where it might be, you know, let's say too cumbersome or, I don't know, too heavy to do it through, let's say, the regular means. If you think about ServiceNow, I know it's a big product and I'm sure you could do all this, but instead of learning and teaching everyone how to interact with service now and give them the privileges that they need why not write a bot that is special or individual bots that are then specialized on doing certain things in a very

easy way because we know the easier things are the the more and faster adoption will be and therefore people will again become more efficient and then become more autonomous. So I really actually like the idea of kind of explaining chatbots as it's a mini app to solve a business or a technical problem on top of mostly existing environments or APIs or software solutions you already have, right? Correct, yes.

That's exactly what the mindset is for that, specifically for that shipping one. It's more convoluted to create that form in the back end and you spend time. If you don't make things easy for the business folks or for busy developers or IT operation folks, they're probably not going to use it as readily. If they have to absolutely use it, then fine, they'll do it.

But if the form is too long, if it's too complicated to go into JIRA and submit this, they're not going to do it. It's real easy just to say add bot assigned task to whoever, you know, in Slack. That's something they do all the time. So you don't have to log into JIRA. You don't have to go through anything there. You've reduced that amount of time. That friction has been reduced.

So in doing that, Andy, you're 100% correct. Trying to make it easy, more seamless. And that way, it gets adopted much quicker and we're able to leverage that much more. And I think if I take another analogy, right, if you think about the banking and let's say the banking websites that came up, I don't know, 10, 15 years ago, and there's a lot of features, obviously, feature-rich websites where you can do all things. But now,

as we see more and more banks providing really good apps, I think the ones that are doing it smart, they don't just replicate every functionality one-to-one that they had in their web app into mobile, but they really target individual use cases so that you can do certain things like transferring money, checking your balance in a very easy and convenient way. And I think the same way as apps on our smartphones have changed the way we as users think about interacting with different use cases the same way chat ops is is changing the way we are interacting with with it services right it's about small chatbots and they can do a certain thing and a certain thing really well and it's very

convenient to use them and if you need another thing then maybe you have another chatbot or you extend the capability of one chatbot and i think think it's also, I mean, this is what you guys have been doing well. And again, if I recount what you said earlier, you wrote one for releasing, you wrote one for self-healing, you wrote one for performance as a self-service. So these are all individual, very specific use cases that you automated so that people like to use it, they see the benefit, and it's easy to use. And I think that's phenomenal. That's the goal.

So I have a question about what I think was probably one of the more contentious things in your environment, which is who gets to pick the bot names? Because I imagine there's a lot of competition. I was sitting there thinking you can do K9 from Doctor Who, you just use bb8 like how do you pick a droid from star wars i mean geez who what's that process like and how long does that take it's a democracy it goes through a channel it goes to a committee it goes to approval through the vp and then no no no no um i think the first time that we had the need to name a bot uh when we

started with Ultron we did a slack pull right we did like slash poly we put in a couple of names in there and then um we just got the uh the most popular one and I don't know if that was I think that then we weren't Ultron because we thought that was the coolest one. But then going forward, we just started picking something within that realm. Going forward, we kind of take the same theme and we have, we develop another bot, we name it Megatron. And then we have a bot

which works on staging environment and we give it a name as Tron. So Tron's everywhere. Yeah. So we try to switch it name as tron so tron's everywhere yeah yeah so we try to switch it up not be so evil so we want a little my team specifically our big uh like shocking right star wars fans in it go figure yeah no right first i heard about that so they wanted to do uh things like that so let's go the friendlier route let's not do uh death star or anything like that let's try to ease into the situation so you know uh bb8 r2d2 those things can run but uh at citrix there's some other bots that were being used as well and i think uh like jarvis and iron man things like that were all already taken by some of the engineering folks and uh we actually

have a slack lunch bot, believe it or not. So we have a Slack channel for our lunch. And instead of having to go to the cafe downstairs, trying to see what's in for lunch, we have our Slack bot that lets us know what's going on. So FTL-lunch, we have a little guy there, you know, the at lunch bot gives us all the lunch specials, what the prices are, what has been changed and updated.

So that's pretty cool to see that the rest of the org is kind of like seeing these things happening and being part of their day-to-day operation, not just for IT work, but also for things like that, like what's for lunch, you know? So just one thing. And when I just say about Megatron, that kind of reminds me of the use case Nestor knows. We both discussed initially. Our production releases were way too complex.

There was a number of factors we need to look into. Like there was 20-odd build numbers. There's 20-odd products that we'd kind of look, we say, hey, this is right, this is right. Check individually, manually taking servers off the load balancer. There's a lot of factors for the production release. So then we developed the Megatron bot, which basically automate all the complexity that we used to do manually um you know so now all we just do in one command

and it does everything it just goes to the service now check if the change request is approved or not if it's approved this if it's deployment window if if now if the bot can deploy or not, all these validations. Then it will go ahead and check our version control to make sure all the versions of the code in the releases, they're checked in, they're accurate. What was mentioned in the CR, in the change request, is there, it's available. Then it will release, not only kind of release in that way it also check if there is a code need to deploy to the azure as well it will check that part as well so previously there were so many

manual steps in that we used to do manually production release which was headache i remember the first production release i did I did a mistake by typing manually an incorrect build number that cost us like, I think 30 to 40 minutes to roll back and fix on that particular node. So that was the automate thing we did in Megatron. Wow, that's amazing stuff. It's, especially with the bot piece, right?

I love how that interacts with Slack and how you can do this. I know some teams are starting to build these things out with TeamCity or Microsoft Teams, really. But this is starting to become a reality. I know a while back, we introduced, on the Dynatrace side, we introduced our Slack integration. And then we also, besides the chat ops, we were trying to like say, hey, there's also voice ops where you can talk to it with your Alexa device. And, you know, being on the pre-sale side, I was like, I wonder how much people are going to embrace this, especially the voice ops side. But I think we're definitely seeing a lot of the chat ops coming. Me personally, I always just kind of feel funny talking to a computer.

So I don't know if voice ops is a step that's taken after chat ops. Do you all have any thoughts on that? Do you imagine coming in and talking to your computers as opposed to typing things in? Does it seem like it would be something that is the next step in the change? Or does it just seem weird to talk to your computer? I think it all depends on what you're doing for a, not for a living, but what resource you have. So for example, in an open work space like we have here at Citrix, we don't have no dedicated officers or work area. We kind of just roam around from anywhere. So for us to pop an Alexa device and just have it and then walk up to it,

we may be in the same area or not, may not be as feasible, but perhaps like a NOC or an area that has an engineering or project team that are going to be there almost every single day, it's pretty cool to have an Alexa device that you can just ask questions or a director maybe in his office or VP can just figure out what's going on. I did try the Alexa Slack integration for a bit, but I had to turn it off because my kids started assigning tasks to me in Jira. So I had to disable that feature because they heard me say it one time in the morning, and then they started saying assign

Nestor Z. I'm like, no, no, no. I have all these random things like fight cookies or take us to Disney World. And so, no, I had to disable that from home. But yeah, it is pretty cool how the feature works. And I do have the same mindset as you, Brian, that I think you ease into chatbot first and then you kind of progress into the whole voice-offs. In my opinion, that's just what I would see there because I think that people are familiar doing that at home they're familiar telling alexa or their google assistant to do a task such as turning on the tv or turning down the lights or turn the fan or the ac with the nest or it could be whatever it is we're not sponsoring anybody here um so i think that as the future comes along with more of these voice ops, I think that's

going to trickle into the enterprise as well. But for sure, starting off with chatbots is probably the most easiest, I think, in enterprise. Yeah. Alexa, delete production. Right? You don't want that. No, I don't like that. Hopefully no one has that command uh in their uh in their alexa right now and i didn't just delete someone's production by listening to the podcast hopefully not hopefully not hey i also wanted to uh remind people that in case everybody anybody wants to see more hands-on what you guys have built uh we actually have

a performance clinic recording on the Dynatrace YouTube channel in the Performance Clinic playlist with Nestor and Zoib, where they actually talked about how they're using JetOps for self-healing and for deployments. So for anybody that is interested, there's a lot of material out there,

a video where you actually show the screenshots and how it looks like in the Slack and then also what's happening in both Dynatrace and ServiceNow and so on and so forth. So I think we should also add this as a follow-up link to the podcast proceedings. Podcast, yeah. Yeah.

I want to ask one more topic because I think some people that hopefully got inspired by looking into chat ops. What does it take to get started? And not necessarily from an organizational perspective, but from a technical perspective. And if that might be something for Suhaib, what are

the things people that want to start with it? Is there like a sample kit or a certain smaller project that you use to start with it? Is there like a sample kit or a certain smaller project that you use to start with, like the lunchbox or the cafeteria bot that is a great starting point? What are the languages that you write it in?

Where does it host or where is the chatbot hosted? Can you give us some additional tips and tricks for people that want to get started? So we started developing. The first thing is to understand the Slack real-time messaging API. So that supports Python, Node.js, maybe a couple of other languages as well. But we like Python to use.

It works very well with our environments so we so i pick python and hosting wise we are actually hosting in in cloud as well as on-prem so we're using docker container to just to run our bot. And in terms of any tool, for example, for our testing, we're using Selenium. The Selenium is integrated fully with the bot. But I will suggest if anyone want to start building a chatbot, the best place to look into GitHub. Look into any example code there, clone it, try to understand it, and go from there. Then copy-paste it, right?

Copy and paste it into your environment and you're done. And then you write owner Nestor and then make a podcast out of it. I mean, no point to reinvent the wheel. I mean, it's very hard to kind of work on a kind of blank canvas. But if you have some examples, either you take from a GitHub or you take from the documentation side, it will always help to get started to do that.

And the one thing I just wanna say, like don't over-complicate it. I mean, once you're thinking about a use case, you know, think very small. Start very small where you just say, you send a message to the bot, bot just reply to you, you know, something like that. And then from there, move on to the bot or just reply to you you know something like that and then from there move

on to the next areas and the another thing i would say master rest apis learn to develop recipes use any framework and java spring to django rest or dotnet Core, any language you like to develop. REST APIs. REST APIs are great in terms of if you want to do, for example, if you want to set up a permission for your bot where a certain users want to talk to a bot. So what you could do, you could develop a permission endpoint where you actually add a bunch of users where you query that endpoint to get the list of users

and see if that user has a permission or not. Very simple use case. Yeah, which is great for production deployments because we had to make sure that not anybody can just go in and tell Ultron and Megatron what to do. So we had to lock that down pretty tight. So that was a great point, Zohei, to bring up security nowadays. So as you kind of progress, where you want to integrate more and more tools,

we don't really overload one bot or one function. What we do, we just keep developing small APIs around it and let bot consume that, and that's how it works in our environment. So Andy, is it time to summon the Summaryator? I think so. Now the question is we need to maybe give the Summaryator a new name because he may need to be

a bot in the future. Can we actually create a Summaryator? Well, the Summaryator was named after the Terminator which was named in honor of you at the Austrian. So it's kind of bot related. That's true. It's,

it's kind of artificial. It's kind of artificial anyway. Yeah. Well, maybe, maybe we'll have to get a committee and some forms and, and raise it up. But if you have,

well, we'll, we'll do two things, right, Andy? So anybody listening, if you want to rename the Summer Raider, send us a tweet at, at pure underscore DT and give us suggestions.

But Andy, if you also have some suggestions as well. Yeah, I have a suggestion. If anybody wants to write a Summaryator bot that I can just trigger now and basically say, Summaryator bot, please give me the summary. And then it does its job, right? That's what bots are for, it's automating things. So that would be, so what that would be, so right now they have machines writing sports articles, right?

They've just taken all the stats. So what that would have to do, you'd have to have a real-time, while we're recording, way to ingest and transcribe the words we're saying. And then it'll have to go put together a summary for that. And then in a nice computer voice, hello, here is the summary, right? That kind of thing.

Exactly. And then it will say, Nestor is great. It has to be. Now, okay, on a serious note, so I think what I, thanks for being on the show, but I think what I learned from the whole thing is chat technology obviously requires automation use cases. That means you have to think of what you can actually automate.

But in the end, it's really all about, and I like this, what you said, no human touch or no human intervention tasks. That means you want to do things where you can automate things away that are normally require human touch or intervention. I really like what you said, Nestor, at the show last year. Bots are not going to take away your job.

They're going to improve your current situation. They allow you to automate yourself out of your current job role into the next. I think that's the big thing. It's not about taking away your job. It's about elevating it. It's about bringing it to the next level i really love your use cases that completely fit into what we see out there will be promoting whether it's around monitoring as a service or performance as a service i really love what you wrote there where you can query your chatbot and

then it gives back you know data from monitoring like Dynatrace. The deployment automation, release automation bot, that's great. Self-healing. And then thanks, Sohaib, for contributing to Captain because that just helps us also to get this project forward and provide all these use cases through an open source project. And last but not least,

I know I want to remind people there is a performance clinic out there, a YouTube video, where both guys, both Sohaib and Nestor have basically walked us through the use cases and actually show how it works. So yeah, I think that's it. I think bots are a great way for us to improve life so we can spend more time with things that are really important in life. And that's, as Nestor said earlier, maybe spending more time with uh with things that are really important in life and that's as nesta said earlier maybe spending more time with family and friends yes i just wanted to mention too i think this is a great story you know going going way back to the first time when you're on the i love the story too thank you nesta and also for coming on this time. But Nestor, thanks for repeatedly coming on because I love the places you are in your journeys. You know, in that first one, when you were halfway through, you never hear stories about people who are halfway through. And it was just amazing to hear that.

But I think what this whole chat ops story that you're telling us now and the processes that it fixed is really the next big hurdle for a lot of a lot of people especially i mean we talked to i talked to a lot of prospects and customers and a lot of times they're like oh yeah we've got devops models in place we really haven't automated anything yet right and well it's great to have devops in place but if you haven't got that automation yet if you have a developer checking in their code and then it takes five days for it to get to QA for testing. And then 20 tickets and a whole series of approvals and everything to get to production, and everyone's manually checking over the results. You know, you've maybe broken down the silos internally,

but your processes are still taking way too long. And you're exactly, the story you two are telling is exactly the story that a lot of people have to get through. So it can be done. And just like anything, you start small, you take one step at a time

and you keep it going. And it's wonderful when you get to that point. I mean, I'm sure it's not all, you know, wine and roses at this point, but it's a million times better than where you were in the past right um so congratulations on that it's great to hear this stuff yeah thanks for being on the show

for that and just like we have evolved and changed the last uh seven years for me here at citrix and zolio in the last couple of years as well the key is you mentioned there getting to a certain point, feeling comfortable with it, but right when you feel comfortable with it, putting yourself in uncomfortable and trying to learn and grow because if you stay there,

you're going to stay stagnant and not improve. And it's just always changing, always improving, always learning. And that's the key right there. Yeah, we got DevOps. Yeah, we put down some silos. That's great. That's step one.

But then some folks might just want to stay there. The next step, we want to go ahead and think about the future. So yeah, something we're trying to hopefully encourage for our team, put in that culture for the new folks coming in as well and see the next level, see where this takes us next. So hopefully we'll be able to share more stories like this in the future awesome and if you want to hear other stories by other people andy i'm going to do your job here perform is coming up it's going to be i think the first week of february in vegas uh we there's always the perform goes so just those are the more local localized performs so make sure you check out, what's the website for that, Andy? dynatrace.com slash perform?

Exactly, or perform.dynatrace.com, either area. But we also have the big performer. We have a lot of great speakers and all that. So make sure you check those out. Again, if you have ideas, if you want to rename the chatbot and you want to throw out a bot suggestion,

a name, go ahead and do that to pure underscore DT. Or if you have any show topic ideas or guests, let us know. I want to see you guys. I want to see both of you in Vegas. And that means this is the open invitation to come

to do a breakout with me and giving more people the chance to actually see you live and then interact with you. Yeah, that would be great. We we're open to that invitation open to that opportunity and wherever you need us there andy we'll be happy to to share that and share the stories because i think that's great when you learn and grow at places like perform uh the dentures go uh that we had in atlanta and you were part of the other ones you get to see a lot of the things struggles that you're going through perhaps or ideas and as uh invaluable um part of the were part of the other ones, you get to see a lot of the things, struggles that you're going through perhaps or ideas

and that's invaluable part of the role. Absolutely. And Nestor, how many pairs of Dynastray shoes do you have now, Nestor? I have one pair left of whatever Dave Anderson has. One pair left? I think one pair less.

So yeah, I think I have four or five. I think you need to donate your next pair to Z So yeah, I think I have four or five. Wow. So I think you need to donate your next pair to Zohib. I got one. Well, Zohib will get his. I got one. Okay, Zohib has.

See? Ah, you will get another one. Yeah. I'm still waiting for my Hall of Fame one. So, you know, see when that comes through. And you know, employees don't get them. I heard. I heard employees don't get them.

You're in a special place there. Anyway, alright. Somebody tried to buy them off my last SKO. That's hilarious. And then sell it on eBay, right? No, no, no. Alright, well thank you all for coming on.