PurePerformance - 049 Traditional Ops to Agile Transformation at Citrix

Episode Date: November 20, 2017

We typically hear about agile transformation being driven from development and eventually pushing it towards operations. But it doesn’t have to be that way as we hear from Nestor and Abeer who helped transform their operations team from Waterfall (Traditional Ops), to Partial Scrum (Intro to Agile) and then Kanban (more defined structures for Ops using Agile principles). Listen in and learn what the differences are between Agile in Dev and Agile in Ops, which metrics they use to measure the success and how they are now pushing towards a DevOps transformation from Ops towards Dev.If you want to chat live with Nestor and Abeer then take the chance and meet them at PERFORM 2018 ( http://perform.dynatrace.com ) where they give us more insights into their transformation

It's time for Pure Performance. Get your stopwatches ready. It's time of Pure Performance. That's right. We had a lot of technical difficulties getting this one going, didn't we, Andy? Well, the problems were primarily, well, I don't want to blame you, but it was on your side. I think so. Or maybe on Skypepe's side we don't really know it's technology and me technology yeah wouldn't it be great to have a monitoring tool that could trace all these problems down to the root cause it's sure yeah it sure would be but i don't know

if they do uh programs like this yeah anyhow uh welcome back andy it's been a while since we had a recording so we haven't spoken in a while but it's been pretty busy i'm sure on both of our sides and um yeah i've been uh i've been fortunate enough actually to well take some time off two weeks on vacation um after my now lovely wife said yes to me. So that's, I think, the first episode as a married man, I believe. And thank you. Thank you. And yeah, it was a great trip that we had.

And last week, I actually made it to Dublin to a testing conference and talked a lot about DevOps and transformation, which is also the topic of today. Nice segue. Nice segue, Andy. Very professional. I know. It's like I've been doing this before.

Yeah. So today we actually have two guests and it would be great to actually have video because before we got started, I had the chance to actually see our two lovely guests today. And Nestor, one of them, is wearing an awesome Dynatrace shirt. Maybe later on we need to take a picture and post it on the website. But there's actually two people on the other side here. And I hope I pronounced it correctly. Abir and Nestor from Citrix. And I

actually got to talk with Nestor a couple of weeks ago, because we chatted about what Citrix has been doing. And he came up with a couple of thoughts of stories that they could potentially tell the world and showing how they transformed their organization. Now, without further ado, I want to actually pass it over to the two of them. And first of all, Abir and Nestor, maybe you want to introduce yourself quickly to our audience, what you guys are doing, your background, what you're currently doing at Citrix, and then really dive into the topic of today with this Ops to transformation. That's the way we labeled it. Absolutely. Thank you, Andy, for the intro. My name is Abir, as you said. I've been working at Citrix for a little over a year now, and I'm a certified Scrum Master. So I actually have been

working with a full-time Scrum team team as well as helping transform some of the infrastructure groups within our organization into more agile processes and principles and mindsets. Cool. Nestor? Yes. My name is Nestor Zapata, and I work at Citrix Systems as one of the lead web systems administrators on that team. We manage a slew of applications ranging from Windows technologies, Linux technologies, containerization with Dockers, AEM, and things of the like. So our main focus is to become a little bit more agile as we're getting much more infrastructure and applications under our umbrella and portfolio. So we want to make

sure we're able to grow as fast as our customers and our departments would like us to do so. So we've been doing a lot of automation in our side of the house, and including that part of digital transformation includes being more agile. So we've had to kind of change the mentality of how we work as a basic operations team into more of an agile team. So that's something that we've been doing for the past couple of years. And, of course, heavily used in our landscape is Denitrace across all our applications. And we enjoy that transformation so far and the tool set and help that Denitrace gives to us. Cool.

Hey, and so here's a big question. Now you mentioned agile, you're on the op side. Can you tell us historically, has agile, like agile principles been adopted by the development side first? And then you saw the need of also applying some of these principles to the operations side? Or, you know, who, who, you know know how did it work historically I know I'll be here you have only been there with since a year but maybe historically do you know that the background may nest and maybe you know a little more how that transformation started yes so historically other organizations what I've seen I've seen the issue that the developers or the pre-production team are going a little bit more agile, and then it takes a long time for the operation guys to get on board.

But interesting enough, here in Citrix, our development team is not – it has certain aspects of agile in the use cases, but the real push from agile or DevOps, as we're going to go ahead and call it, came from our team, specifically from the web operations team. So our manager at the time focused on automation and focused on us being a little bit more quicker to respond to our issues and agile in what we use. So we started doing scrum sessions. We started doing stand up meetings and we started going that route. And along with that came a lot of changes on how we were reacting to issues. So the push came from the operational side. And now we've kind of pushed that mentality into the pre-production and development team.

And they've adopted some more concepts of agile and are benefiting from that as well. That's cool. So the motivating, if I understood this correctly, the main motivating factor was that the operations team was not fast enough to react to problems. And therefore, automation and a more agile mentality and mindset was hoped to solve that issue. Is this kind of a good summary? That's a good summary of that. Andy is a master at summarizing. I'm not sure if you know that.

And Abir, I mean, you are a certified Scrum master. And I would assume that you played a major role in actually teaching people what does Agile actually mean. Yeah, absolutely. So my role, I actually had slightly different of an experience than Nestor in that the team that I had been working with and have been managing is more of a development team. So my introduction to working with Nestor and the web operations team was kind of my first introduction into more of that operations focused mentality. So it really did open my eyes and there was a bit of a learning curve on my side where before I could go in and provide suggestions and support the change for them, it required me kind of understanding a bit more of what their day-to-day looked like, how they processed tickets and responded to issues, how escalations worked in their group.

And through learning some of those tidbits of information, I was able to kind of take that experience of knowing what worked for a development team and taking the agile principles and figuring out what worked better for a more operations-focused team. So where Nestor mentioned they had actually started out more Scrum-like and shifted into almost a Scrum bond type of mentality or methodology, I actually worked with them on refining it into more of a refined Kanban process instead, as I think that that has a little bit more of the flexibility needed for an operations-focused team. So were you surprised when you were learning that operations is just working differently and has different constraints than when working with pre-production people like developers in

particularly in the past? Maybe not surprised as much as intrigued. You know, I kind of understood that there's a need to be reactionary on operations. I don't think I realized quite how reactionary it really needed to be and how much pressure from the business is applied to SLAs and to fixing issues that arise pretty immediately. So that did throw a bit of a curveball into identifying what works and what makes the most sense. But, you know, the spirit of Agile is continuous improvement, right? So, you know, we kind of went in, we evaluated, identified some potential solutions, and the team has continued to review them and reflect and make some slight tweaks and edits to the process that I helped them create in order to make it continue to be an efficient flow. Mm-hmm. So when you looked back, you said you started a year ago, what were the big learnings and the changes that you did in the first year?

Kind of, if there's like some key lessons learned that you wanna tell the audience about in case they also think about how to make their ops teams more agile, more DevOps-y. Looking back at the last 12 months, what are the key, the big things that people can learn from you? Yeah. So one of the biggest things was coming from a scrum mentality with a two to four week

time box to iterations. The biggest thing I learned was that's not a really good flow for an operations reactionary team. So switching to a Kanban one week planning iteration, I think was the biggest thing. Instead of trying to force into those longer iterations when you're an operations team and you have to do a lot of firefighting, I think that one week planning gives you enough structure and enough diligence to do some planning, but allows you to open the option to continue to strive towards fixing things and, you know, reacting to issues and SIs and, you know, MIs. So that was the biggest thing. And

in addition to that was really just kind of changing the mentality for approaching these firefighting situations from an all hands on deck to a subject matter expert reacting and then only bringing in additional members of the team as needed, which opens up the other members to continuing to work on automation and other improvement processes versus the old mindset where it was there's an issue, all hands on deck, everything stops. And, you know, you lose a lot of traction on multiple efforts because of a singular problem. I was just wanted to ask, when you talk about the automation piece of operations, are you speaking?

It might be both, but are you speaking in terms of automating the operations processes of, you know, spinning things up and down or is this more automation of reactions to certain predictable events and repeatable events like sort of the no-op style of certain events occur we have a playbook let's automate that playbook or is it all of the above that would be all of the above so part of our motto is if you have to do anything between now and the day you retire, it should be automated. Creating a new app or restarting a server, none of that should be done manually. So our mindset is that we have to do it between either in this job that we have, our job description, another company, whatever it is, we should automate it. And that goes to just the basic tasks on operational,

but also the testing that we have and run books for certain things that we have an issue that pops up we know what's gonna happen and we go ahead and restart certain services and servers or contact somebody we have a run book and in that run that we have scripts that are ran from for example Jenkins and anybody from our team or even down to the service desk level one and two you can go ahead and run these scripts and not be able to bug down any other team member with certain tasks that used to take probably an hour now have been reduced to five to 10-minute tasks.

Awesome. Cool. Can I ask you, so if I go back a year again, when you started looking at one-week iterations, did you kind of reserve a fixed amount of time per week for these firefights? Did you have some historical data saying we need to set aside this amount of time, let's say 20%, 30% for firefighting, and we can only plan, I don't know, 70% of our other time

for making improvements or doing certain tasks? How did that work? Yes, that is correct. So one of the things that Kabir mentioned that she had to kind of do a different mindset is that we told her we have a lot of system incidents, major incidents, and last-minute tickets that come in. The mentality is that everything has to be planned either in Kanban or in Scrum. Well, that doesn't work that well because if production systems and its multimillion-dollar operations cannot be performed because our system is down,

it doesn't matter what's in JIRA, we're going to take care of that production issue that's affecting money coming into the company. So we allocated about 20% of our week that in case of any firefighting issues, anything we had to do, we can go. So each person has a 20% cushion on any issue that could surface. And then the rest of the 80% of the time that was left were things that were in our G report that we had pending tasks or WIP and things like that. So we did leave a cushion about 20%. And we gauged on that on the metrics we don't have official metrics for that but kind of kind of eyeballing it per se of what we experienced in the last month or so so generally it's been between 20 to 25 percent and we've become pretty close to that number

and have you are you i mean i assume this is a great number to track as well in terms of how much stabler how much more stable your environment and your operations team is. Has this number been consistently 20-25% or is this number going down or is the goal for you as an ops team to actually reduce that number? So yeah, one of the goals is that, Abir mentioned to us as we get more proficient in what we're doing, we automate things a little bit more, then that number from 20-25% should be reduced.

And we have seen that. We have seen the amount of incidents based on the automation, we have been reduced to about like 15-18%. We're not going so much and passing the 20% for each week. So that's been helped because we're organizing our tasks. We're trying to improve our analytics to be more proactive about things that are reactive. So yeah, certainly that number, our goal is to reduce it by a certain percentage. And any success that we have with that comes from the structure and the agile mentality and activities that we're doing. That's pretty cool.

Now you mentioned so that you have one week sprints that means do you have planning meetings and review meetings um every week monday friday or how does how does how does a week look like if you if you go from monday to friday what's the what are the key milestones like we have we to have it on Friday, our backlog and grooming session, but since now we have a different resource in the Dublin area due to the time zone, it was better for him to be on that call as well. So Monday morning, first thing in the morning,

we all come in and we all get the task and our backlog that we plan to work for that week, whether it be the tickets that we have in snow, whether it be any assignments, projects, or things that we think we can deliver by the end of the week, we put those in there. And as each session goes for our daily standards, we keep putting things in work in progress, backlog, or just things in the back burner, and we'll complete them as they go. So on Monday morning, we'll come in, we'll allocate what we think should be the rest of the work for the week. We'll be generous with the time, that 15, 20% that we have, and we go on. If we see that it's gotten better, that we're getting faster at what we're doing,

we go ahead and add a little bit more tasks to our timeline there. So we had the retrospectives as well. That thing helps us obviously look back the week prior and see what went well, what we didn't do correctly. We've improved a lot of our processes since then. You know, whether we're closing tickets out incorrectly, we're taking too long to close them out. So this has actually improved our ticket SLA in Snow and ServiceNow. So one thing that used to happen in the past, we used to get tickets assigned, we used to get bogged down with other items, and those tickets kind of fell between the cracks

between team members. So now we know we're highlighting these items and we've improved our SLA based on the retrospective because we improved any process that we saw deficient the week before. So always improving, always making ourselves better as a team. You know, it sounds... Oh, sorry, go on up here.

Oh, no, that's okay. I was just going to say to add to that, the kind of benefit for an operations team working on one week iteration is the meetings for the planning and the grooming and the retrospectives are all shorter than your normal traditional two to four hour sessions to get through things. So they're able to spend, you know, a shorter amount of time, still get a good level of planning to communicate to their customers, to communicate to their management and have that level of transparency on the board. But they're also not bogged down by hours upon hours upon hours of meetings when they need to react to things. Right. And it sounds like it's a very the environment you're doing this all in, like the corporate environment is extremely supportive. I know we we saw on our own transformation in these ways that it really seems to require a top down support from from the C-level all the way down saying, we're going to make these changes, we're going to support you in these changes,

and we're going to allow you to make the mistakes necessary in order to get this running properly. It sounds like that's kind of what you guys have been through, but has the experience been any different? No, yeah, you're absolutely correct. And we are fortunate to have it from our CEO down. They're constantly speaking about wanting to see more agility in our organization, wanting to identify more ways to have shorter iterations, better release cycles, more improved automation. So we've been very fortunate and it's continuing to trickle down. Our entire engineering organization has been working years on becoming fully agile and they're three years in and almost there. And so we've got a lot of that support and we have a lot of the collaboration,

which is even more helpful. You know, we've got Slack as one of our primary communication channels. We're able to create an agile Citrix channel, share stories about what works and what doesn't work in the different kind of teams and groups. So it's been very fruitful and it's been not as combative of a change as a lot of changes can be but were there any i mean it's great that it went kind of smoothly and you see the benefits but especially in the beginning where are people or certain part of the organization that could not cope with the change you know was, was there friction? Were there people that couldn't deal with the change? Yeah, I'll let Nestor speak to a little bit of the culture changes

that we went through. So, yeah, as being a traditional, what you might call a waterfall team or a basic operational team, it was kind of hard for some of our current team members, specifically team lead and things like that, to kind of grasp the fact that when something happened, we were not going to have three or four engineers looking at the same monitor and giving ideas. It was kind of hard for certain people to grasp the idea, well, how can I allocate 20% of my time? What if this week brings more firefighting issues or more

emergencies than what we planned out? So there was a certain pushback on whether it was a ticketing system or the technology being used or whether Jira was the best way to go. But as we were able to produce a benefit from that, we were able to showcase the improvements that we made, not just on our systems or application processes, but as our jobs go, we were bettering ourselves as IT professionals. So as those things came to the light, that pushback that was once there came less and less resistant. And then as the weeks went by and a full quarter went by, I started being open and think, okay,

I see the benefit now. I see why we should do it this way. So from our own team, we have a couple of folks that were kind of like hesitant about it. I wouldn't say a complete pushback, but there were a lot of questions. But what if, but what if in our retrospective and grooming and not attending every single standup. So as we saw the benefit show there really quickly, we were able to bring all of them on board. And as a matter of fact, some of our other operational teams, their integration, their stock database,

now they're intrigued to know how we function, how we operate so smoothly, how we know everything that's going on. And now they're asking us questions that normally would have gone probably to a beer on how we can do agile. So that was pretty interesting to be kind of an example on the operational side to be like a trailblazer,

if you will, on this side of the house, an operation team running in somewhat of an agile fashion. That's pretty cool. Yeah. And in terms of that, just for any of our listeners who might be undergoing this transformation right now or who are going to be entering into it soon, you mentioned seeing results and being able to see how it's improving.

What would you say are important kind of metrics or statistics to bring back to folks who might be downing it or even like the management teams to show, yeah, this is the payoff of our labor and this is a worthwhile experiment, or not experiment, but change. What do you use to prove to them that it's working? Well, one of the things that was very visible to us was the fact that we were putting what tickets we were planning to work on. The actual ticket number in our SLA improved greatly, talking about 30, 40% of improvement perhaps in our SLA, because now we weren't letting things sit through the cracks. We weren't letting items go into the next week,

and we didn't know about these sometimes. So that right there showed value. And just having that retrospective of that daily stand-up meeting, it also showed the value that every stand-up, everybody knew what we were working on. So when our managers would come by and say, hey, who can I grab to do a specific task? We were able to go to our G-report and say, well, XYZ is being done by, you know, IT engineer one, two, and three, but IT engineer four is open because he's working on something. There's a backlog. So maybe he or she can be tasked with this right now. So management was able to see clearly how their team was being utilized. So it was very good for the team lead managers to show upper management, what teams are working on, how much was getting done. Instead of blocking them down

with the status report, not all details go through. So that was also something very important. So reducing the SLA and the deliverable, excuse me, about a certain project or task that were assigned from business users, instead of saying, yes, I'll get to that, and then it getting lost in a shuffle of emails or just a quick conversation,, I'll get to that, and then it getting lost in a shuffle of emails or just a quick conversation, we were actually putting in JIRA and then make ourselves accountable. Like, hey, you promised this person that by Friday you'd get it done. It's Wednesday.

Have you worked on it? Oh, no. Let me get on that. Let me put it on the board. Let me put it as a priority. So things were being done a little bit more granular and things were much more visible. So the result to the business and to the customer base was much more improved efficiency that we could deliver to them. That's pretty cool.

Actually, I want to mention one little phrase, actually, Nestor, I think that you put in the email and the preparation of this podcast. We always ask people to come up with a list of bullet points. And I really love one phrase, which I believe is actually describing what you just said, which is stop starting and start finishing tasks, right? You basically really work on the backlog. You want to make sure they're not tripping over to the next week. You're measuring work in progress and you're trying to keep work in progress as small as possible. And also the time, a ticket stays open, reducing that as much as possible.

I love the phrase, stop starting and start finishing. I take full credit for all that. I thought so. Even I can't take credit for that one, Nestor. But that is the spirit of Kanban and one of the biggest between, and you mentioned also the work in progress limits, between stop starting and start finishing and work in progress limits. I mean, if you can kind of get a good handle of that and a good grasp on that, I think you can really excel and agile in Kanban. I remember going back many years when I was working on your side of the fence in testing, how many people would just go up to the operations folks and interrupt whatever they were working on? We're talking about five years ago or something before Kanban was really picking up. And I'm amazed they got anything done.

So it's really, really an important concept, that whole Kanban bit, and really just working on your tasks and don't start the next one until this one's done. Oh, yeah. People tease us now because people come up to ask a question and they'll be like, nope, if it's not on the JIRA board,

those guys won't touch it. And they say that's a joke, but it's known now that there has to be some sort of structure. And if it's not on the board, if it's not on the JIRA board, those guys won't touch it. And they say it's a joke, but it's known now that there has to be some sort of structure. And if it's not on the board, if it's not an emergency, then it's going to be evaluated and put on the back burner until next week if it could wait that long. So if not, we'll take something out from there and plug it in. And our managers know that. Our teams around know that. So it's kind of good that they know that that's the the kind of scope that we're working on and and you know i going back to i think what you had

commented on if any of your listeners are going through this or going to go through this and and what were some of the struggles that they probably will go through and one of the biggest things that i have faced dealing with multiple teams is finding a way to get people to shift their mindset from meetings as being almost an imposition on their calendar and, you know, something tedious or something they don't want to participate in into seeing the value of having these meetings, whether it's a sprint review or a retrospective or a planning session or even your 15-minute stand-up. So one of the things that probably anybody going through this challenge or through this change will face is having that culture change of shifting the

mindset of your people and your customers and your, you know, anybody in dealing with this change, that meetings are still going to be perceived in the beginning as an impediment or just a blocker from them being able to do actual quote unquote work. But stick with it, show the value after the meetings, the more that people can continue to present the value that comes out of that meeting, the more easily the change will come in the other people's mindsets. I like that. I mean, I, you know, I also have had times in my life where I had meeting fatigue. I'm not sure what the right word is. It's like, you know, you have to go to this meeting and it's going to be 20 people. And then it takes 15 minutes to introduce everyone.

And then in the end, it feels like wasted time. And I think there's, I mean, it's great that, you know, we come a long way, I believe. But not every organization has changed their meeting culture. Can you give us maybe one or two tips on how to make these meetings better, more efficient, and so that people actually walk away and say, this was actually helpful for me. Actually, this was not, it didn't feel like I lost time and that I should have used for other work, but actually that it was beneficial, like one or two tips and tricks on how to change the meeting culture, because I think this is a big part from a cultural change too.

Yeah, absolutely. I think two things that I could say is making sure that there's some structure to the meeting. I think structure and understanding of the goals and the outcomes are critical to the participation of the individuals attending. And that would be point two is getting that participation and that collaboration. So people that might not seem to be go-getters or the most vocal, you know, asking them questions, getting their insight, you know, really just, you know, as the facilitator of the meeting, making it a point to get everybody's opinions and, you know, getting everybody involved. And in the beginning, again, it might seem like pulling teeth.

It might seem very arduous to accomplish, but as time goes on and the structure is known, more individuals I have found begin to willingly and actively participate in the collaborative conversations versus having to be pulled into it. And I think engagement is key when you've got a group of people on a meeting. If you're not engaged, you kind of start playing games on the side, reading emails, doing something else, and you are now viewing that meeting as just something in your way versus something you're benefiting from. Yeah.

Hey, in all your transformation now within your ops team, how did the relationship with the development teams change and how are they involved? Because I assume not only do you want to become more efficient and more agile in your team, but I assume your organization is also moving towards what we quote-un unquote called DevOps continuous delivery, delivering software faster directly from development into production. Is this, I know I'm stretching this a little bit, but what has changed in the last year with the relationship to development and what's going to happen next maybe because i

assume dev and ops is getting closer together oh yeah we've um closed it up quite a bit there so some of the items that we've improved on is to kind of showcase the value we've we've delivered to the business and then that caught the attention of the pre-production team and the development team saying well how can we participate in that as well so we started telling them what we were doing kind of the agile transformation that we've had so some of the simple things that we've done is basically they're on a different floor than than we are a different building actually so what we've done is certain days out of the way like a monday or tuesday we go and sit on their side of the house and we sit down we talk we converse about the items that we're working on. So they can see

the struggles that we face. Sometimes they think the red tape or when they put a change request, it's, you know, red tape and, you know, political things not to get their change out quicker. But now they realize why it's done and the SOX, you know, requirements that's needed when we make a deployment into something like a financial system. And also we are there on that side, and we see the stress that they go to to develop something, to try to deliver something, to test something very efficiently. So we're taking on our continuous development skills

and scripting and things like that and kind of like giving them to them so they too can benefit from the tools like Jenkins and Chef and things of the like. So we've shared quite a bit of knowledge and kind of helped them become a little bit more agile themselves. And we're continuing that transformation and both leadership on both sides of the house are now kind of getting a little bit together. And they're noticing the benefit, the positive effect that they have done to their teams. And we're just getting closer with the concept of whenever we do a deployment, kind of include us in, get our buy-in. So when it comes to production, there's nothing brand new that we're seeing for the first time.

So there's a lot more collaboration and input from our side and both of them to us. So that's been a great benefit for us in making things a little bit more seamless when we deploy. And the code, when it gets tested, also gets a little bit cleaner as well. That's cool. Is there, like I'd say, in a year, two or three from now, a vision where there is no ops team anymore and no dev team but it's just going to be engineering teams that are pushing changes into like your life system or your staging system is there a plan yes there is a plan actually we're taking that in chunks by technology so there are certain technologies for example new york projects that we deployed using chef and the tool sets of Denitrace to monitor everything. So we've gone from the test environment to the pre-production environment all the way to production.

And that technology itself is being deployed by what currently was a pre-production engineer. In the past, a pre-production person would never be allowed to touch anything with a production name on it. But now we trust the system, we trust the automation that we can have anybody from that team go ahead and deploy. So we're taking one technology at a time, one system at a time. And eventually there has been discussion at a much higher level than mine appears to go ahead and merge these two teams together because they're seeing the benefit that comes from combining both technologies and both environments under the one same roof. So those talks are in play. I'm excited to see what is going to come in the next few quarters.

But for 2018, we expect quite a few changes to happen, and we're looking forward to that. And going back in terms of on the previous question that kind of led into this one, in terms of the idea of you all doing your part and the development teams, you know, working their side of it, obviously, in the future, they're going to merge and you're going to have this really beautiful system running the work where you're at now. You said you had all these meetings and all that. But can you possibly foresee you getting to where you've gotten without having those conversations with dev and can you can you you know say whether or not dev would have been able to do that if they weren't having those conversations with your teams i think it would have been very difficult for us to get to to that

level without having conversation because there would have been no value they would have been seen we would have been seen like an obstacle, and we would have seen taking over a pre-production environment as a struggle or like a determinant to whatever we're planning to do. So we're kind of offloading some basic operational tasks, like the creation of some new users and certain basic tasks, and then focus on more of the continuous delivery and improvement with the tool sets that we have. So performance and the metrics that matter is a big thing that

we're working on now. So without having those discussion with those teams, I think it would have been very difficult to showcase what we've been doing and what benefit would have come for joining both those environments together. Right. So could I bring that up only because I have a buddy in, hey, Scott, if you're listening, who's in a similar situation where he's working with the architecture team. And I'm sure this is, you know, I'm not just asking for him, but for anybody really. But there's, you know, we were discussing, he's got this new role.

They got this new guy in on the architecture team. And they're like, hey, we're going to really improve the way we're doing things. We want to start getting at, you know, a lot more agile and do all this crazy fun stuff on the testing development side. And he was asking me like, oh, who should I, you know, who should we have in that meeting? And I'm like, well, who's your operations team? You really need to get them in there in the beginning to start collaborating with them because, you know, these points have to merge at some point. So I just think that's a great point that, you know, you're talking with the dev teams,

they're talking with you, and that's all started early so that when you do start getting to that merge point, you know, most of the, you know, if you think about it as two roads merging together, well, you both have the same vision, same path, and you're both building the same kind of road so that they can merge together seamlessly. So that's awesome correct yeah absolutely um i got one little topic that i want to just quickly discuss even though it has been mentioned before um monitoring i mean we we really try to focus on the on the transformation but you mentioned that monitoring is obviously a big part of what you do because you, as you look at the metrics, are there any lessons learned, any things that you have changed maybe in the last year

on how you use monitoring to, you know, improve your processes, especially, you know, how the, you know, maybe going back again a year from now, back a year, how it was back then and how it is now, what you've improved from a monitoring perspective? So as a monitoring perspective, one thing that we try to improve as an operational team was include the CNOC or the monitoring team

that we have here at Citrix early on. Because what would happen sometimes, and this is not just with Danitrace, but any other product that we use to at Citrix early on. Because what would happen sometimes, and this is not just with Dynatrace, but any other product that we use to monitor any other system, we would go through the project, we would go ahead and deploy, we would put it into production. And once it was in production, there were times that we were like, okay, when this breaks, who's going to contact us? And then we'd be like, oh, yeah, we've got to contact the monitoring team.

And then at that time, once it was live, we'd go back to the monitoring team and say, hey, we need these 20 servers to be monitored. These are the threshold. This is what we've got to do. And it has to be done by the end of the day because we kind of forgot or somebody forgot to put that line item in the production. So we had been there once or twice before. So it wasn't the best model to have

so as about a year ago we started including a line item on the project timeline that we had to include monitoring we had to sit down with them for them to let us know what was the best margin tool to use for specific technology specifically with denitrace what we're doing is we have it in our pre-production environment so any application that online, we go ahead and build out a dashboard for that. We make sure that everything is good. We ask the business users and the developers, what are the key items or key things we need to find out? That way, we have business transactions already created for them. We have specific dashboards created for them.

So they can be specific for IT, development, or business users because those dashboards are going to be different for each of those three categories. A business user might not want to know that a specific node is high on memory because when they hit the load balancer, everything's fine. So for them, it has no effect. But for the engineering staff, they need to know each node and their specific metrics for that. So we've been working very hard to incorporate predictive analytics on our side, the monitoring more in-depth on our side. So it's not just is it up or down. Is it flowing through? Can you make a whole transaction? Can you attach this license?

Can you attach this file to process payments? So it's not just is the service up and running? Is it fully operational? And we do that specifically with Denitrace, with the peer path, the transaction flow. We're able to see all those items come to light. And we use that from our pre-production to our production environment.

So we highlight the importance of the application performance management and the monitoring very early on in our process now. And it's become a staple in anything that we do and that we spin up. Cool. performance management and the monitoring very early on in our process now. And it's become a staple in anything that we do and that we spin up. Cool. And Andy, I think that's a really important point, right? That Nestor mentioned pre-production, right?

So if you're getting developer input on what to monitor plus the operations and everyone else's, you can, you know, monitoring those metrics in your lower environments early on is what's going to provide the feedback to those development teams and their development cycles to know whether or not whatever they're developing can be pushed forward. Because as we know, there's always this, yes, the test passed, right? But at what cost? So it sounds like you're all getting early feedback in those earlier cycles by leveraging those common monitorings and having discussions about what needs to be monitored. So that's really awesome, too, I think. Yeah, and also if you are building dashboards for each individual app, then I think it's a great validation on whether these dashboards are actually useful later on in case there is a problem.

So do I see all the information or do I have blind spots? So I think this is a key part to validate your monitoring in pre-production. Would that be monitoring as code well monitoring i think monitoring is called is a little is a it's something a little different for me but uh it's monitoring as code for me is allowing developers to define in their code what they want to be monitored right yeah i just meant more more in terms of setting up your monitoring and validating all that stuff and graduating it through your lifecycle with checks and balances as you go through. So by the time it hits production, you've got everything worked out and vetted, treating it as its own code deployment in a way.

Yeah. And then you need to test your monitoring, too. That's actually an interesting aspect now. How can we test monitoring? Stay tuned. Yeah monitoring stay tuned yeah stay tuned yeah yeah exactly hey um well thank you so much abir and nestor anything else that we missed anything that you said this is something i want to just you know make sure that people out there are aware of things that we ran into and that you should not run into. Yeah, I think the biggest or a good takeaway for anybody coming into this is just to know that it's not an immediate process change and it's not an instant results kind of factor. So I think one of the things I always try to teach when I'm

discussing agile to new people or introducing it is the adoption of agile should be done in an agile way as well. And so instead of trying to completely uproot a team's processes and methodologies and what they've been used to doing. Uh, there's always the ability to slowly introduce the change, add standups to replace a weekly one hour meeting ad, you know, JIRA or some kind of tracking tool to replace whatever post-it notes or Excel spreadsheet are utilized today and slowly start introducing things and getting your team involved. And, you know, it's important to have somebody understanding agile and processes to be the advocate of that. But I think you have more success when your team feels like they've had some voice in the adoption of the methodology. So, you know, for instance, you know, instead of

coming in and saying, we're doing a 9 a.m. stand up every day, be here or be square, you know, you introduce it as, hey, does 9 a.m. work better or 10 a.m.? Or, you know, just simple things that give the team a voice make a big difference in the results. It's not dictatorship, but it's a collaborative team outcome. Absolutely. And that's the spirit of all this, right, is everyone helping each other out and everyone buying in. So, awesome points. the spirit of all this right is everyone helping each other out and everyone buying in so awesome points nester anything from your side that you want to add so from my side as an operational personnel it's that we have to change the way we operate the old traditional way we used to do

things it's not going to work out as we start this digital transformation as we move things into the cloud as we start trying to be a little bit more quicker with our deployments and automation in order to keep up we have to become much more agile than what we're currently at right now the business model we had is not the best for the future and also to the folks out there that no matter how many fancy tools you have how many cool things you, whether it's Jira or whether it's using Confluence or Workday, whatever it is that you have out there. We've seen and heard this at many conferences, and they say that DevOps or agile transformation is 80% culture and 20% tool set. That is something very, very important to know because you have all the tool sets, but if you have nobody wanting to do the digital transformation or wanting to make that change, it's going to be very difficult to do that. So first get the buy-in from the ops guys, the dev guys. Talk about it.

Discuss it. And do it in bits and pieces, you know, so you can start with a certain technology and make that a little bit more agile. And then when you see the benefit grow from there but know that it truly is a cultural change in a in a mindset of how you normally did your job before cool brian is it time it is i think it's time for the summer raider let's do it so we need to we need to have her on the show more often i like i like it when we she starts smiling and laughing um so i think i think if i want to just you know sum up i i really i love the fact that agile is not something that was invented in death and then pushed on operations

because they saw the need but agile transformation can also come from the other way around. Like in the case of Citrix, where the need to transform the way operations was executed had to change. I like the fact that you were sharing your experiences and figuring out that a one-week iteration might be better suitable for operations teams, reserving between 20% and 25% of the weekly time to firefighting and to handling situations that just come in, but also measuring this number and hopefully getting it down over time, which actually shows how better and how more efficient you are in dealing with critical situations. And automation is obviously a big part of it. I also like the fact that just as with on the development side, I think tracking work in progress, setting a limit to work in progress is a key thing.

Not to be overwhelmed with too many parallel tasks that you have opened and never finished. I again like the phrase, whoever we need to give credit for, stop starting and start finishing. I think that's a great phrase. And in the end, all it comes down to, it only works if the culture changes as well. Change might not always be easy, but it's the right way forward. And people that were resistant in the beginning will see that there are benefits in the end. And some people just feel it. And some people need these metrics to see that the change is actually really working.

So that's why I want to say thank you so much for what you shared. Brian, anything, any last words from you? I would just like to say this. What was really interesting to hear on this one today for me, and thank you both for sharing, was a lot of times, either on the podcast here itself or just in general stories out on, you know, online, and a lot of stories you hear from companies about this are ones that have gotten to the end point. And it's, Hey, this is where we, you know, we made it to the final goal. And this is some stories you're, you're all in the middle of it. And we don't hear too many stories from the middle. You've got a long road behind you of

things that you've already done. You obviously see there's a bit of a long road ahead. Um, but you have the, the backing of your organization, the team members are all coming along, but we don't hear too much from, from people in the middle of making these improvements. So I just really appreciate hearing the story from, I guess, like the rest stop, if you want to think about the, the highway model, right? You're at, you're at one of those really nice rest stops and got a lot of great stuff done. And, you know,

we're going to be checking out in the next leg of the, of the trip. So hopefully we'll get to hear back from you when you hit the next, either rest out in the next leg of the trip um so hopefully we'll get to hear back from you when you hit the next either rest stop or the next big milestone you come across because it'd be great to hear um you know what other developments and and hurdles or just success stories you had from there so thank you all for being on thank you for having us this has been definitely enjoyable and and new and insightful for us as well so i i definitely appreciate yeah likewise so it was a wonderful experience to have to share the stories with uh you brian and andy as well so looking forward to the next opportunity if not um not

to perform yeah exactly we totally we totally forgot about that because Nestor, if you go to dynatrace.com slash perform and look at the speakers, you will actually see Nestor listed there because some of this stuff will be presented at perform as well, which is a nice plug

for people that are interested in hearing more from Citrix and others. We are hosting our annual global user conference in Vegas at the end of January. So make sure to sign up and join us there. And Andy, you'll be there?

I will be there. And I will be there. I'll be doing some of the learning sessions and I'll also be doing the live podcasting again with our good friend Mark Tomlinson and probably James Pulley from PerfBytes. So Nestor and Abir, if you end up attending, I'd love to have you guys come by and say hi on some live podcasting. Yeah, most definitely.

Absolutely. Abir and I will be presenting, so she'll be co-hosting. Awesome. Looking forward to that. Absolutely. Great. All right. Well, thank you all

for coming on today thank you talk to you soon bye bye thank you