Soft Skills Engineering - Episode 457: How do I get off the on-call rotation and "big tech" == "big leagues"?

Episode Date: April 21, 2025

In this episode, Dave and Jamison answer these questions: I am a senior software engineer in a big tech/faang company and this week is my first ever on call rotation. My team is doing a lot of CI work, monitoring pipelines and support queues during on call. It is probably not as much of a hassle as on call for product teams, but for me personally on call was the nearest I have ever been to hell. Our on call is not the regular getting pinged when something goes wrong, instead we have to manually monitor a dashboard 12 hours constantly for 7 days as the alarming is quite fuzzy. I am the only EU remote worker that has to adopt to the on call PST timezone. That means, my on call shift goes from 3pm-3am in my timezone. It is day 5/7 and I am down 24 energy drinks already, cause this was the only way to stay wake. Knowingly, that this would be just a short-term tradeoff against health, I am now living through the most explosive diarrhea I have ever had. On top, I am sleep derived, dizzy and every body part hurts. That would already be terrible on its own, yet I additionally have a young family, with a 4 year old and a toddler. The on call week, has not only been though on me, but especially also on my children and wife. I don’t have time for the kids at all and my wife is doing 100% of everything at the moment, including waking up, breakfast, bringing our son to kindergarten, cooking, cleaning, playing, everything. She is also quite exhausted therefore. Besides On Call, my job has been great and a huge monetary opportunity that is very rare in the EU, therefore quitting just because of 4-5weeks/year is not an option I am considering. Yet, I am wondering if there could be any way of smuggling myself out of the on call rotation. I have seen, that a staff level engineer on our team is not participating in the rotation, but that might be because he got a lot going on with other teams as well. A listener named bebop asks, Is your average “Big Tech” dev “better” than a random dev selected from a large non-technology company? I can’t help but feel that if I want to level up my career, I’m going to have to either move into big tech or some unicorn startup.

it takes more than replacing agentic workflows with a stored procedure to be a great engineer this is soft skills engineering episode 457 i'm your host dave smith i'm your host jamison dance soft skills engineering is a weekly advice podcast for software developers who think a little not highly enough about agentic workflows i feel like every time there's a technology that becomes popular kind of builds up a bunch of abstraction around it and then you always have some reaction to say what if we just did the simple thing you've got like ruby on rails and then there for a while there was sinatra which was like what if you just handled http requests and did the rest yourself you've got what htmx for kind of javascript client side stuff so maybe stored procedures is is the

backlash yeah you got this giant pile of expensive coal burning gpus what if you instead just did some sequel yes well you can be the stored procedure thought leader in a decade probably when there's a backlash that is big enough yeah exactly exactly i saved us from agentic workflows yes should i thank our patrons please do thank you so much to the folks that contribute at the level that we shout them out every single week thank you to croissant connoisseur i only learned to code so i could look cool in public but now i get excited about cleaning up legacy code please help nice noah labhart i'm making good progress on my dad jokes cron job but i need to discard all qa and testing requirements to meet my deadline yes a really expensive tube nice

alexander kuznetsov nick molyneux attribute error none type object has no attribute to string javier gonzalez chewy ted timbrel i quit my job in 2025 and got a way cooler one as a software dev thanks ssc podcast i like to do the cha-cha whoosh sad emoji i think it's because it doesn't say like a sissy girl anymore because i'm playing enough okay rest in peace become a senior engineer.com is a newsletter you should read doing my part makes dave's heart swell but he should probably get that checked yes dan from drone to play chase w norton never is not just a crater on mars flamingo emoji i like chicken i liver myomics myomics please deliver trash panda get status kyle boss kensi dodds that guy over there nevar is not just a planet in the in the vulcan system

jenny kim the stochastic parrot helicon.ai best observability tool for ai red panda is best panda how much wood could a much wood could a wood chuck could pretty good pretty good b plus i thought it was an a plus so i disagree all right jonathan king it's not a beautiful functional user documentation i'm your host steven colbert what nice imagine one of those medical disclaimers read in a super fast voice after that please don't see us due to ai productivity gains william angel dot net no longer sells your data with love ragnar braden canes john grant britney ellick come to the slash new conference in newcastle australia may 28th and 29th moldy hard skills are safe to eat just cut one inch around the mold moldy soft skills must be discarded and

ouch my 401k yes so say we all yes or maybe if you're a zoomer you say you have 401ks lucky yeah exactly yeah thank you thank you so much to uh all those folks who contribute at this level if you want to join them you can go to softskills.audio and click support us on patreon where any amount will get you a shout out nope doesn't no it's testing you and you noticed right and yeah you pass good job we'll get you an invite to our slack team and enough of an amount we'll get you either a one-time or weekly shout out and at some unspecified number higher than that i will add you to the growing list of of tattoos on my body of all these names which as they change week to week becomes a problem yeah you're running out of skin yeah the strike through

feature bottom of the foot tattoos are starting to really smart but i do it for you because yeah of love dave do you want to read our first question yes this comes from an anonymous listener who says i am a senior software engineer in a big tech fang company and this week is my first ever on-call rotation my team is doing a lot of ci work monitoring pipelines and support queues during on-call. It is probably not as much of a hassle as on-call for product teams, but for me personally, on-call was the nearest I have ever been to hell. Okay, continuing. This is a good story, by the way. I'm going to just keep reading. Our on-call is not the regular getting pinged when something goes wrong. Instead, we have to

manually monitor a dashboard 12 hours constantly for seven days as the alarming is quite fuzzy. i am the only eu remote worker european union remote worker that has to adopt to the on-call pst time zone pacific time oh man that's a long that is a long that's like a 10 hour probably 10 hours depending on where you are in europe that means my on-call shift goes from 3 p.m to 3 a.m in my time zone it is day five out of seven and i am down 24 energy drinks already because this was the only way to stay awake knowingly that this could be just a short-term trade-off against health i am now living through the most explosive diarrhea i have ever had oh no i am on top i am sleep deprived dizzy and every body part hurts no one tells you about that

that part of on-call is the explosive diarrhea causes oh man that would already be terrible on its own yet i additionally have a young family with a four-year-old and a toddler the on-call week has not only been tough on me but especially also my children and wife i don't have time for the kids at all and my wife is doing 100 of everything at the moment including waking up breakfast bringing our son to kindergarten cooking cleaning playing everything she's also quite exhausted besides on call my job has been great and a huge monetary opportunity that is very rare in the eu therefore quitting just because of four to five weeks per year is not an option i am considering yet i am wondering if there could be any way of smuggling myself out of the on-call

rotation i have seen that a staff level engineer on our team is not participating in the rotation but that might be because he got a lot going on with other teams as well okay let's start with the diarrhea part that is not what a phrase i imagined you saying when we began recording today i'm gonna recommend some fiber in your diet yeah i'll do anything to fix my stomach issues except not drink 24 energy drinks anything else so i'm gonna say a bunch of stuff that i assume will not work but you being in europe could be an advantage because you could say great i will cover the my i don't know 7 a.m to 7 p.m time zone or i don't know some some time that is more convenient for you daylight hours or something yeah and that way nobody has to cover their middle

of the night kind of follow the sun model yeah that feels so obvious that there must be some reason why that can't work maybe you're on a team that is pacific time based and yeah i think it's like i guess part of the deal or something like that yeah you could you could kind of split it up into subshifts but yeah that that probably won't work but if it could boy that would be great it would still be a sucky on-call shift while you were on call but part of the pain is the the horrible hours not just the horrible shift the other thing that i'm going to assume you can't do is just why don't you just make the alarming less fuzzy yeah i presume if this is a big tech company saying that thousands of engineers have have shed blood and been chewed up and spit out

by this gnarly alarming system or system of systems so it's probably not trivial chewed up and spit out much like the food you eat during your on-call rotation yes yes exactly yeah and sometimes i have seen on-call rotations where it's not so horrible that you're you're not just always busy fighting fires so you use the time that you're on call to improve the situation in general it sounds like you can't do that if you just have to stare at a dashboard yeah then you probably don't have time to actually improve anything systemic i know right like isn't that the classic advice like you know after you finish your on call hopefully you left the system better than you found it it's like nope yeah i left it exactly as i found it being stared at by a human

being yeah yeah i guess i'm assuming those are not possible because of big tech fang reasons but if they were boy would that be great so i i knew of a team at the big tech fan company that i worked for some years ago who had a similarly intense on-call arrangement like this like lots and lots of issues lots of staring at the screen lots of weird hours and they actually went to a 24-hour rotation that was it they said every developer does 24 hours only that's that's the full duration of the on-call rotation assignment it's like yeah anyone can do that extreme stuff for 24 hours but beyond that it's like seven days of that in a row seems kind of hard does when i worked at a big tech company they did a bunch of e-commerce and around black friday we had a 24 7 it was about a

week of multiple people covering dashboards the whole time it wasn't you respond to an incident it was literally there was a constant zoom call with two or more engineers sitting there waiting for stuff to break and kind of doing other work because it wasn't always on fire yeah that was once a year and you were together with someone else similarly suffering and even then we had it kind of staggered so you would do like four-ish hours at a time and and we tried to rotate it so it wasn't someone just up in the middle of the night every time you do yeah so this this sounds worse than any on-call I have been a part of. Yeah, me too. This is probably the worst ever that I know of. I like being on-call too. I enjoy firefighting, but I would not like this.

If there's one thing I'm bad at, it's manually being very careful and thorough. And so if I was just like manually sifting through a constant stream of alerts to find the one that actually mattered for a week in a row i would oh i would fail yeah i would probably fall asleep that's probably what i would do yeah what should our listener do well i want to go back to the diarrhea i really would go to my manager and propose a more a shorter duration for this i would say look this is like this is inhumane and there is a really good chance that you can point to incidents that came to your team because of this rotation setup like guaranteed if you're doing this seven days in a row 12 hour shifts per day at unholy hours like this guaranteed someone has made a

catastrophic mistake in that time yeah and if they have then you should be able to point to it and say look this this rotation is not working let's change it now that the question asker says they only have to do this four or five times a year which tells me you've got approximately 10 or 11 team members yeah so what if you did three day rotations instead like would that help you think that would be easier to because i'm thinking like seven days is a long long time three days of intensity i could probably pull off but seven i don't know yeah i i mean it would help you'd have more of them, but it'd probably be easier to do it for a shorter time interspersed with shorter breaks in between. Yeah. I'm also thinking of your wife. I feel like she would probably rather

be a hundred percent responsible for everything for three days and then wait a month and then do it again instead of for seven days and wait two months and do it again. Exactly. Guess who else is staring at operational dashboards 24 hours a day? Yeah. It's not just you. Yeah. Brutal. i guess it makes sense that if you're up until 3 a.m working you're probably not getting up with the kids oh for sure i mean that's an option that could be worse for you and better for your family is you you still get up then you're then you're kind of doing the like night shift lifestyle which is horrible for lots of reasons but then you have to try and go back to sleep and then i assume that you're not working before 3 p.m when you're on call yeah i wondered about that probably not

how how is that possible i don't know and okay i'm thinking more about this too i'm assuming that their normal work hours are also not shifted by this much so it would be weird if they also just started work at 3 p.m and instead of going to 3 a.m they went to like midnight or whatever i guess 1 a.m sometime that is a normal work day i don't know there's a bunch of different ways this could be weird yeah i'm gonna i'm gonna guess yeah like if this is pacific time 3 p.m sorry if 3 p.m is eu time pacific time could be something like 7 a.m or maybe 6 a.m depending on where you are in europe and so maybe the on-call rotation actually goes like 6 a.m to 6 p.m pacific time but for you it's 3 p.m to 3 a.m oof still even still even though it's daylight hours for your pacific

time co-workers that's a lot 12 hours of high intensity on-call time for seven days in a row without a break is a lot yeah i mean even if you just say what about losing an engineer for a whole week maybe a little bit longer because they're probably a little bit fried after this totally even if it's during less egregious hours it is pretty awful there's got to be some way you could make a business case for we should invest in making this less horrible i imagine people quit over this i imagine people burn out over this definitely the worst i've ever seen so maybe it's a promotion incentivizer it's like yeah once you become staff you don't have to do this anymore so get to working on that sweet sweet promotion yeah that's the other thing we haven't explored is

great now you must you must become so essential you cannot be spared wait staff of engineering team's not participating in the rotation ask them ask hey how did you pull this off i want that i want what you have yeah how come you don't have to be on call if they get really uncomfortable like they start sweating it's like please don't tell anyone that i'm not on the rotation they just haven't noticed yeah turns out that uh our confluence page is publicly editable i quietly removed my name yeah three years ago that was the first step on my journey to becoming a staff engineer exactly it's realizing that you can just do things i guarantee you that everyone else on the team who's on call hates this they

maybe hate it less because of the time zone stuff but you should be able to have a bunch of i mean what if you got the engineering team together and said hey everyone hates this i hate it especially but i know nobody loves it what can we do to make this less horrible make make some kind of technical plan for here's here's a project we could do to make it so you're not monitoring 12 hours because the dashboards are so fuzzy maybe you still have to dig into it more than you would like to it's not just wait to get paged but there's got to be some way you can bite off a reasonable chunk of work that isn't like, we'll just do nothing for the business besides work on dashboards for six months in order to make this better. I think I would try that. I do think it's worth

bringing up to your manager as a concern that you have. I mean, if there's something making you miserable at work, I would hope you would talk to your manager about that thing, unless it's them. They're the ones making you miserable? Yeah, you're making me miserable at work. What can we do about that but i think it is fair to say this is really hard for me and i'm having this pain in my life because of it your manager probably wants you to be happy and effective and it doesn't sound like you're happy and effective during this time yeah i guess the question is to what degree does your remote location play into your pain and does your manager simply not hear about this problem from the other team members because it's actually slightly more bearable and it doesn't cross the

threshold for crazy amounts of pain for them because it's mostly daylight hours and and you might be nervous about rocking the boat with this remote arrangement yeah like well you already have this special situation anyways now you're complaining about it like this is this is this is what you signed up for kind of is that what you mean exactly or you just gave me a reason to think about firing you and replace you with someone local yeah they're probably gonna be more expensive if you're moving someone from the eu to california probably cost more money i yeah i think i would do two things i would try to come up with a specific project that has a reasonable scope to improve the technical state of being on call so it's not as manual maybe that's impossible and it it's hard

to know anything about the scope of that and i think i would talk about it with my manager and say what what can we do and if the answer is nothing then you know but at least you know it's not going to change without you doing anything probably so all right well have we answered the question i think so yeah i would be miserable if i were in this situation too it is not just you not being tough enough yeah you know software engineers those famously tough you know how like when you go into the navy seals you go through that training course where they keep you awake for like a week at a time and make you swim in frigid water that's important for software engineers too clearly is why yeah this on-call process will just make you stronger yeah in fact

i think that's called hell week and yeah yeah you just have several of them this listener calls basically the same says they feel as close to hell as they've ever felt well i guess you're doing it right then cool should i read our next question let's do it okay this is from a listener who claims their name is bebop i doubt that but that'd be awesome if it is listener named bebop asks is your average quote big tech dev better than a random dev selected from a large non-technology company i can't help but feel that if i want to level up my career i'm going to have to either move into big tech or some unicorn startup uh-huh comparing yourself what do you think is behind this question is it i actually didn't look at where they work i assume that they work at a

large non-technology company and so they're saying i want to i want to get better can i just move to this to a big tech company and have like really talented co-workers basically yeah i think that's what they're getting at. Yeah. I kind of wondered this in my early career as well, which was, am I being influenced by people that are going to maximize my own proficiency as a software developer? And you kind of don't know, right? Like you don't know where the ceiling is. And you find out, the best you can do is find out that the ceiling is higher than you thought when you get exposed to a new caliber of people. Yeah. If we're just answering the question as stated i would probably say yes i think with no data just my my gut feel if you take like a bank

and then like a fang company i would assume if you could somehow reduce the level of engineering talent to a number the number would be higher per per engineer would probably be higher at the fang company but there's still there's a lot of variance at every company even a fang company is going to have a lot of variance and if you're at a large non-technology company but they still have a bunch of engineers there's going to be great engineers there too so you could also try and figure out like who is the best person i could work with here as as a way to kind of level yourself up who's like the best where which is the best team here it might be higher variance maybe too at a large non-technology company or at least higher range yeah because there's just more people

or were you saying at a fang company compared to a big tech no no at a non-tech company yeah yeah sorry all right i don't know this is all speculative and i also feel weird because i'm i'm just i feel like i'm just judging humans broadly across one attribute which is what company do you work at but well it's like the only valuable way to judge someone i think i mean it does happen on if you if you are hiring you look at resumes i feel like working at a fang company looks better than working at a bank so i think it kind of already it is perceived this way whether it's true or not yeah and i think that probably kind of makes it true a little bit also if i want to level up my career i'm going to have to move into big tech or some unicorn start you don't have to

do that you can definitely level up your career in all kinds of places yeah for sure but i also think you can level up your career at a small startup not at a unicorn by doing a bunch of stuff you have no business doing. If you move to a big tech company, they will have had many thousands of person hours put into these like beautiful, well, maybe beautiful is the wrong word, but fantastically sophisticated tools and systems for scale and building and handling all these problems you get if you are operating a bajillion servers. And if you're at a tiny startup, the answer to every question is i don't know i'll go figure it out yeah you solve it um yeah so i think that's another way to level up it totally is go somewhere where it's all on you another way

to say that is that at the big tech companies a lot of the hardest most interesting problems have already been solved by someone else yeah and the new current hardest problems are are probably not the ones you're going to solve if you join a big tech company totally if you join netflix you're not going to be solving i don't even know what their hard problems are today finding good movies seems to be the number one finding finding good movies yeah that's kidding not canceling a show after one season that's the hardest problem yeah we hired a software engineer to make us stop canceling the good shows yeah no but seriously like i've worked in very small startups and there's something about the full weight of ownership on your shoulders you if it if you

don't do it it's not getting done whereas when you're a little cog in a giant big tech company you might be a really stellar high quality cog and you might be surrounded by really high quality cogs but at the end of the day your influence is most of the time very very limited and it means you might get inspired by some interesting things you see but you yourself statistically are less likely to actually contribute something really interesting yeah big trade-off i think i think a good a good career has both elements of both types of experience i agree and i don't know if it's because that is what i have done right and so i assume that is the right way to do it or not but no probably not it's just it's objectively right yeah probably

it probably is yeah when i think back about my time at a big giant mega tech ish company i learned stuff i would not have learned elsewhere and kind of outrageous scale and giant systems and things like that. But I do have more fun just building stuff and having more wide open space to build stuff. And I learn a different kind of thing when I do that. Yeah, I guess we've kind of pivoted this to another question, which is like, how do I level up at a small tech company or a small startup? I think there's a lot of those, so it should be easy to find one. So a unicorn, I don't know if this is still the definition. In the past, it's been a startup valued at a billion or more dollars uh guess what that is already a big tech company um unless you're like instagram or one of

these outrageously highly leveraged per employee or highly valuable per employee companies most tech companies by the time they get to a billion dollars of valuation are going to be many hundred or thousand of of employees so that's still not small and you still might not have the freedom to just go do a bunch of stuff yeah i think i think i'm just turning this into a question where the answer is go work at a small startup because that's what i feel like i want to say go do what jameson did yeah and then start a podcast yeah start a start a podcast yes grow a pumpkin in the window well that's in front of your screen right now oh wow not on purpose though i was gonna say was that an accident yeah some pumpkin seeds fell in there nice and they sprouted like

weeds i would like to credit the warm glow of my monitor keeping it alive during nice yeah the cold winter months totally i thought that your pumpkin is growing primarily because of all the patreon contributions and the warmth radiates off of them yes that too goes into artisanal pumpkin fertilizer oh last last comment for me on this is that just i just want to directly answer this question is your average big tech dev better than a random dev selected from a large non-tech company i would be willing to bet a nickel that if you randomly selected a developer from a big tech company and then randomly selected a developer from a large non-tech company the big tech developer would be better by some by some reasonable measure you know like able able to

deliver code faster more creative i don't know what the right judge and that's just based on my sampling of a lot of developers over the years from large companies that aren't tech versus large tech companies. But that's not to say that all big tech developers are better than all big non-tech company developers. I definitely ran into a number of, you know, like a not zero number of developers who I thought, oh, wow, you are just not like not the greatest developer, you know, at a big tech company. So they're there. But overall, yeah, like you can, I have met some amazing developers at these big tech companies, like really smart, really, really great software developers. So I think it's awesome to spend at least some time in one of these companies in your career yeah I

mean another way to ask this is do you think it's harder to get a job at a big tech company than at a large non-technical yes as a developer I would say probably I think they definitely try to and pride themselves on having a higher bar and trying to filter more yeah there's certainly exceptions and you can be great anywhere and your value as a human does not depend on where you work but i do remember especially earlier in my career i had this feeling of like i don't think i count unless i have worked at google somehow like i i think i have to try to go work at google just so i can say i worked at google and prove to everybody that i can do it and i think they certainly enjoy that that feeling that like people want to go there to show that they're

competent and capable i don't know i'm just kind of rambling about this all right that's what we like you to do you can be great and learn anywhere good i'll keep doing it you can't stop me nor would i want to anyway all right what can people do if they want their own questions answered go to soft skills.audio and click the ask a question button where you can fill out our form to ask us a question thank you so much to everyone who does that we love your questions they just keep flowing in like like kindness flowing into a troubled heart which is my heart and it feels less troubled is kindness a fluid i don't know okay i should probably have that project is it ooblick is that the stuff where you squish it and it becomes harder

yeah yeah thank you thank you so much for listening we will catch you next week