The Data Stack Show - The PRQL: What are the Different Flavors of Orchestration?

Episode Date: May 27, 2022

Eric and Kostas preview their upcoming conversation with Viren Baraiya of Orkes.

exciting news we have another data stack show live and we're going to talk about streaming there are huge questions is streaming a superset of batch what is real time and what does that mean well we've collected some of the top minds in the industry we have people from stripe materialize open source projects that enable streaming. And we're going to ask them a bunch of questions and give you the chance to ask your questions live as well. It's June 1st. Go to datastackshow.com slash live and register. Again, that's June 1st, datastackshow.com slash live and register. We'll see you there. We just recorded a fascinating show with Viren,

who started a company called Orcus. The company actually is based on open source technology that he originally built at Netflix and then came back to commercialize later in his career. Absolutely a fascinating story, Kostas. I think he sort of hit on a lot of major milestones that people sort of dream of, right? Like being in Netflix, developing open source technology, working at Google, developing some interesting new products there, and then eventually commercializing an open source project, which is pretty amazing. It's in the orchestration space, which is interesting. And Kostas, I actually love the category orchestration because it just makes me think about like an orchestra conductor.

And so it's kind of like a calming, like visual for me. I don't know if it's the same for you. But Kostas, what I want to know is in terms of orchestration, like how new is the space? And why do you think that someone would develop an orchestration tool if there's already, you know, a bunch of stuff out there like airflow, etc. Especially that a lot of our audience probably uses. Yeah, I think what is important to understand, like what is actually something that we both learned a lot about like during our conversation today is that orchestration is not just one thing, like there are many

different flavors of orchestration and different people work with different tools because just like the needs are different. You can start from the, let's say, very high level type of business orchestration, which an example of a lightweight orchestrator for business processes is something like Zapier, right? Which is not used by developers at all, right? Like it can be used by people that are like marketing, in sales, in like just have like a small commerce website or something like that, right? And you can go like very, very low level at the level of like, let's say,

you have to provision, let's say, virtual machines, right? And you need also like an orchestrator there that takes care of like provisioning and making sure that like all the resources are there and like all that stuff. So, and of course you have all the different layers in between. One of the interesting parts of the conversation that we had also has to do with like, what's the difference between the orchestrators that we have in data engineering and microservices? And I think that's something that's like, I think very, very useful for any data engineer out there who just need to figure out how to orchestrate their pipelines and what vendors are available for that or tools and what not to use and why.

So I think I'm much more well-informed and much more educated around orchestration today. And I would encourage everyone to listen to this podcast like you should. Totally. I think, and I'll leave this as a tea. Viren talked a lot about A-B testing full flows, which was fascinating and how they built this technology in Netflix to actually A-B test flows at scale, which was fascinating. So tune in for the next episode to hear about that and more

on the subject of orchestration. And we'll catch you on the next one.