Grey Beards on Systems - GreyBeards talk EMCWorld2015 news with Chad Sakac, Pres. EMC System Eng.

Episode Date: May 18, 2015

In this podcast we discuss some of the more interesting storage announcements to come out of EMCWorld2015 last week with Chad Sakac, (@sakacc on twitter and VirtualGeek blogger) President, EMC Global Systems Engineering. Chad’s was up on the “big stage” at EMCWorld helping to introduce much of the news we found thought provoking. Chad said he was growing … Continue reading "GreyBeards talk EMCWorld2015 news with Chad Sakac, Pres. EMC System Eng."

Hey everybody, Ray Lucchese here and Howard Marks here. Welcome to the next episode of Greybeards on Storage, a monthly podcast show where we get Greybeards storage and system bloggers to talk with storage and system vendors to discuss upcoming products, technologies, and trends affecting the data center today. Welcome to the 20th episode of Graybeards on Storage, which was recorded on May 14, 2015. We have with us here today Chad Sackack, President of Global Systems Engineering for EMC. EMC World 2015 was held in Vegas last week, and there were a number of big announcements. Why don't you tell us about some of the announcements at the show, Chad?

Yeah, you bet, Ray. And to you and to Howard and to all the listeners, thanks for having me on. It's great to be on. You know, 20 episodes is a strong legacy, guys, so be proud. So let's see. What did you guys think stood out at EMC World? It's always interesting to hear like what what's the stuff that echoes in the in the community in the blogosphere the strongest okay

i thought the dssd thing i led with that on my blog post on emc world day two three and stuff like that it's uh it's an amazing machine i'd like to i'd like to get to see the guts of that and i saw that your your post on virtual geek uh had a lot of interesting stuff on it as well. It was funny. After I posted that, I got a little bit of the, Chad, you're going awfully close to the line. You zoom in on the pictures. You can actually see a lot. Yeah, yeah, yeah.

You didn't open the kimono, but you did untie the belt. Yeah. Well, Thank God. Part of my job that I think is the most, one of the parts that I think is the most fun is I get to be part of the M&A process, right? So in the M&A process, I'm the representative of a field at large, right? So, you know, if you just kind of sum up the opinions of all of the SEs, I'm there to try to represent them. And in essence, they represent the customer, right? So you're there to answer the question, is this a science project or will people actually use it? Will people actually use it?

Does it solve a problem that we see at the customers? Is it technically viable? And then interestingly, is it sellable, right? actually use it? Does it solve a problem that we see at the customers? Is it technically viable? And then interestingly, is it sellable? Sometimes you can have a yes, yes, but a no on that last one. Yeah, we can all think of products that fall in that category. Exactly, right? So you can think of a boatload. So the thing that also happens is sometimes I'm part of the strategic planning process. And DSSD was a company that we actually invested in very early. And then we were doing a bunch of strategic planning around what will be the other ways that NAND disrupts the business. And what are post-NAND disrupts the business, right? And what are post-NAND disruptions and stuff like that.

And one of the outputs at the end of that strategic planning process was Xtreme.io is extremely awesome and great, and we're doing lots of good stuff about flashing the hybrids, but we're going to miss a window where we could have very different densities and performance envelopes if we were to discard current NAND packaging. That was the strategic input. And at the end of it, I'm like, look, we're probably going to have to do something organic, meaning development, or inorganic M&A around this. And the EMC Ventures team said, hey, Chad, you should take a look at this company that

we put an investment into. And that was my first introduction to DSSD two years ago. And the moment that I got a chance to see it, I went, is this solving a real customer problem? Yes, it solves the problem of how do you get NAND-like performance, sorry, DRAM-like performance, but with capacities and sharing models that are associated with, you know, storage, quote-unquote, arrays, and shared storage in some fashion,

and how do you do it with capacities that are NAND capacities in economics, but again, like, closer to DRAM kind of latencies than people associate with SSDs, right? Yeah, yeah, yeah. And, you know, it comes through. And architectural, you know, and application architecture. That's the thing.

That's the first part. Does it solve a customer problem? The customer problem is that for most modern hyper-transactional systems, they don't sit on a relational database. They sit on some newfangled data fabric that's usually an open source chunk of code. And its persistence layer is almost always a key value store or some form of a very high performance HDFS layer, right? one of the focus areas of the product. It was more of the I'll call it the modern no SQL kinds of solutions or HDFS versus the standard application environments that we know and love today that are out there running the

world. So put it this way, this thing can, so DSSD can present a block target. So there's a piece of code that you put on the client, and you can think of it as like a block emulation device, right, that translates the object interfaces, because natively it does present the blobs of NAND as an object. But, of course, it could actually, if you think about it, HDFS is an object.

You have an HDFS object. A key value store is a key location and value object, right? So all of those modern things don't use a POSIX file system or a SCSI command set as their presentation model. They may have it way down in the stack. But what these guys realized is, hey, why have, you know, NAND devices that then get presented as a LUN that then, you know, communicates over a SAS or a SCSI interface and then communicates into a POSIX file system, and then you layer an object layer on top of it, they're like, look, we could actually just map the NAND pools into objects. Now, there is a block driver, Ray, so you could in theory, and some of the early beta customers that are using it now are just using it as a brute force,

super fast storage device behind an Oracle database, right? Right, right, right. But that's not its sweet spot. Like, its sweet spot is definitely these new data fabric elements. And I guess for the listeners to the blog, this thing is the definition of face-meltingly awesome. So, you know, we hadn't publicly stated the numbers yet, but we did on stage in EMC World. In a 5U chassis, it will start at roughly around 150 terabytes of NAND.

So if you think about the density, 5U, 150 terabytes. Three terabytes per U, and that's NAND, not reduced capacity, right? Correct. 150 is 30 terabytes per U. Yeah, 30 ter not reduced capacity, right? Correct. 150 is 30 terabytes per U. Yeah, 30 terabytes per U, right? Yeah. And then just to be clear, guys, we also said very quickly we're going to follow it up with one that's almost 300 terabytes. Double that, yeah, yeah, yeah.

Right? Right. Now, the capacity density, so in other words, capacity density, it's orders of magnitude greater than anything out there, right? Now, by the way, there's others that have made big claims. Sky Air is the only other one that is in the same sort of capacity density spaces. But if you take a look at anyone who's even close to, you shipping something for reals this thing is in a different universe I'm just thinking that SanDisk's Infiniflash might be close but that's a stupid

device they've got 512 terabytes in 3U but that's basically a JBOT it's got no smarts both of them though what's interesting about both of those examples is they kind of highlight you gotta you gotta discard the the ssd form factor yeah right the ssd form factor was is clearly a temporary accommodation you know it it was the how we how

can we use this nan stuff now not how can we best use this NAND stuff? Well, so like if you take a look at the DSSD, what they call a flash module, right? Right. It's got 512 die. So that, you know, if you take a look at the NAND dies, they're stacked four high. And you've got a set of, you know, a grand total of 512 dyes, which means that there's a huge amount of parallelism relative to, you know, something that you'd see on an SSD or even a, you know, a modern. No, most SSD controllers only have eight back-end channels.

Exactly. So that then translates into a huge amount of additional parallelism per flash module, right? Okay. You're not going to tell me how many back-end channels there are on the controller on that module. So the controller is totally custom. Yeah, no, that I figured. I can tell you that the controller as it was designed can drive far more than the 512 that are already there in use. So, you know, in other words, there's even more headroom there.

But one thing that was interesting was, you know, the design center wasn't just capacity density. It was around how do we actually drive all of those dyes? So, in other words, how do we build parallel of those dyes? So in other words, how do we build parallelism into the controller? How do we build parallelism into the PCIe fabric? And that was the other thing that Bill Moore showed on stage. He pulled out the PCIe fabric, and you can see that it's a very dense PCIe mesh. Yeah, there's lots of interesting things going on in PCIe switching lately. I was talking to the guys at PLX the other day,

and thinking about a few years ago, Next.io and Zego and those guys were trying to do... Extension, yeah. Well, they were trying to do PCIe switching for IO for peripheral sharing, but this kind of thing is way more interesting. I think actually the reality is that we're going to enter over the next few years that there will be a much more prevalent and industry standard external PCIe fabric switch market that will… Yeah, kind of rack scale where the whole A rack is a PCIe fabric switch market that will... Yeah, kind of rack scale, where the whole A rack is a PCIe complex. Right. And frankly, one of the things that's interesting is in acquiring DSSD, we're acquiring people that have built some of the largest, densest PCIe fabrics now, right? Right. So when you acquire something, you're acquiring more than anything.

That puts you in an interesting position in terms of something like VX Rack. So you could imagine how we could apply that in a ton of cases, right? Yeah, yeah, yeah, yeah. Now, but back to the DSSD thing. So, you know, think of 150 to 300 terabytes in 5U. Think about somewhere between, you know, 10, 20, and then in some extreme cases, maybe even 50 million IOPS in that same capacity, that same space. And think of latencies that while still an order of magnitude, in some cases two orders of magnitude slower than DRAM, it's an order of magnitude or two orders of magnitude faster than an all-flash array.

So you're talking about tens of microseconds? Tens of microseconds, in some cases even less. Yep. So this is like memory channel, this is like the Diablo stuff, and speed-wise, I mean, Jesus Christ, microseconds, tens of microseconds. You can't do an IOP in tens of microseconds. So that's... Sure you can, Ray. It's the 21st century. Yeah, yeah, sure.

Now, what we were talking about, like, okay, what are we going to say and what are we going to do? There's a part of this which is we're not trying to play a psychological game of bait and switch or a negative reverse close. Like, oh, I'm not sure if you can handle enough cars. This car is too much for you. But the reality of it is that this year, 2015, we've already got six or seven customers using them, so they're real. The beta process is actually going to be starting what we would call the directed availability. We'll start actually now, and it'll run throughout the rest of the year. It still needs a lot of hardening, right? Oh, yeah. So we're kind of like, hey, get excited because it is pretty awesome.

But don't put your company on this quite yet. Yeah, but it's going to take a little more time for us to continue to harden it, build it out, validate the use cases, but it's very interesting. Yeah, I mean, the whole discussion of the world has been going software only for quite, oh gosh, I'd say five or six years, and now here we have a hardware solution that's just going to blow the doors off of everything. That's amazing.

So my observation, and I've been trying to get people to internalize this, and I've been failing, so maybe you guys can answer that. Right? The reality of it is that software innovation and hardware innovation are actually cyclical. And what occurs is that, you know, hardware innovation is like a sine wave, and then behind it, phase shifted by 90 degrees, you've got a cosine wave

that is the wave of software innovation, right? Right. And for any given industry or for any given, you know, point in time, you might have a relatively low rate of hardware innovation and a relatively high rate of software innovation. And then at other points in time, you have the converse. And historically, that cycle has gone through a number of times.

Right, but you only get really cool software in periods when the hardware is more than the current software needs. Because then the developers go, wait, I have all of this extra resource to deal with. And if Xeons weren't fast enough that we were running servers at 5% utilization, we never would have thought of needing VMware. Right. The interesting observation is that foundational innovations then create more space for more innovation. Right, right, right.

You can't see what's going on until you get there, to some extent. You can't see past the horizon, really. You don't know what you're going to need next week until it's next week. The thing that is going to be really fascinating for me to watch just as a pure, you know, industry observer, as well as participant, you know, we all, we're all kind of like in this and watching it at the same time. Right. Well, we spend more time watching, but some of us do at least. It's really fascinating to me. and I'll make a statement that breaks my Chad pre-sales manifesto of never kind of go negative. It's amazing to me that Oracle remains as successful as they are. And the reason that they do is because their databases and their enterprise stacks are very good and very compelling, but also very hard to displace.

That all said, if someone wanted to create a high-performance transactional system, they wouldn't put it on a relational database anymore. Right. Number one. And that's the cash flow. Doesn't that depend on the requirement for consistency? Nope. And that's the cash flow. Doesn't that depend on the requirement for consistency? No, because frankly, there's ACID-compliant and SQL-oriented database models that smoke relational databases that are modern open source variations. But they don't have the whole ecosystem of this application only works with blank, with Oracle. Right.

I mean, the truth is people don't buy Oracle the database because they want Oracle the database. They buy Oracle the database because they want the ERP system that runs on top of it or the industry-specific vertical application. The ecosystem. The ecosystem matters. Yes. That's what I'm saying. That's the basis for its immobility. But at the same time, while that's all true, if you can do something orders of magnitude better, it starts to create a ton of pressure. You get the database that's an order of magnitude better, and then you get some set of young'uns who start building a couple of those vertical applications on top of it.

And then the people who have existing vertical applications go, hmm, we better support that too, or the young'uns are going to put us out of business. And increasingly as people actually take those enterprise applications and are using them in SaaS models, if you're the SaaS vendor and you actually control your own stack… Right. Then you definitely don't use Oracle because you don't want the expense. Yeah. And by the way, I'm not just throwing the, you know, the Sling or the Pebble. Oracle is an example, not specifically Oracle. Yeah.

And likewise, interestingly, you know, the observation that we were making from a storage ecosystem that was another driver for the DSSD buy is, hey, there's a ton of people who buy high-end storage, very high-performance block devices behind relational databases. So what are we going to do if, you know... Yeah, if they stop using the relational databases, they'll stop needing the block storage. We better have something else to sell them. And hence DSSD. Okay. Go ahead. Have you guys thought about a memory interface to DSSD?

You know, kind of RDMA-ish? It actually does support an RDMA, direct RDMA mapping. Yeah, so then you start, then that opens up the door for things like two-tiered in-memory databases. Really, the world changes when we have volatile and non-volatile memory at two different speeds, but they're just memory, not block devices.

So, you know, one of the early beta sites is SAP. SAP is... I'm shocked. ...is exploring it for HANA use. Today, they're just using it using the block interface because HANAana just you know expects i've got a pool of dram or and then i have a persistence thing that i that i log to right um and they're using it as a super fast vehicle for log uh you know writing logs and

log restart for hana cluster uh you know node restart but um but you know they're as we're playing with it it's also an ability for us to look and say, hey, should future versions of HANA recognize that their memory hierarchy includes multiple tiers that are, in effect, should be treated as memory? Right. And then you can start building the kind of things we do in hybrid disk arrays into that database and move pages or tables or records between tiers. Right on. Yep.

Okay, I think we beat the DSSD thing to death. Let's move on. What would you like to talk about next, Extreme I.O. or software-defined stuff? Guys, it is your show. I'm merely your guest. Before we move on to those, I got one question about VXRack. Yeah.

Is it running ScaleIO as a hyper-converged layer, or is it running ScaleIO on dedicated nodes? So the thing I think... Or do I get to choose? The answer is you actually get to choose. Okay, great.

Because I have a problem with hyperconvergence at scale with the VMware because CPU cycles on a vSphere server cost about twice what CPU cycles on a bare metal server do because I've got to buy vSphere and

Windows Data Center server and a backup application and they're all licensed by the socket. If I've got enough buy vSphere and Windows Data Center Server and a backup application, and they're all licensed by the socket. If I've got enough servers, I'd rather run the storage layer bare metal. Howard, I'm not sure whether it was you. I saw someone did a good blog post that did a write-up and a teardown of EvoRail as a category. That was Howard, yes. I saw that. That was actually quite good, in my opinion.

Thank you. So the – Make sure and tell Chuck. Yeah, I'll talk to Chuck. So look, so first things first. VX Rack basically takes commodity industry standard servers. There will be 15 variations of them that are allowed inside the configuration. That includes like which processor and which memory? Correct. No, no, no,

no. It's 15 variations, not including CPU type and memory type. So 15 different kind of chassis. Yeah. It's not full-blown 15 different chassis, but if you think about n number of chassis with some amount of variation in chassis family. Yeah, that's a lot. Correct. That's a lot of qual. So there's a lot of variation of the nodes, right? That probably means that includes like 40 drive nodes. So it includes very dense storage nodes, very CPU dense nodes, right?

So first things you've got to imagine is you've got to imagine it is, it is a, a, a engineered system versus an appliance, right? In other words, you can see why we're making that distinction now now because it's got a high degree of flexibility in the design center. Right, and the whole idea about an appliance is one size fits all. Right. Now, the second thing is that the node types can have multiple personas. So the persona can either be a hyper-converged persona or a storage-only persona.

Oh, okay. No compute-only persona? And persona or a storage-only persona. Oh, okay. No compute-only persona? And there is a compute-only persona, so all three of those variations. And then in the compute and hyper-converged personas, you have a choice of personality that is a vSphere personality, a KVM personality out of the gate, or just a straight-up bare-metal persona. Right. So, you know, bare-metal persona. Right. So, you know, the economics... Now, the storage model in VXRack,

and again, you know, these ideas when you do the tech preview and you have 15 minutes on stage to put it out there, all of the details don't come out quite right. But basically, VXx rack should be thought of as a brand you know so it will have a family in other words there will be one family that has this broad hardware variation and broad personality variation including storage only variations and that would be vx rack let's call it the vx rack 1000 right right um and that has be VX Rack, let's call it the VX Rack 1000, right? Right.

And that has to use Scale.io as the SDS persona because obviously you have lots of different variability of configuration, OS type, et cetera, et cetera, et cetera. You've got to draw the line somewhere. Right. Now, there will be a VX Rack, call it like a VX Rack 900, that is an Evo Rack-only persona that will use the Evo Rack software stack from VMware. Right. And that will have different bounds of what is allowed and what is not allowed based on what the Evo Rack manager software allows.

Yeah, yeah, yeah. But we can assume that the storage layer is vSAN. In that case, the storage layer will be vSAN. Yeah, yeah, yeah. But we can assume that the storage layer is vSAN. In that case, the storage layer will be vSAN. Yeah, yeah, yeah. Now, just the thing that I think is very interesting is, first things first, response from customers has been outstanding to the VXRack announcement. While I can't tell you exactly when it's going to GA, I can tell you that when we use the word tech preview, it means that it's not going to GA in quarter. Because one of the rules is if you announce something, it has to GA in quarter.

And big companies have to have rules. And there have to be rules. It's a revenue recognition thing. Right. So by definition, it means it's not going to be before july 1st right right um but i gotta tell you it's surprisingly close so customers are really so july 2nd i didn't say that uh so chad was on record not saying that so customer reaction has been great um one thing

that will be very interesting for us is that i would wager now this is my personal opinion that what we'll discover post ga is that there's actually a need in the market for less variation than we're actually planning for right and and you know the first the first for the first little while the the go-to-market for vx rack looks you know very engineered system-like it'll be interesting to see like if we find out hey 90 percent of the customers you know 40 cluster around these requirements and 40% cluster around these requirements, then you can build an evolution of our vSpecs Blue appliance go-to-market that, you know, has more variability than it does today, but, you know, clusters around those use cases. The other thing that I thought was kind of interesting was the discussion that it was

like a VCE customer experience in a VX R rack environment. Can you kind of talk to that, Chad? Totally. So mommy and daddy have been fighting, and they're living in different houses. No, no, no, no. It's a one-throat-to-choke. It's a management environment. But the X is, you know, we're not necessarily using Cisco. That's correct. So guys, here's the short version. So EMC as a company thinks that this converged infrastructure

thing is not a fad. Yeah, I would say so. Right? So if you think it's not a fad, in fact, if you think that enterprise, mid-market, SMB, and SP customers increasingly are going to want converged infrastructure in all of its forms, meaning don't think about how it's built. Think about I want it all as one thing so I can just accelerate whatever I need to do. It's the acquisition process. It's not the technology. It's not the technology. It's the support, cycle, warranty, acquisition, blah, blah, blah, all that jazz, right? So you look at that and go, okay, great.

Well, we have one thing that's going quite well, which is if you have a budget that is more than a million dollars and you have more than 1,000 VMs and you need traditional data services, you know, vBlock is going like gangbusters. Yay. But we couldn't, you know, within the constructs of how VCE was founded and how it was set up as a JV, you know, we had to constantly dance around things like, well, what if a customer wants NSX? Well, you can add NSX after the fact, but it's not supported, you know, by VCE, right? So the customers, you know, drove us down the path where like, okay, clearly we're going to need to change something. Ergo, VX block is the X means that there's something more than just the original VMware, Cisco, and EMC foundation. But the other thing that we kind of realized is, hey, look, these rack scale and appliance formulations that use a hyper-converged model have two advantages over block-like models, right? Number one, they can start smaller, right? Number two, they can be designed for extreme simplicity. Not always, but, you know, simplicity and flexibility are kind of two different poles,

right? So if you trade off simplicity, you. Right, so if you trade off simplicity, you get more flexibility. If you trade off flexibility, you get more simplicity. But regardless, you can design for extreme simplicity. And the other observation is that they can have a different operational model. So their OPEX envelope can be different because they're composite out of different ingredients.

So you go down that path and say, okay, great. Well, we're going to need to have some variation where it's not necessarily going to be a B-series Cisco UCS server. And ergo, you bring VCE, which is correctly thought of not as a V-block company. It's more correctly thought of as 2 000 people with five years of experience running a converge infrastructure business with two billion dollars worth of run rate revenue right and and so now you could say okay now we can loosen some of the shackles and you can create different formulations of converge infrastructure for different markets, different workloads, different use cases. Right. Right. So, so what's the VC experience thing that the Delta, you know, that I think is an interesting one is at large scale, there's a really basic question that matters. Like, should you, uh, be designing this system with this network spine and leaf fabric design in mind or this spine and leaf network in mind?

We've got scale IO deployments at customers where they're just home brewing it, so to speak, 50, 60, hundreds of nodes, either hyper-converged or not hyper-converged, right, is the network starts to really matter. Yeah. And so with an appliance, it's like it shows up in a box with, you know, a screwdriver. You rack and stack it yourself. When someone says, which should I use? Just go, ah, any 10 gigabit Ethernet switch will do.

Unless I have 50 of them. And then it's like, well, we never contemplated that in the appliance design, right? Right. A VCE experience starts with someone who sits down with you. You go through an engineering phase. In other words, what are you trying to do? What is your target scale? How fast do you think you're going to grow and scale what are the different personalities that

you think you're going to need okay great now i'm going to leave and then i'm going to come back with a we think this is how we should design it there's some back and forth back and forth okay great then it's like that's exactly what we need off it goes to the factory that where vc engineering and manufacturing right chat yep i got a correction to your bespoke suit metaphor which is it's not the bespoke suit it's the custom shop it's the made to measure where the the shoulders on the shirt are pre are already, but they take in the body to match you. It's halfway in between. Yeah.

So that's actually a very good point. You don't have complete variability, like you might have in a bespoke suit example, which is what I put on the blog post. But you're right. It's made to fit, right? Yeah. Which is different than, you know, basic tailoring of getting your... Well, it's different than having it complete and changing it. We made it as far as we can before we have to start customizing it to you. Right. But once it arrives, the first, for example, the process of using the VX Rack Manager, you can't just take a single node and put it in from scratch. You can actually add a node to an existing VX Rack, but the VX Rack Manager starts

by going, okay, great, a VX Rack has arrived from VCE. And that is going to be inside a rack. It's going to include the top of rack switches, and it'll include however many nodes you initially configured. Right. Okay. So the VCE experience is a couple things. It's the single vehicle for procurement, single vehicle for warranty, single vehicle for support. It has this idea of we're going to work with you to engineer and design the system, and then it's going to arrive

as a unit. Right, and a little bit of wrapper software to help manage it as a unit. Go ahead, Ray. Yeah, I was thinking we should probably get on to something else at this point. That's because I was segueing. So while we're talking about adding nodes, I was very glad to see that you announced that we can do online upgrades on Xtreme.io, which was a point of contention. It certainly was. It was just annoying that you had this scale-out architecture,

but you couldn't do that. No question, right? And look, to be very clear, customers saw value out of scale-out even pre the ability to dynamically scale-out. In other words, there are lots of customers now that actively are using two, three, and four-node ExtremeIO clusters.

But you support three-bit clusters? Sorry, we support one, two, and four-node, four-expert clusters. Okay, fine. Because not supporting odd numbers is also a minor point of contention. Yeah, The observation, fundamentally, guys, being as

transparent as we all always should be, is when we acquired Extreme I.O., which was now almost three years ago now, we knew that basically their architecture had gaps. So when you do the technical due diligence,

you look at it and go, huh, you know, like no snapshots, no remote replication, no ability to dynamically add nodes, no ability to withstand a dual disk failure within a single X brick, right? Right. And you look at it and go, hmm, you know, has the engineering team thought of this? And the answer was, even well before three years ago, the answer was yes. In other words,

they had already built in the metadata mapping to be able to add snapshots and blah, blah, blah, blah, blah. But the features... They knew how to get there, but they weren't there yet. They weren't there yet, right? And, you know, the truth is,

if I was a customer a year ago and you, EMC, said we'll have dynamic expansion in two years, then I would just buy the system to scale to six months after when you're promising me that that's ready. And to tell you the truth, we did that with a ton of customers too, where we basically said, look, what we'll do is we'll do it on an economic utility. In other words, we will sell you the two bricks or the four bricks now, and we'll charge you for one brick or two brick or whatever.

As many more you go. The joys of having deep pockets, you can afford to do that. Right. Now, I'm very happy to say that. Right. Now, now, you know, I'm happy, I'm very happy to say that the, you know, the, one of the most important things, again, just like with DSSD, Extreme.io and DSSD were acquisitions that we make much earlier in a startup cycle than we normally do. And it's because they're that good and the engineering team is that strong. And, uh, frankly, you can, you can see that it's going to happen right right um and and usually when all of those things are true there are other people who are also kind of

sniffing around right um with extreme io we looked at and said look we can see the path it's already built into the code the fact that we now added the dual disk failure within a existing x brick and somehow magically it didn't cost any capacity meant that all of the mapping code was already there for dual in effect parity protection. Even the parity generation was there.

That's right. It just was not fully implemented. So I got to tell you my one day I'm going to get fired from EMC for being Right, right, it's very compelling and customers love it. Right. It's like the, okay, you now have filled in sufficiently all of the things that are necessary to cover a huge swath of the market. So, big snapshots, you know, brain native. You've gone from minimum viable product to minimum competitive product.

Right. And, frankly, I would argue that we cover the spectrum and gamut of use cases in a very, very compelling way. I'm like, okay, now take your foot off the gas of feature ad and just stay laser focused on core product quality, right? Because the business is now ramping very, very quickly. And fit and finish and all of that. Exactly, right? The only place where I think we need to keep our pushing hard is that right now we're finding it's very compelling and very competitive. And where we show up and where we disrupt ourselves, we do very, very well with our customers and our partners versus protecting the install base and doing stupid stuff like that. But what happens? partners versus protecting the install base and doing stupid stuff like that. I think next year you're going to start getting customers wanting

to do heterogeneous clusters. It's a tough game with this architecture. The first round of Extreme IO customers are going to get to the point where they want to expand that cluster and they want the new higher density nodes and they want two clusters. So I think that we're going to need eventually, no doubt, Howard, mixed clusters. I would argue one other thing is that we've been hyper conservative around drive and NAND life. Yes. Yeah, I think we as an industry

have been so worried about write endurance that we've just been throwing extra flash at it and wasting money. So, you know, like the Xtreme.io team makes a big, big deal over the fact that, you know, we've got all of these things to mitigate write amplification, right, which we do. I would argue we probably have, if not the best, certainly some of the best write mitigation

techniques that you see in the whole all-flash array market, and that we've maniacally stuck with EMLC. And again, you know, I think probably most listeners know that you know nan types are kind of rated based on right per day um cycles right right so so we use emlc which is rated for like 10 to 12 right per day um you know kind of usage versus cmlc um which is you know more targeted towards, call it like three to five write per day kind of cycles, right? Right. But if you look at the ET phone home analytics you get from the box boxes,

your customers are doing half a write per day. Well, so actually I posted the data from all of the phone home. Oh, nice. Yeah, yeah. Okay, so how close was my guess? So it's ridiculous. from all of the phone home on the blog post. Okay, so how close was my guess? So it's ridiculous. I mean, after petabytes and petabytes of writes against Extreme IO clusters,

basically we're like, hey, there's not a single customer who has less than 20 years of use left on their sales. Right? And we're saying, like, therefore, we're very proud of ourselves. That's one way to look at it. The other way to look at it is you sold those guys way too much flash. What we could have done is we could actually put in, you know, CMLC for a lower economic price, you know, and they'd still have six years of life left.

Yeah, and over-provision a little bit less and all of those things. Right, but, you know, I would argue that that's probably... I think we as an industry have to reach the point where if the top 5% usage customers have to replace an SSD occasionally because it wears out that that's okay. That's part of the support contract anyway, so it's not like it's... We should view them as partially consumables, as opposed to today where you guys are selling something that will definitely last five years, it'll never wear out.

Now, here's the thing that I think would be fascinating to me, right? So literally on the blog post on Virtual Geek, we've got all of the data from thousands and thousands of customers now. And I can't think of anybody else who has published or shared their population of customers and this where stat where stat and the reason i'm just saying that i have gotten that information from vaughn so so um and it it showed up it you know it was a conversation we were having that actually ended up with a blog post by somebody else but you know they i think have replaced a dozen ss. Now, so here's the question, Howard, right? So I would argue that Pure, like Extreme I.O., it meets my definition, my own Chad Sackatch personal definition of an all-flash array, because its I.O. stack is designed assuming NAND is the only persistence layer. And there's a ton of stuff in there that is used,

both data reduction techniques, alternatives to RAID models, and all sorts of stuff to mitigate writes as much as possible. And again, I'm not a big fan of going like, we use EMLC, you use CMLC, it sucks. Oh, no, that drives me crazy because the difference is not, you know, this is stuff that the Flash Foundry made for thumb drives and this is stuff that the Flash Foundry made for Enterprise SSDs. The details are very deep and, you know, having a good controller makes up for having a flash. Now, that all said, I would love, and I'm going to actually find out what it is for our traditional hybrids that have got high flash drive configurations. Right. It would be interesting to compare Xtreme IO to VNXF. Right. And the reason I'm just pointing that out, right, is like I think we could make the pivot from EMLC to CMLC relatively easily for Xtreme IO.

I think it would be much harder for our traditional storage stacks. Without the deduplication, without the data reduction stuff that's intrinsic and in line to Xtreme IO. You're going to get more writes. And to the storage stacks that are designed as all flash arrays. And by the way, I'll put my own stuff in there, but you can think of many other storage stack vendors who have got a storage stack that was not designed. Oh, yeah.

Half the systems are more from any vendor who didn't pop up to create a Flash-based system is going to have that problem. And I would just say I'm not trying to be a fear monger, but whereas I feel like we've over-engineered it a little bit in all Flash array land, I think it's going to be interesting to see over the next few years what happens with those things. Because you're right, it's totally fine if your array vendor calls you up and says, you know what, we're seeing that you're only down to about 20% drive lifetime available on this particular device. So we FedExed you a new one. We're going to FedEx you a new one. To me, if I'm a customer, that's totally cool, right? To me, too.

And if we do that, and even if the storage vendor scales it so that only the top 10% of customers ever get that phone call, we're going to save 15% or 20% on the amount and price of the flash that we're using, and the customers are all going to be better off or 20% on the amount and price of the flash that we're using, and the customers are all going to be better off. I'm totally with you. Now, let me be very explicit. I'll speak here as EMC, the largest single storage vendor out there.

We currently don't have a system and process set up to do that, right? Right. And so all the back-end support organization changes necessary to make this so routine. Well, and it's a software surrounding the solution, to get it out there and analyze the data and stuff like that. So my point there is not so much about the, quote-unquote, real all-flash arrays that exist out there, but the arrays that are sold as an all-flash array. Right.

Right? Right. Legacy all-flash array. And as they aggressively drive the use of TLC and CMLC, I really hope that they're putting some thinking into the scenario to be able to go and do exactly that, right? Right. Okay, guys, we're running a little late here.

I thought we should get into the software-defined stuff that came out. I mean, there was a lot of software-defined announcements in the EMC world. Oh, a lot of software-defined, i mean emc and open source and yeah yeah so you want to start there go together with viper controller and move on so so like look i'll try and see if i can do this one in a short little so a long time ago meaning about a year ago you know amongst the exec staff we went and said, the reality of what we're seeing is that for these new workloads, workloads that are designed around what we would call Platform 3, IDC would call Platform 3, but the right way to characterize it is workloads that don't depend on

infrastructure resilience, right? What we were finding was even if we would have technologically clearly better intellectual property, customers would elect to consume intellectual property that was free and frictionless. And what I mean by free and frictionless is that there's no barrier for you to be able to get it, use it, try it, um, those sorts of things. Now I want to make, I want to make a distinction here, free and frictionless. Think of that as like the biggest, you know,

bubble. Then there's a subset inside free and frictionless that is free, frictionalist, and open source. Okay. Right? And by free here, you don't mean without license costs, right? You mean freely available. I'm saying available to be able to access the intellectual property in some way, shape, or form without talking to a sales rep. Trilobal. Trilobal. Trilobal.

Yeah. talking to a sales rep trial trial yeah and and it what's interesting is almost none of the customers that we talked to we talked to truckloads actually took the free vehicle right and then actually use that for their production deployment so in other words well just like they don't use centos in production they use red hat bingo right people actually, you know, use and play with Gluster. And then when they pivot that over and try and use it in a production environment, invariably they say, you know what, I'm going to get it from Red Hat and I'm going to pay Red Hat for support. If I'm going to buy, if I've been using, you know, Ceph for learning and development, when it comes time to make that part of my OpenStack deployment and I'm using that for Swift compliant storage, I'm going to buy Ceph Enterprise. And every one of those we discovered. Because when something goes wrong, I still want to be able to pick up the phone.

Bingo, right? one of those because because i because i when something goes wrong i still want to be able to pick up the phone bingo right um but what we realized was you know our entire go-to-market let's be let's be honest like if you want to actually use something that emc has right i gotta see the guy in the armani suit with the gucci guy's going to show up. You know, he may show up in a hoodie, might show up in a Armani suit. But there's going to be a sales dude who's going to talk to you. EMC, you know, owes its success to that army of sales dudes. And this is an interesting transition for an organization that's so sales-centric as EMC. There's no question, right? What we were discovering is we'd go to customers and we'd say, you're using Ceph. Are you happy? And they'd say, well, it's not working particularly great. Performance for the transactional stuff is not that hot, and it's working okay on the object side. But we're

confused. You're also an EMC customer, and when you bought that VMAX and ExtremeIO and Isilon cluster, we included in your ELA ScaleIO and the Viper controller and the ECS software object stack. And they go, yep. And we're like, so are they not good? They go, no, they're really good, work great. So why don't you use them and they're like well because the team that runs and operates that environment basically biases towards free and frictionless and then depending on what you're talking about also may require it to be an open source project yes it's it's layer 10 of the osi model, metaphysics. People have these philosophical opinions that drive their buying.

So what we discovered is that where ecosystem is critical, open source is critical. Right. Where lock-in is critical, in other words, things that are basically control points in stacks, open source is critical. Because people basically are very, very worried about, you know, what does that mean if I use it? And then you later decide to, you know, turn into a Faustian bargain with the devil, right? Right. And hold overhead. So what we basically said is, okay, as a rule, everything in our emerging technology division, every piece of software has to be available freely and frictionlessly.

Now, what does that mean? It means that you can use it however you want to use it with no support except for the community, you know, have at it. And we've done that now with ScaleIO. We've done it with the Viper controller. You can expect to see it with the ECS object and HDFS stack. And you will see it also with the Isilon software stack. That could get interesting. Is that going to remove the dependency of Isilon on InfiniBand or

is it really going to be, I can build an Isilon cluster? That's another topic for another day and another point. Okay, yeah, I understand. We can only go so far into future so you have a career to worry about. Now the observation is, okay, so step one you know decision made everything has to be free and frictionless what will be their economic models well we discovered from most customers that they actually didn't have any beef with our license plus support model so their problem wasn't you know with with license versus and support and people assume that

as we said this incorrectly that everything was going to go down the red hat model where you basically pay for support right so scale io still has got it if you want it you have to license it and you pay for support maintenance and support and that includes patches and updates and blah blah blah blah blah right if you want the free one it's community only support and it's not going to come with patches and updates and et cetera, et cetera, et cetera. It's going to be whatever shows up to download and consume. So the economic model actually hasn't shifted because the issue actually wasn't with the economic model. The issue was that the only way I can actually ultimately get my hands on something is I need to listen to the sales team and I need to

take what they're saying on faith. It's the frictionless thing. Well, yeah. I mean, having spent a lot of my career as a consultant to the mid market, some of the same things can be said. I can't find out how much the price is on this EMC product without talking to the sales guy, but I only got two days to fill out this rec. So I'm going to go to the Dell site because they give me prices. So it's going to be interesting to see what happens. But when the scale IO thing goes online for download, which is going to occur at the end of the month, it was a bummer that we couldn't have everything totally teed up to be there

at EMC World. Yeah, but these things happen. These things happen, right? Our rule of free and frictionless is like, it has to be completely frictionless. It's not like, give me a truckload of your personal information and someone will get back to you. Right. Um, or fill out this form and you might get contacted. No, it's like, bam, binaries there. Yeah. Okay. So it's not an email two days later. Right. Now the, the other thing that, um, what we decided is we said for some of these things, even free and frictionless is insufficient and we're going to have to go full on open source.

And, and, uh, the one that was the driver around that was the Viper controller. And the Viper controller is going gangbusters. Ray, you were actually there, I think. Right, right. Yep, yep. So literally you could hear the SAP dude, what he said on stage. He told us, you know, as a customer, you know, very clearly. And the customer council, we asked them, like, do you like it? Yes. Are there

things that you don't like? Yeah, I don't like that you don't support boom, boom, boom, boom, boom. I don't like that you don't support, you know, this northbound, you know, automation tool. You know, I don't like that this operation is not supported. Okay, great. When we asked them, like, what is the single most important thing that we could do, they all, tool one, the customers and the council basically said open source it. They all had slightly different reasons, right? So the SAP dude said, hey, because we can actually accelerate this feature. Yes. And because we're still a pretty heterogeneous customer, and we want to make sure there's an intrinsic driver that works with that ecosystem.

Right. And then other customers basically said, make it open source because we're freaked out about how much it's now a core part of our stack. Right. And if you guys ever decide to discontinue it we're a dead man we're we're a doo-doo right like we're making this the way that we talked all storage which was the original philosophy of the product right right right right now the good news was that we had already actually made the decision that we were going to do it so it's like phew you know thank

god for that nothing like having the customers go, yeah, well, you're already planning. That's the right thing. That's the right thing. Okay, good. Good validation. But, you know, so, guys, it's not that we're religious about it. It's more that we're just really actually pragmatic, that we've come to the conclusion that people want an open source model for these things where it's geared around community, innovation, speed, not just free and frictionless. Free and frictionless is the superset for everything. We've actually done the work to make sure that, you know,

we've got all these things ironed out. And like we said on stage, I mean, we're all in. The Viper controller and Copperhead is just the first of many. There's other things that you can imagine where community would be really important, like our object stack, hint, hint, nudge, nudge, wink, wink. Yeah, and I think that open sourcing Viper is kind of the best way to broaden its support. You guys were never going to dedicate 100 developers to support every storage array ever made.

And don't forget, it's not just the downward integration that needs to be open, it's the upward one as well. We actually had a lot of customers who were using UCS Director and they're like, we like UCS Director, it works great, but you know what, it doesn't have a ton of support for the storage ecosystem. And we were trying to figure out, like Cisco was like, maybe we could do something and embed and OEM, but it's hard.

And now it's like, no, it's easy. If you guys want to actually take the trunk of Copperhead and you do your own thing with it, go to town. Great. Well, Chad, we're pretty much out of time, and the Skype gods have not been kind to Ray, so he's been dropped off of our call. So we probably should wrap up.

Thanks a lot for being on the podcast. It's been a pleasure. We will certainly put you on the very short list of people we will invite back whenever you're available. Have you got any final words for the folks? One, thank you for inviting me. Two, to all the listeners, thank you for listening to this long dialogue. And three, Ray, wherever you are, we hope that you're well. I'm sure he's well. He's calling on the phone right now. Very good, Chad. We will talk to you

later. For all of you folks, so long from Greybeards on Storage. Join us next month when we will have another industry luminary talking interesting tech about the storage world. For Ray Lucchese, I'm Howard Marks. Thank you very much. It's been a pleasure, and the gray beards are gone for now.