Grey Beards on Systems - 46: Greybeards discuss Dell EMC World2017 happenings on vBrownBag

Episode Date: June 2, 2017

In this episode Howard and I were both at Dell EMC World2017 this past month and Alastair Cooke (@DemitasseNZ) asked us to do a talk at the show for the vBrownBag group (Youtube video here). The GreyBeards asked for a copy of the audio for this podcast. Sorry about the background noise, but we recorded live at … Continue reading "46: Greybeards discuss Dell EMC World2017 happenings on vBrownBag"

Hello, and welcome to an episode of Greybeards on Storage, recorded here on the V Brown Bag stage at Deli MC World 2007 in the lovely Venetian Hotel Resort, Casino, and Hell Hole of Las Vegas, Nevada. I wouldn't call it a hell hole. Or is Las Vegas a hell hole? No, Las Vegas is a hell hole. Oh, I gotcha. And I am your left-hand co-host, Howard Marks, and my associate… This is Ray Lucchese, and I'm the right-hand host, I think. And… Oh, it's wrong.

Sorry. Oh, the other left. The other left, he says. Yes, we know storage, but directions we have trouble with. Yeah, it's that coding thing, you know. And so this has been the first real Dell EMC world after the merger. There was a small event in Austin last fall. Yeah, which I didn't go to.

Did you go to? No, I didn't go to it either. And it was planned so quickly that it was mostly the Dell event with EMC added. Right. I, for one, am impressed with how effectively they're making the public face of the merger look. Yeah, it looks very good. I mean, and Mike Dell, Michael Dell talked at length early on in the first keynote about

how things are coming together and how they're sort of partitioning the various market segments and that sort of stuff, which I found pretty interesting. Although I found Michael is pretty much of a ham. He was on the stage for, I think, five days or something like that. Yes. Although, you know, in my personal dealings with him, Michael is an incredibly impressive gentleman. No doubt.

I would have to say I haven't had any personal dealings with him. I met him at PC Magazine back in the day and probably hadn't even shaken his hand in 15 years. And then I saw him at EMC World last year and he said, hi, Howard. And I wasn't wearing my badge. So there's just a memory there that's very impressive. I can't do that sort of stuff. No, I mean, the people come up to me and say they're fans of the podcast, which,

you know, A, I find terribly embarrassing. And B, it's kind of like, oh, right, we went three years ago. I have no idea what your name is now, and you're wearing a badge. Yeah, yeah, yeah. No, I have occasional fans of the podcast come up all the time and appreciate what we're doing, and, you know, occasionally I want to take a picture with me and stuff like that. It's pretty embarrassing for me. But I enjoy the notoriety, I guess, or whatever you'd call it. So what's new here at Dell EMC World these days?

Well, I just came out of a meeting with an outfit that I'd seen at EMC World a couple of times and didn't know what they did, DataDobi. What is DataDobi? They're a file migration Adobe? D-A-T-A-D-O-B-I. Okay. They're a file migration tool. Oh, okay. And like Robocopy, they'll walk the file system and copy the files.

But unlike Robocopy, they'll do that iteratively because you always know... Through the directory string? Yeah, you know, you say, I'm going to move the data from this FAS to that Unity. Oh, okay. And there's 10 million files and 4,000 of them are locked. And, you know, the user ID you're running under doesn't have permission to some of them. And so they'll make a pass and then give you a report and you can fix it. And they'll only look at those files again.

And so it's really that, you know, making that forklift upgrade much easier. Right, right, right. And, you know, I like coming to these events just so I can see those little guys. Yeah, there are a lot of actually Dell partners or EMC partners that are in the area and such, which is kind of interesting. I only really have been on Solution floor for that tour they had yesterday morning, which didn't actually show much other than Dell EMC, really. Well, I have to ding the organizing folks just a little on that.

The idea of serving the press and analyst and influencer group breakfast on the show floor but not telling anyone but Dell employees that they should be in their booths to talk to us, I think was a mistake. It would be much more effective if everybody was there. I imagine, I'm not sure we'd have enough time to do everything that we'd want to, but you and I kind of just randomly go through the process of the show floor versus the tour, which was a different option. Yeah. And speaking of the solutions exchange or the show floor, there seem to be less third parties than I've been used to from EMC World in the past.

Now, you know, that part of that's an organizational thing. Interestingly, there's a wider diversity of third parties because we have, you know, our friends from Datrium. Right, Datrium is here. I saw Craig. Who typically at EMC World would have been well you can't exhibit your storage guys. Yeah, yeah. But they're Dell partners. Yep. And so you know we've we've talked several times about how we would rather see independent events with

a broader range of exhibitors. It's hard for a vendor like these guys to to support that sort of thing. I mean, I understand. Obviously, VMware is the prime example of you see a lot of different vendors at VMworld events and that sort of stuff. Right. Which is hard to see at these more specific vendor events. In terms of the actual Dell EMC product announcements. A lot of interesting stuff. Well, a lot of evolutionary stuff. A lot of evolutionary

stuff. I'm a little frustrated about the discussion of 14G servers. Because it's pre-announce. Because we're geeks. It's not going to be out for another couple of months? I don't mind

talk to me in May about products you're going to release in July or August. Okay. I mind make a big deal about the fact you're coming out with a new group of servers in July and August, but because Intel hasn't released the details of what those Xeons are, all you can say is they're going to be great. And you can't really talk about, well, you know, instead of having four NVMe U.2 slots,

they'll now have 25. Yeah, they didn't mention the fact there was like 19x more NVMe slots and stuff like that. So they've got some of that information there. But it's an Intel thing. You can't blame Dell EMC for an Intel. No, I blame Dell EMC

for making the announcement this early. Ah, I see. You know, making the announcement this early. I see. It's just a tease, and I'm a little bit put out by the teasing aspect of it. I imagine it's an Intel timing thing as much as anything. Maybe when they first started this thing and had intended to release the 14G announcement, it was earlier in it. I hadn't considered that Intel may have slipped the schedule.

Not that they, I have no idea. No, it is possible. It's certainly a possibility. But, you know, I'm in the middle of acquiring new servers for the lab, and so. And you want the 14G servers. I, well now. They're available, yeah.

Now I'm at the, I want the 14G servers, and I don't know if I can wait, because my current servers aren't up to the tier zero stuff. Right, right, right. And you certainly want tier zero for your lab, because you're going to be doing a lot of that stuff. Yeah, and so, you know, in terms of Dell EMC, they've kind of abandoned tier zero temporarily. For the moment, until it gets a little bit more mature. Well, DSSD was kind of the classic example of what happens if you start early, and the only way to solve the problem that early is hardware.

But hardware takes so long that a software solution beats you or comes to market at the same time cheaper. I'm kind of a hardware guy. I like the hardware engineering side of things and I thought DSSD was an interesting product but it had a lot of hardware and I think the challenge there is you've got, like you say, there's a certain road race here between the software only solution and a hardware solution and if you're going to play the hardware game you've got to continue to keep beyond that software only solution. You have, a hardware based solution has to have a substantial advantage.

Oh yeah, 10x plus, right? And when, yeah. The one thing they did tell us about the 14G servers that I'm really excited about is NVDIMs, of course, because NVDIMs are the critical difference between the storage appliances and software-defined storage. Ah. Because if you're running vSAN, a write has to get written to flash. An SSD. Yeah. So write latency is the latency of the flash.

Right. So write latency is the latency of the flash. But if they have NVDIMs and if they have RDMA NICs, then a write can come into one node in the vSAN. That node via RDMA can write not only into its NVDIM but into a partner's NVDIM. At NVDIM type of latency. At NVDIM and RDMA latencies and so now we're talking about 150-300 microseconds of write latency in vSAN. Now I'm not predicting, you know, I'm predicting but I don't have any knowledge that that's coming but it opens the door for that. You know, HPE now offers NVDIMs on their latest generation of servers.

If that becomes a standard feature, then the software guys can start taking advantage of it and saying, order the NVDIM and your write performance will be five times better. I think the critical question for these guys and a lot of the players is, I mean, the software-defined storage is becoming more and more prevalent in the customer environments and getting more and more powerful with all flash and the capabilities are starting to become more and more equivalent almost to what's available in a standalone storage appliance. Why would you want a standalone storage appliance anymore? I think if you have multiple operating system hypervisor environment, it's a no- appliance anymore. I think if you have multiple operating system

hypervisor environment, it's a no-brainer. Yeah, there's a couple of reasons. If you're not entirely virtualized. Yeah, oh, sir. And there are good reasons to not, well, to not even, even if we're talking about an all x86 environment. Right.

You know, leaving aside the, and we have that one hpux or oracle on sunbox right um you may have the sql server that requires 256 gig of memory and 20 cores yeah and it wants it all and you could virtualize, but it would take up the whole host and it wouldn't buy you anything. Right, right, right. But the other thing is, depending on which HCI solution you're looking at, the cost may not be as low as you think. Some of the more expensive HCI solutions can, in fact, cost more as hybrids than a converged solution with an all-flash array. And lastly, while HCI simplifies operations in many ways, it complicates them in other ways.

How so? Well, when we brought VMware into the data center, there was the efficiency play. I can get rid of these 40 little servers running little tasks and run them as 40 VMs and get the utilization and the space and all of that. And in fact when I was in New Jersey my lab was in a client's data center where they had designed the data center and then when they moved in they virtualized so that they had three times the space they needed. They needed it so they had the space. So I got to camp there without so that I so that when they needed tech support they could just poke their head in my office. That

was good. It was a good deal for a while. Yeah, for both of you I imagine. But when we virtualized with external storage all of a sudden the server became stateless. If I need to do a firmware upgrade, I put it in maintenance mode, I run vMotion, all that stuff gets off of it, I can do a memory upgrade, I can do a firmware upgrade, I can maintain hardware in the middle of the day without working nights and weekends. It was stateless. And now when I add storage back, it's not stateless anymore. And I either have to take that node out of the cluster

and reduce my cluster's resiliency while I'm doing maintenance, or I have to migrate all the data off, which generates a huge amount of network traffic and takes hours. Right, right. And even if I said, okay, I'm going to put this in maintenance mode and I'm going to program my vSAN so that it doesn't start an auto rebuild for the first hour. Yeah. When that node comes back, there's a burst of network and storage traffic to bring it up to date.

But you can add storage to something like vSAN without having to take the node out. I mean, it depends on what you have to do, right? It depends what you have to do. Memory is a different game. Yeah, if you have to add memory, if you have to add a PCIe card, the slots in servers today are hot swap PCIe in general. I don't know anyone who has the nerve to actually do that. Frequently you have to pull the riser out so it's just like not really...

It's technically hot swap but nobody hot swaps it. I was talking to the Unity guys out there and they were able to... I mean they effectively the same controller cabinet you know and packaging goes across their whole line, and you can actually just hot-swap the control units. Right. Well, I mean, much like the flash, the pure flash array. Right. That they built a cabinet that takes two x86 servers in canisters.

Yes, yes. And that can be six-node x86 servers for the low end and 22-core x86 servers for the high end and 22 core x86 servers for the high end. With more memory and all that stuff. And that means that you can update live. You know, tell it to fail everything over to controller A, pull controller B out, put the new bigger one in, fail everything back. And this ties into the changes that we're seeing in the pricing and upgrade model. Where it used to be, if you bought a CX300 and you outgrew it, you needed a CX500.

You had to buy a whole new clarion. You might be able to move the shelves over. But you also had to re-buy all the options. Snapshots cost money and replication cost money. Your old system's gone. Your new system needs them all. And it really was a forklift upgrade. And if we look at what's going on in the market in general,

we're seeing, oh, well, you can upgrade those controllers and just pay the difference in cost from some vendors right and I mean actually the most interesting thing Dell EMC announced to my point of view is a new consumption model the flex model where you can buy a VX rack and pay by the month with a 12-month commitment, which is not the full list price over those 12 months. And then the price goes down for the next 12 months. But if in month 14 the project you bought it for is over, you can just return it and stop paying. So it's not quite an elastic cloud consumption model,

but it's not the forklift upgrade and rebuy all of the options again model that we had 10 years ago. So, you know, and that, I think, is really the advantage that Dell EMC and HPE and to a lesser extent IBM can all support that sort of thing. Because that's financial engineering. These big financially stable companies can borrow money at say prime minus one and they can build into

this. Kind of like us which is prime plus seven or something like that. Or if you're a startup you really can't borrow money. At all, yeah. And, you know, you have your capital and you have to pay sales guys with capital. Right, right, right. So you have to really be parsoma. Right. And so the big guys can do this financial engineering, borrow money really cheap,

build an imputed interest rate into the financing of getting the system. That's, you know, we went from prime minus one to prime plus one, but of course we all know there's a little bit less discount flexibility if you're using the special financing. And so they'll make their money on the finance side, that having the finance side makes them more flexible. You know, much as I love the new tech from startups, this becomes a really good reason to play with the billion dollar boys. Oh, the other thing I was talking to a customer today, I mean, there's an advantage with the big guys, I mean, because they've got effectively a sunk install base. So this customer was looking at a new startup, which would have been like maybe a quarter

of a million dollar play, but their old equipment is four or five million dollars, so their support leverage is significantly different, too. Right. Yeah. So. That's interesting. Interesting. So the VMAX stuff, they came out with a refresh of VMAX, refresh.

They refreshed practically everything. Did they refresh the data domain? So the IDPA thing, right? They did some minor refreshes on data domain, but I actually am not paying much attention to data domain anymore. I think the day of data domain has passed because the way we make backups change. Data domain was designed for weekly full daily incremental. Right, right. And that

produces and a month's retention. Yeah. So that means there's at least four duplicates because we made four backups within that month. But the whole weekly full daily incremental process was based around tape. That if you had a monthly full, if you ever had to do a full restore, you had to find 31 tapes and mount them, and that was more work than anybody was willing to do. So you had the full and a couple of daily backups and that was it. And now we're backing up to disk. And because we're backing up to disk.

We have been backing up to disk for a long time though. No, but the backup soft, for the first five years of backing up to disk. It was really tape. It was virtually tape. It created a file that emulated a tape. Yeah. Even if we're not talking about virtual tape libraries. I know. a file that emulated a tape. Yeah, yeah, yeah. Even if we're not talking about virtual tape libraries.

I know, I know, I know. But today, every backup application does incremental forever. If you're backing up vSphere and a lot of the data we're backing up comes from vSphere, the incrementals are changed block tracking incrementals, not whole file incrementals. And so that we're creating 10 duplicates and we bought a data domain to decreate them as we backed it up. Well, now we're creating two duplicates. Maybe, yeah.

And the cost of a data domain is roughly 8 to 10 times the cost of an ECS. So if I tell my backup software, back up to the object store, I could actually save money that brings the object store into my data center so my application developers have an object store. I see it as a good foot in the door. Right, right.

And when we start talking about very long retention, it just makes much more sense. And so what they've done, they've got the DD virtual edition and stuff like that. You can run that in the cloud now if you want. And that makes a lot of sense because the restrictions of public cloud storage and getting around it. I've started thinking that, you know, we talked about storage virtualization in my career at least twice. At least? Oh, God, I've been four or five times storage virtualization. Well, you've been a storage guy longer than I have.

I used to be a network guy, don't tell anybody. But it never got any traction because... It's different environments if traction was good. USP, VSP. Yeah, yeah, yeah. And IBM. Right, right. But that's pretty much it.

Falcon Store never really got any traction. EMC had like three solutions over the year. None of them got any traction. And much of that's because it was, let me aggregate five smart arrays. And in EC2, EBS is dirt stupid. And it might be that the solutions like FalconSt Store or like Data Domain as a VM, as an EC2 instance, make sense in the public cloud because the backends are so stupid I have to layer on snapshots and other services.

And that's what the... Do you think there's a VMAX virtual edition in the future? Well, I know there's a VMAX, there's the future? Well, I know there's a Unity VE. Yeah, that's true. I don't know what the Dell EMC position on using it in production is. Those things used to be demonstrations. Yeah, yeah, you have limited terabytes and stuff like that. But I don't think we'll see a VMAX virtual edition.

Because the secret sauce, the reason you buy a VMAX is RAS. Yeah, yeah, yeah. I don't buy a VMAX because I like the way VMAXs do snapshots. I buy a VMAX because that thing runs all the time. Yeah, yeah. And because it supports my mainframe. Yeah. So I don't see that feature set running as, it would have to be eight VMs tightly coupled

to provide that level of RAS, and I don't see that happening. Ah, interesting. At one level, I think from a software perspective, you could do it. It's just a question of whether you want to put that effort in. And you're running on EC2 anyways, which can, you know, probably has worse reliability than the VMAX.

Right, no, no. And as my friend Laz Vakiridis, who works closely with public cloud guys, says Amazon occasionally likes to shoot things in the head. And your EC2 instance just blows up. If it was a cloud native app, that's fine. If it wasn't a Cloud Native map, it reboots. But you don't want storage systems being shot in the head. So, I don't see that happening.

But talking about that VMAX RAS, I was talking to one customer who's buying Nutanix, you know, and of course this being Dell EMC, well, they're buying the Dell XC version. Yeah, Nutanix, yes. And they're using it for applications that they're taking off the VMAX. Really? But because at some point in time, some committee decided that application required higher resiliency than a clarion could provide. Okay, that's why it got on VMAX in the first place.

And they couldn't by a unity because that would mean having to reassess that decision. Right. But if they move it to HCI, they just have to say, this is resilient enough. So the application that never needed to be on the vmax in the first place yeah can move the HCI but can't move to unity it's just another proof point that politics technology yeah yeah politics matters a lot and so you know why people buy things you have to dig into multiple layers yeah that yeah no I understand understand but you know, why people buy things, you have to dig into multiple layers of that.

Yeah, no, I understand. But, you know, we got Unity updates. We got Extreme IO updates. Right. I'm very happy about some of the Extreme IO updates. Really? And they're faster, they're better efficiency, they're denser. And they're actually more scale-out.

Than they were before? Yeah, in the original Xtreme IO, which is now X1, a bit of a retronym, you could only have an even number of bricks. You could have two bricks, you could have four bricks, you could have six bricks, or eight bricks, you could not have three, five, or seven. And I frequently describe that as a scale-out architecture, but not a scale-out implementation. You know, it was very, you know, some of those, and so they've addressed those things.

But the big advantage is, you know, we talked about NVDIMMs. Well, they added NVRAM. Now, in this case, it's a PCIe NVRAM card. Right, right, right. But write latency is now... Well, I mean it's primarily for the write buffer. Really, I was told it was like for a metadata backup in case of a power failure. But they are now acting when it's in the NV RAM in both controllers in the brick.

And of course the controllers are InfiniBand connected, so it is really that RDMA we talked about with VSAN. And so now they can have not just read latencies well under a millisecond, but write latencies well under a millisecond. Yeah, they did improve the write latency considerably. And over the past five or six years, the read-write mix has shifted more towards the write. Do you think it's more than 50-50? No, I don't think it's more than 50-50.

I think it's... It used to be 80-20 kind of. I think it went from 70-30 to 60-40. But if you look at VDI specifically, VDI is 60-40 the other way. Yeah, yeah. VDI is more write-intensive than it is read-intensive. Right. And so, you know, it becomes a better platform for those write-intensive applications.

And just for the, I haven't figured out whether my application is write intensive or not. Yeah, yeah. So, you know. Well, that's interesting. So it's, you know, we haven't seen this story play out. These big mergers take a while. They came out with a new version and update of the SC Compellent series. Saw that.

Which, you know, I'm slightly. I haven't really talked to those guys, but more processors and better performance kind of thing. But it's a low-end solution, right? Well, frankly, Compellent was always software-defined storage. You know, even the original Compellent was a pair of servers with NVRAM cards added. Yeah. And shelves of disk and smart software. Yeah, yeah.

So it fits really well when Dell acquired them. The positioning's changed several times. And I think basically, and I heard this from a former Compellent executive who shall remain nameless, that both lines have sufficient revenue to justify the R&D. And we frequently think about there's a merger and well there's two mid-range disk arrays. There's a Unity and the SC. Why do you need both? And the answer is they're different customers. It's not even big, small, direct attach. It's just they're different customers. And

Compellent always had the happiest customers I ever met. Really? That people loved their Compellent. And part of that was the company provided good service. Part of that was while they were competing with Clarion, they didn't make you rebuy the software. You could because the controllers were just servers. When new controllers came out, you just bought new controllers. And the array to which you

bought the options was still the same array even if you replaced, like George Washington's axe, the handle and the head. It still occupied the same space and had the same software licenses. We used to call that mind-melding of storage between two different controllers It still occupied the same space and had the same software licenses. We used to call that mind-melding of storage between two different controllers and stuff like that. In this case, it's licensing. They have killed some of the Dell storage products off. Frankly, and I don't blame them at all for killing off things like fluid file system. They have Unity and they have Isilon. Isilon, now. Why would they need that?

And fluid file system fit kind of halfway in between. And there's just no reason to spend that R&D for the number of customers they had for fluid file system. Right, right. But the product rationalization has been less vicious than I expected. They borrowed a lot of money and everybody expected there to be huge cost cutting. And there's been some obviously, but not that. And it will continue to roll out no doubt over time as they rationalize what's going on. And products just reach the end of their natural life, which you know,

pretty much is what happened to Equalogic. It was just that Equalogic was a great solution, but today I think 90% of the customers would be better served by vSAN than by Equalogic. And so why have it? And Michael Dell mentioned a couple of times the $4.5 billion of R&D spent per year, which is pretty good. And certainly better than what EMC was doing originally. And, you know, the combined group would have done if they, you know, if they'd separated and stuff like that. So do you read that as an increase in R&D from... I say it's say it's on plan. What's not on plan,

now EMC was very aggressive from an acquisition perspective. Yes. And they're certainly not looking as much to do that considering where they're at today. They just made the biggest acquisition in God awful history. Right, well I mean, to some extent, you know, you just had Thanksgiving dinner. Yeah. And you know, you don't want any snacks. You know to digest this before we think about the next one. But there's a lot and we both know there's a lot of startups out there with interesting technology that could fit very well in a number of different solutions in these products. Speaking of which, did you see Micron's announcement last week? No, I didn't.

What happened to Micron? So Micron is OEMing Acceleros software. Oh yeah, I did see that. The new SSD controller. Well, you know, when Micron hired Darren Thomas and that group, I said publicly that you don't hire the guy who used to run Dell Storage to sell SSDs. Right. There was something upstack coming down the road. Yeah.

And it seems this is the first one of those. First shoe of that stuff, yeah. Where they are packaging a server and packing it full of U.2 NVMe Micron drives with Acceleros software, which we've discussed ad infinitum on the podcast. The various versions of podcasts and blog posts, I might add. It's very impressive stuff. Yeah, yeah.

And so now they've got a tier zero kind of alternative to Western Digital Sand Discs and Finiflash, which is more a bulk flash solution. So great, kudos to our friends at Accelero for pulling that off. Yeah, it's a great, I think it's a great solution. It looked like they had a very smart team pulling that together. The question is going to be when they add data services and how big an impact that has on performance.

Right. Because the first NVMe-based shared array is now out. Yeah. It's not NVMe over Fabric yet. It's still iSCSI or Fiber Channel. Right. But they're going to do that.

And when the mainstream storage vendors with mainstream services can get their latencies down, how much of a market is there left for really fast? So VMAX guys claim 350 microsecond latencies as well. I mean, this is pretty serious stuff, especially for non-NVME. I mean, standard SAS, SATA, you know. Yeah, I mean, we have, there's lies, there's damn lies, there's statistics. No, no, no, no. And I'm not really, you know, saying they're lying,

but I am saying that, you know that you can't give me one number. There's seven numbers I have to see. Oh, at least, at least. How deep was the queue? How big was the I.O.? Yeah, yeah, yeah. How big was the data set? Was it doing anything else?

How many channels did it cross? Even if there was one I.O. operation, two of VMAX all by itself, and it came back in 250 microseconds, that's frigging impressive. It's impressive if it's a read out of the RAM cache. It is. It's incredibly impressive if it's a read off of an SSD. And, you know, I can be impressed or I can be very impressed. And sometimes I would like to know which.

I guess. They did mention 6.7 million random read hits. Yeah, but I didn't see where it was 6 million random hits at 350 milliseconds. No, microseconds. So it's two separate numbers. Right. But yeah, I understand that. But if I look at least recent LRT from SPC1, it's like 10% at best.

It is read hits and stuff like that yeah so it's an impressive number even if it was a read hit yes and and the really interesting thing is you know we're we're just at the beginning of flash is normal yeah God and and that means now application structures can change. I talked to these guys to see, you still sell hybrid? Hybrid, well, we do, but it's VMAX 3 and it's really there for customers that are making a transition over time and sooner or later they'll all be on flash. Yeah.

And if it were, I think that the current flash shortage is going to slow that transition down about a quarter. But this time next year, I would expect primary storage at 50,000 in array and up. Leave out the SMB world. Right, right. It's going to be 85% or 90% all flash. There are good reasons to do hybrids in the SMB world because you have nobody to manage anything and you only have one storage system. But if we're talking about people who have data centers and have multiple storage systems and maybe an object store, then 90% of those primary storage systems next year should be all flash.

Yeah. And about this time next year is when I expect the rate of the cost of flash to return to the 30% compound annual decrease rate. Right, right. That we're in a shortage now, and prices are actually rising and there are delivery problems. But there are fabs under construction and the transition from 2D to 3D has been going slower and yields haven't been as good as they're supposed to be. But the transition from 32 layer to 64 layer seems to be smoother than

the transition layer from 2D to 3D. Do you think Micron will move beyond tier zero? I don't think so. Well, I don't see the… I mean, it's an iffy question, you know. I mean, they're obviously a drive OEM for lots of these customers, right? I don't see them moving past all flash. Oh, yeah, and they will never. Yeah, why would they? Yeah. for a lot of these customers. I don't see them moving past all flash.

Oh, yeah, and they will never. Yeah, why would they? And I think that the market, because of consolidation, has changed so dramatically that fear of competing with your customers isn't what it used to be. And I remember our friend Steve Ciccola designed ICE at Seagate and then Seagate spun it off to Zyotech because they didn't want to compete with their customers. But back then there were seven hard drive vendors and if EMC having a disk array annoyed EMC, EMC could buy Fujitsu or Toshiba or Samsung disk drives. Today, in the disk drive business, there's two and a half vendors.

In the flash business, there's really four. Depending on how you talk about the Western digital Toshiba joint venture, that's one or two. But the Intel Micron joint venture is one. SK Hynix is one. Samsung. Samsung is 40% of the market all by themselves. That's about it. And so if you're a micron, you have to worry less about whether you're going to annoy your customers because they've got fewer places to go.

Yeah, at best. I mean, I see it being a gradual transition beyond tier zero if they even decide to go down that route. Because it's, you know, they want performance, performance, performance, and that says less and less data services, really. Right, yeah. At our level, yeah. Yeah, there's, you know, when you- Well. This is some magic sauce we haven't figured out yet.

It's happened before. People working on that, I'm sure that we'll see that over time. Gosh. So what about Isilon? So Isilon came out with an all-flash version of their solution, also denser. And they mentioned the fact that it was both compute scalable or storage scalable independent or together. So you could run VMs on Isilon controllers?

I don't think you can run VMs on Isilon controllers, but they did move from what was essentially a server to much more a four server in canisters solution, which may gave them greater flexibility and they added a great big JBOD because sometimes you just have a lot of data and you don't have a lot of IO. And they've improved their small file layout. Right. That Isilon was originally like most of the scale out systems of the era. It was HPC kind of guys. Lots of data.

Well, it was HPC and oil and gas. I'm big reads, big writes, not 4K, not small files. Not user level kind of stuff. And they're making it, and so today's Isilon is much better for the things you would have used Solera or an FAS for than strictly what it was before. So speaking of Unity, really, they increased their file system size from 64 terabytes to 256, I think. And before that, it was like 16. So within the last year, they've kind of multiplied by a factor of 16, I guess. So it's pretty impressive.

And, you know, people just have more and more data. Right. I mean, I remember since I've been a storage guy, when I was at SNW, vendors claiming they had a petabyte customer. Right. Which was pretty impressive. Which was impressive. I mean, this is less than... Not only is you can't hit a guy out there who doesn't have at least a petabyte customer. Right. Which was pretty impressive. Which was impressive. I mean, this is less than...

Not only is you can't hit a guy out there who doesn't have at least a petabyte and a half. Well, now they're talking about petabytes in a rack. Yeah. It's not a... Effective, quote, unquote. Yeah, it's not a petabyte customer. It's a petabyte in a rack.

And this customer has a row of 20 of those racks. So, you know, the quantity of data we're dealing with has just grown immensely. Right. Well, I think we've babbled on pretty much long enough. I think it's good enough. So. And appreciate the time here with the V. Brown Bank Group.

Yes. So, from here at beautiful Venetian Hotel, Casino, Resort, and Hell Hole. In front of the hot lights, mind you. Yes. We are at the V Brown Bag stage making our contribution to community. Once again, this has been Gray Beards on Storage. I'm Howard Marks. Goodbye, Ray. I'm Ray Lucchese. Goodbye, Howard.

See you.