Storage Developer Conference - #128: Surfing the World Wide File

Episode Date: June 15, 2020

Hello, everybody. Mark Carlson here, SNEA Technical Council Co-Chair. Welcome to the SDC Podcast. Every week, the SDC Podcast presents important technical topics to the storage developer community. Each episode is hand-selected by the SNEA Technical Council from the presentations at our annual Storage Developer Conference. The link to the slides is available in the show notes at snea.org slash podcasts. You are listening to STC Podcast, Episode 128. My name is Endio Ruiz Cabrera, and this is my colleague, Fran Lee. We work on Microsoft, and we are part of the Windows SMB team.

Today, we are going to be talking about some changes we have made to allow SMB traffic to flow through quick connections. But first, let's go over some updates on the work we have done the last year. So this is work that is already finished. So we have added to the protocol the normalized name query. So now we have native support for file normalized name information. We have also done some directory caching enhancements. Now you can cache around 500,000 entries for larger directories.

Also, when you're trying to query a directory, you will try to use one megabyte buffer, so we minimize the number of round trips we do. We have added negotiable SMB trapping compression, so now we support the algorithms Express, Express Hoffman, and LCNT1. And also we have added negotiable SMB in the name context for multitenant servers.

So also, this is the work now that we are currently working on. So we are trying to add patterns, scan compressions, algorithms. And storage RDMA push mode to persistent memory. If you are more interested in this subject, I suggest you stick after this talk. So Matthew will be talking about that right after this.

We are working on SMB3 signing so specifically we are trying to switch from AAS CCM to AAS GCM and we are adding new signing and encryption algorithms such as AAS 256 GCM256-CCM, and encryption of RDMA payloads for SMB2, direct placement reads and writes. So this is going to be the outline of our presentation. First, we are going to be talking what are the challenges you face when you're trying to do SMB traffic over the internet. And then we'll do some overview of the QUIC protocol.

I'm going to mention what it is and some of the advantages it has as a new transport. We are going to do an overview of what SMB over QUIC is and mention some of the advantages and downsides it has. We are going to be covering how our SMB components integrate with the QUIC protocol for client-server communication as well as some steps that both clients and server have to take if you want to establish a connection and allow data transfer. So, quick uses TLS 1.3.

So that way it provides authentication and privacy. For that, you'll need to provision some certificates on the server. So we are gonna cover what you need to do to interact with us in that way. Also, we are gonna cover what you need to do to interact with us in that way. Also, we are gonna be covering some of the management tools we have added to allow this provisioning on the server. Lastly, we are gonna mention some performance measurement

we have done on this to see how fast this is, and then we'll do at least one demo. So, why? So, how many of you have used SMB to access local resources? Okay, and how many of you have used SMB to access resources over the internet? Wow, I am surprised. We know VPN?

Okay. Really? Yeah, but in. I see. Yeah, you're using different protocols to do this, but you can never do native SMB because usually, yeah, it's usually kind of hard, at least for normal people, for regular individuals

to use SMB over the internet. And part of this is because, I'm sorry, I keep pressing keys. So part of this is because even though SMB doesn't have any limits on where it can be used, nobody is limiting you to use it only in local area networks or wide area networks. You seldom see traffic of SMB traffic going over the internet natively. So part of the reason for this is

port 445 tends to be blocked in your local router, which is probably a good idea because you want to minimize your attack surface. In general, internet is not a very safe place to be, to have a bunch of ports open. So you usually want port 445 closed. But yeah, I'm sorry. We cannot help with that. But if you are a very knowledgeable individual

and you want to allow port forwarding your router, so you can access your resources from anywhere in the internet, you want to access your local resources at home, or you want to access from, if you want to do that, you are still probably out of luck because I think most ISPs block

communication going to your port 445. So, in and out. So, yeah, I think that's very, I think they think that that's very unsafe and they won't allow it. So, yeah. That's why if you have some clouds, some cloud provider and you have your files there and that cloud provider wants to open 445

for you to access your files through SMB, probably still, if you have Comcast or something like that, probably you won't be able to communicate with that. So the only thing you can, I mean, one of the things you can do here is use VPNs. So, in this case, you will make a look as your client and server belong to the same local network while the traffic is going encrypted over internet,

but in general, this is overkill if all you want to do is use this on SMB traffic, and it's also very inefficient. So let's do a very quick overview of QUIC. QUIC is a new secure networking transport on top of UDP. It's the basis for HTTP3. I think HTTP3 is going to be using mostly QUIC because it's very good for HTTP connections.

It was mostly designed to go hand-in-hand with HTTP2. And it allows faster connection setup. So usually with TCP you have your client needs to send a connection request to the server. The server has to send an acknowledgement, and then the client sends another acknowledgement to the server, and then with the information shared in this exchange, then you establish the TCP connection.

So you wasted at least one round trip there, right? But this connection is not even encrypted yet. If you also want to have some encryption, you need to do an extra round-trip to get TLS. So QUIC has security built in, so it expects to do TLS immediately, so TLS information will be part of the first round-trip. So all in all, you'll just need one round-trip

to get an encrypted connection with Quik, which is better than TCP, which you'll need at least two. Also, it works, it responds better to congestion and packet loss. So if you have some front-end servers and you're directing traffic to some back-end server and you're directing traffic to some backend server and you get some congestion, QUIC has, QUIC packets have a connection ID associated with them.

So when there is some rerouting of these packets on the network, the load balancer can use this connection ID to send the packets to the write backend server and you don't need to like, establish new connections or anything. You don't need to do any new round trips. It also offers better security. Since Quick Uses certificates,

you can trust now that when you connect to a server, the server has to present a certificate to the client, and you can trust that this server is whoever it says it is. Even if someone tries to poison your DNS, you're fine as long as you can verify that that certificate is valid. Also, it has TLS 1.3 integrated, so you can avoid man-in-the-middle attacks

because, yeah, your connection is encrypted. So it presents some improvements over HTTP2, that goes over TCP, so you don't have head-of-line blocking, or at least it's not as bad as it is in TCP. So usually with TCP, if you are taking too long to process some packet, since TCP requires that the packets are delivered on the same order

they were sent, all the other packets that you may have received need to wait until this first packet is fully received and processed. QUIC uses multiple streams and connections, so if you have this packet that for some reason you need to retransmit and you need to wait for it, all different packets on different streams you can keep receiving them in parallel,

so you don't have that problem. It is better, it offers better transition between networks. So usually with TCP, if you are trying to move from one network to the other one, your connection is kind of associated with this network, so you most likely will have to wait for this original connection to time out before the client can reconnect with the server

on the second network. Quick as I mentioned before has a connection ID associated with the packets so if in the second network the server starts receiving these packets with this connection ID that it knows, it doesn't matter since the connection ID has nothing to do with, it's independent from the network. You can, the server can just start receiving these packets immediately

and resume transfers. Quick is also better than loss recovery. So TCP packets have up to four SACK blocks, while quick SACK frames have up to 256. So you can, on the presence of packet loss, you can keep making forward progress for longer before you have to kind of stop. You can keep receiving packets even if you of packet loss, you can keep making forward progress for longer before you have to kind of stop. You can keep receiving packets even if you have packet loss.

There is no retransmission ambiguity. So each quick packet has a sequence ID. So if the client sends a packet and then retransmits it, whenever it gets an ACK from the server, it can know if this server is acknowledging the original packet or any of the retransmissions. So that's something you don't have with TCP and that's transmission ambiguity.

And also it was created, I mean, QUIC was created after TCP. So it brought a bunch of good ideas that TCP was using for congestion control. So you get the good from TCP, and then you add some newer techniques, and you get a better congestion control in general. So it offers portability and agility because it was designed as a user mode library so it's

easier to port these to different platforms and also it allows for more agile development and testing of quick modules because it's user mode you don't have to but so and something nice is it's being standardized by IETS. Question? Sure. I'm told that the Google version of QUIC and the IETS version of QUIC are not quite compatible. Which version are you developing on?

We... Can you repeat the question? Hmm? Oh, I'm sorry. So Google... The question is, Google's version of QUIC and IETF's version of QUIC are not the same.

Which one we are working with? Do you... Yeah, so for that, it's better to redirect that question to our networking team. I can get back to you and try to. I don't know. We just use them.

We use the API they provide for us. So let's go to a brief overview of SMB over QUIC. So the advantages this give us using this new type and transport type is that we get more security because as I mentioned before, servers now have to present a certificate to you so you can verify, you avoid server spoofing from now on.

And since it has TLS baked in, you get encryption for free. In fact, I think by default we disable, I mean, unless you specify otherwise, we disable encryption because QUIC has that already built in. So it offers greater portability. In case you are trying to connect to 4445 and you see it block, then you can just move and do SMB over QUIC, import UDP443,

which will usually work. In fact, there are some experiments Google has done and shows that 93% of the connection success, you have a 93% connection success rate when you're trying to connect to UDP443. And the Windows kernel quick implementation allows multiplexing of multiple protocols on UDP443 using ALPN.

So, but you also get some downsides when you're trying to do this. One of them is performance. This is doing quick, SMB over quick is worse than doing SMB over QUIC is worse than doing SMB encrypted traffic. And one of the main reason for this is a lot of networking equipment is optimized

to understand TCP traffic, and so part of this logic having allotted to the hardware, QUIC is kind of new and that has not happened yet, so most of what we are doing is being done by software, so yeah, you'll suffer performance penalties there. Kerberos cannot be used without means to reach KDC from the client, so you'll have to have NTLM enabled

if client cannot reach KDC. This is not bad for this connection specifically because this NTLM change will happen inside the TLS connection, so you will be safe. But you had to enable NTLM in your connection and some other component that you're not aware of may be using it and you may want to be aware of that. And lastly, TLS encryption is machine to machine

rather than user to machine, so that's also less than ideal. So if there are no questions on this section, I will pass it on to my colleague, Franny, to talk about implementation and integration. Yeah. So I think some people here may be familiar with this picture, and as we released the SMB Direct, or RMA, or SMB in 2012 server. Now is time we introduce another security transport type because of the quick.

So our quick, for SMB layer, so our quick is just another protocol option of transport type, use side by side as our window sockets and RDMA. So what that means, so what we change for our SMB to protocol stack if we introduce the QUIC stack. So our SMB will be later on helped with the quick stack. So quick, MS quick supports multiple quick stream

in single connections. But we're only using one single quick stream in single connections. It's proposed to replace TCP. And also our multi-channel, our everything multi-channel, just handle the multiple quick connection or TCP connections. So we don't have much change right here. So by default, if a client negotiates to server

on the quick connections, we don't enforce the SMP signing encryptions. So we rely on the QUIC connections, we don't enforce the SMB signing encryptions. We rely on the TRS encryption for the QUIC connection. So, and the already talking, for the QUIC, SMB or QUIC, we will need certificate, and we will provide some management utilities to handle how we impose the

certificate for the quick connection successful. So for the SIMB authentication, we don't have change for quick purpose. But because the quick connection is a security connect, TRS connection is security connection, so we do allow the client and the server to negotiate on new contacts to agreement. So if we enforce the SMB2 encryption

on the quick connection or not. That means sometimes you want to, okay, a quick layer. We have already have encryption, but you still prefer you have SM2 encryption. We provide these options. So now we can talk about, so from our client and to our server, so what's happening when we introduce the quick connections. So basically client will open a file,

or open a share, and Client will find the server name by DNS, and at that time, Client will fire a parallel on TCP connection or quick connections. So Client will start, and whatever the channel we set up first for the negotiation. Then if that's whatever the client will negotiate interface with the server which gets the most optimized the protocols.

And that's either by TCP IP or either by QUIC. It will depend on which connection you connect first. And after that one, the client will continue sending whatever the session set up and whatever SMB connections,

SMB messages. But the client don't know if the server supports Quake or not, because it would be impossible to download the servers. So we run the TCP connection and the Quake connection in parallel. But we'll give a high TCP IP connection. So by default, we will first try TCP IP connections.

And also, when let's use command, we introduce another, introduce the mount option. You can specify the threshold type, either Qlik or WSK. Any questions? So you're saying that TCP gets a head start. Yeah. But let's say you have a routed enterprise network,

like Microsoft Campus. Yeah. And there's a glitch. Yes. Let's say you have a routed enterprise network, like Microsoft Campus. Yes. And there's a glitch. And the initial TCP handshake gets messed up. Now you're connected by quick. Yes, we will follow up by quick. But that means you now have what should be a slower connection, even though it's on a local router network.

Okay, can you repeat the question? You say? If that's the case, in an odd case, and we have this problem actually with Hyper-V, the SCVMM, in the case that something goes wrong at the initial startup of the TCP connection, which can happen, it's a network glitch, and normally you just try, but by this point, the quick connection might be established, and normally you just try, but by this point, a quick connection might be established, and now you have a slower than normal connection between yourself and the server.

Yeah, I agree with that, Matthew. Do we retry the TCP connection? We try, right? I think every new connection attempt will be, I don't again, but I think when we say negotiate multi-channel, I believe when we establish secondary connections, it should pick up TCP again. Again, we do this

So you're doing multi-channel over QUIC and TCP at the same time. That's how that would work. So if you have a client and server that support both, even if you connect by quick first, so I can do it if multi-channel is involved, which it would be because you're

doing quick, it's a more advanced version, so then you would automatically also make the TCD connect. That's what I believe should happen. Yes. Can you either summarize that or give the mic to Matthew or something? I think that's an important... I can talk about it at the end. summarize that or give the mic to Matthew or something.

I think that's important. I can talk about it at the end. Okay, that's fine. I just want to make sure that gets on the recording so people understand that. Put a slide in after we're done talking. Okay, we have some code snipped and we're talking about a little bit about

in client side, so what happened when we're using QUIC. So firstly we need to open a section, open a section, that basically open a handle. Then we need to connect, make a connection open. Then we do the stream open. Then we will start the standard streets. So that's it, the basic sequence from client-side. SMB layer reliant on our MS quick implementation.

What's the second form of the first call? The SMB over quick ALVM, what's that? MS quick? That's it, the handle, I'm sorry. Can you repeat the question? Can I repeat the question, what's the question? Yeah, so in the first call, the second parameter says SMB over QUIC,

is that a constant? That's ALPN, a constant, it's SMB, actually, a string. And that's not a constant that the quick stack has to know about, that's something private between client and server? Yes, that's correct. We sell the same LPN name between the client and the server. It's a string that IAM manages in a registry. I didn't get you.

That's defined the constant and predefined in the quick layer. We defined either HTTP or therapy and for SMB. Okay. So, now we go to the server. So, that means in the server side of what service, if we want to use the SMB quick, what happened there? So, first, the way we will open the server first we will open the endpoint on UDP 003. This is me.

And once the listening is up, then we will try to receive the new connection from the cloud, click connections. If when the new connection coming in, then we're trying to find the new certificate for this in our server. If everything looks fine, we find the certificate, and, and other conditions meet, then we, we, server will accept that connection. Then server will beginning to receive the SIMB message through the stream. So, at server, in the server, we don't know, so, what's the user prefer by default,

TCP, IP, or QUIC, so by default, we start both deletion around TCP or QUIC. And the, but we provide options, so you can override. So if you want to just want to the TCP IP, or you just want to the quick. So that's it. It will code the SNP for the server. So basically, we need to open the listener,

and we need to start listener. And after start, waiting in the callback function, we need to tell you what connection is coming, and we need to tell when the stream is coming. So basically, we rely on the MS quick layer to send out our message. So from the client side and the server side. And we need to mention a little bit on here, so for the quick side and the server side. And we need to mention a little bit

on here, so for the quick, how the buffer alignment. So we cannot assume with SMB, single SMB message holding one single buffer. It may possible cause multiple quick buffer. And one quick buffer may possible contains multiple SMB2 message. So from the client or server, so they throw the stream callback to receive the SMB2 message. Then the client or server will pass in this message and do the upper layers of SMP2 stack. But we're talking about, because we are using the Quic, we rely on TRS 1.3, so we have to introduce some server certificate for our server.

So we introduce server certificate mapping and to allow the server knows know what certificates will be used for the new connection coming in from the client. So basically, the server will support multiple principal names to mapping to the single certificate or multiple certificate. And by default, the user cannot assume there is a certificate in the server. So the user have to import the certificate

to the local system star for the use of the SMB servers. So, but if the customer want to block the 4.4.5 and only allow, if the customer only want to use QUIC, they can either block the TCP 4.5 or simply don't start the TCP 4.5 listener on server side. And from the cloud side, as I mentioned earlier, for the next use command, we provide the mount option, mount point option.

You can specify what type of the connection or transfer type you want to use, either quick or WSK. Oh, we have another. So basically for the management, for our server certificates mapping, we introduce three command. So new and remove and get. So how to use the server certificates mapping?

So it's pretty simple. Then you have to have the server certificate and a star in the local system star. And you can do the new server, SMB server certificate mapping, and you give the server DNS name and the sound printer from the certificate. And if you don't want to use that certificate at all,

or you want to get a new certificate, you can remove it and import a new one. Any questions so far? Okay. As we mentioned earlier in this talk, compare with TCP connection. And we have, for the large size I.O., TCP IP have better performance.

Maybe the throughput is about two times of quick. So we already did some initial investigation on this issue. It looks in the quick layer, and one single thread is stay in one CPU and process the sound stream and decryption the message. So and for the fair comparison so in the for the TCP IP we enable the smp2 encryption because in the

quick layer they they already, by default, and you cannot remove the encryption for it. So, for the small side I.O., we see almost similar performance for both read and write. Any question for that, Dave? Both one connection? We did multi-channel, not one, not a single connection.

So earlier slide said you are using multiple flows with a single quick connection. How many flows? In each quick connection, we only allow one single stream. But we rely on SMB to multi-channel features. So we save the connections for both. Yes, that's correct. Yeah. So any questions?

Then we can switch with them. So on that chart, on TCP, I'm assuming that encryption and signing are turned on, and on Quip, they're turned off because they're separately encrypted. Yes, that's correct. Can you repeat the question, Frank? Can you repeat the question? So you asked if the encryption or signing is enabled for TCP. Encryption and signing on the TCP is enabled.

Enabled, yes. But disabled on the QUIC. Yes, because QUIC already does some encryption inside the QUIC layer. So for fair comparison, we have to disable the QUIC. But we do support, so if we want to do encryption. So that's okay. A note about the TCP being twice as fast for RTIO. Is there any plans to make this adaptive so that we're staying? So I think the network team is investigating so why all the traffic, I mean,

why the single-car handle all the decryption or send message. So that's basically from the switch, we analyze and see, okay, that now balance through and across all the calls in the CPU. I think I can give you an answer for that. So at the SMB layer, when we do encryption, we... So in my laptop, and I host a hyperventure,

and I have WAN. And based on that, so we can start up. And another... These are the one page we have to filter on, the TCP and the UDP traffic. We cannot see SMB at that time. Because the TCP and in the Azure, I have a VM. And as we know, we cannot assess the share in the Azure host VM before.

So in those demos, I will see, I will show the TCP will not work. So even that's, that's, we cannot pinable. And we have a register in client side, all right, if we want to use TCP IP, we want to use the quick, so. So we will see what happen. So we will see the ping timeout.

And also we see the retract retransmission for the TCP. That means in this IP, I drew a block in the TCP traffic. Okay, we don't have any connections established. Then we can now, we can try our quick to pass through the firewall stuff. So basically that's when we will force the client connection on the quick. We will see, okay, it first tried the TCP, and then it followed up to fail to the quick,

and we will see some payload there. And after that's done, I can write some files, system test on this. Basically in this test, I will creating a five megabyte, five megabyte files, just, and make sure my test will run a bit faster so I know we do a secret rate and the chicken right and the random read and random right and the mixed the read and write so any questions so far that's running. Yeah. I have one question.

Why did you choose QUIC and not direct TLS over TCP? I think the reason from my side, right, and we have in-house, we already divided for the MS-QUIC. For the... we already divided for the MS Quick. The MS Quick is ready for us to use. Otherwise, I think the network team may be... But I mean, you're going to take your lesson from the day.

Should it... Some are like 20 years ago. Yeah, some have spent a long time. So I think the biggest thing there was the ability for Quick to multiplex over the same port. I think that was kind of important because... Right. So the primary, I think it's probably a Windows implementation reason too, because we did not have a good TLS stack from kernel. So from user mode, yeah, of course, TLS works over TCP 443,

and QUIC kind of came up as the alternative. So there was a shared library development which was developed for both kernel and user mode. It's almost like a static library for the most part. So part of it was implementation reason, and the second part of it was the multiplexing, the ALPNs being clearly defined. So I guess SSL also did allow that, right?

You could multiplex. But would you run over SSL 443 or... Okay. And we do the robot copy with the compression. Yeah, that's kind of set up in my home behind the Netgear router with DDNS and whatnot configured. It's kind of a case where it's a slow link. I think it's 30 Mbps or whatever, but it's actually accessing stuff on my machine at home.

And I think we have specified the compress option too, so the copy file. That's a new option which we have added, so I think it should complete like in half the time as if you would not use compression. Thanks for listening. in this podcast, be sure and join our developers mailing list by sending an email to developers-subscribe at sneha.org. Here you can ask questions and discuss this topic further with your peers in the storage developer community. For additional information about the Storage Developer Conference, visit www.storagedeveloper.org.