On Sun, May 18, 2025 at 1:00 PM Martin Wege <martin.l.wege@xxxxxxxxx> wrote: > > CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@xxxxxxxxxxx. > > > On Thu, May 15, 2025 at 2:29 PM Chuck Lever <chuck.lever@xxxxxxxxxx> wrote: > > > > On 5/14/25 5:50 PM, Martin Wege wrote: > > > On Wed, May 14, 2025 at 1:55 PM NeilBrown <neil@xxxxxxxxxx> wrote: > > >> > > >> On Wed, 14 May 2025, Jeff Layton wrote: > > >> Ignoring source ports makes no sense at all unless you enforce some other > > >> authentication, like krb5 or TLS, or unless you *know* that there are no > > >> unprivileged processes running on any client machines. > > > > > > I don't like to ruin that party, but this is NOT realistic. > > > > > > 1. Kerberos5 support is HARD to set up, and fragile because not all > > > distributions test it on a regular basis. Config is hard, not all > > > distributions support all kinds of encryption methods, and Redhat's > > > crusade&maintainer mobbing to promote sssd at the expense of other > > > solutions left users with a half broken, overcomplicated Windows > > > Active Directory clone called sssd, which is an insane overkill for > > > most scenarios. > > > gssproxy is also a constant source of pain - just Google for the > > > Debian bug reports. > > > > > > Short: Lack of test coverage in distros, not working from time to > > > time, sssd and gssproxy are more of a problem than a solution. > > > > > > It really only makes sense for very big sites and a support contract > > > which covers support and bug fixes for Kerberos5 in NFS+gssproxy. > > > > Brief general comment: We are well aware of the administrative > > challenges presented by Kerberos. :-) > > Well, the primary roadblock for development and TESTING seems to be > the absolute insane setup requirements. Every doc I find talks about > sssd, LDAP, Krb5. > > I think what is needed for small scale testing is the so called > "grocery shop setup": > Two machines with local accounts (i.e. /etc/passwd), one NFS server > with Krb5 server, one NFS client with Krb5 client. > NO LDAP, NO TLS, NO Krb5 preauth > > Unfortunately all docs describing that are GONE. > > > > > > > > 2. TLS: Wanna make NFS even slower? Then use NFS with TLS. > > > > > > NFS filesystem over TLS support then feels like working with molasses. > > > > We'd like to hear quantitative evidence. In general, TLS with a NIC > > that has encryption offload is going to be faster than NFS/Kerberos with > > the privacy service. > > You can test it yourself: > 1. WAN config with 0.1s latency per hop, 2 hops, leads to "simple" > tasks like mkdir, mkfile, mknod to take > 1.6 seconds. It doesn't matter if you use TLS or not, a 100msec transit delay is going to result in abysmal performance. Btw, last week I was doing remote testing during the Bakeathon from the centre of the universe (Valemount, BC. A village with a population about 1000 3 time zones away from Ann Arbor). I got a ping RTT of about 60-70msec, so a lot better than what you listed above. But, yes, performance was not terrific. However, with or without TLS doesn't matter for this case. Try testing with/without TLS when your client->server ping RTT is a reasonable value. > 2. Parallelism is ruined by all traffic going through the TLS layer. I don't see why TLS will make any difference w.r.t. this. Each RPC usually (not always) ends up as a separate TLS data record. Any serialization is just the same as a TCP connection without TLS, from what I can see. > 3. Latency is unbearable high, and [2] is not going to save the > situation this time (never was a solution anyway) Yes, but why would you assume this is used in such an environment? (Try Cifs/SMB over connection with a 100msec transit delay (200msec RTT) and see how well it performs.) Maybe someone can try pinging a server "across town" and see what the RTT is? Also, no one said this was for WAN environments. Performance (or lack thereof) across WANs is another topic. > 3. TLS layer does not respect RPC boundaries, a problem shared with > ssh and other encryption and compression programs. I cannot comment w.r.t. other implementations, but for FreeBSD, each RPC message is typically a separate TLS data record, as I noted above. (Once in a while, two RPC messages may get queued on TCP send queue such that the KTLS sends them as one TLS data record, but it does not delay waiting for more data than 1 RPC message before sending a TLS data record.) > Simply said, TLS > has no concept or api similar to TCP CORK, or just flush. RPC packages > get "stuck" in the TLS layer, leading to excessive waiting times until > more data arrives. This might even not be fixable without a better TLS > protocol, or at least an API which handles message boundaries. It's not a problem for the TLS protocol (a data record can be any size up to 14K, if I recall correctly. It might be an issue for some TLS implementation that chooses to wait for "more data" before sending a small TLS data record. > > > krb5p cannot be offloaded, full stop. > > Sigh. > > Who is claiming that? Same marketing gurus who "foresaw the rise of the TLS"? Well, RPCSEC_GSS using privacy (which is what krb5p is) only encrypts the data portion of the RPC message and not the header. To offload it, the offload hardware would need to know the exact byte ranges within the RPC messages to encrypt/decrypt. It isn't going to happen unless the whole world adopts RPCSEC_GSS with privacy using Kerberos mechanism. TLS on the other hand gets used quite a but. When was the last time you accessed a web site with "http" and not "https"? > > > > > An increasing number of encryption-capable NICs are reaching the > > marketplace, and the selection of encryption algorithms available for > > TLS includes some CPU-efficient choices. > > Poinnt [3]. TLS is just very poorly suited for RPC, and no hardware > acceleration will fix that. Maybe. I have not been able to get any experience with offload hardware. (I have heard that the QAT stuff does require significant setup time, so I suspect that it does not help w.r.t. NFS over TLS, which uses lots of small TLS data records, but I have not seen any test results to confirm this.) > > TLS is designed for large chunk transfers. All the tiny bitsy RPC > packages don't work well over TLS, unless there is a lot of parallel > traffic, or large READ or WRITE transfers. > > > > > Thus our expectation is that TLS will become a more performant > > solution than Kerberos. > > Which marketing team came up with that "prediction"? > > Judging from WAN experience, judging from the design and API deficits, > NFS over TLS is simply a failed experiment. > > Just some statistics: > Our company is consulting 86 companies trying to deploy NFS over TLS > since the beginning of 2024, and NONE of them thinks it is usable, > ready for production or even close to something like that. Everyone is > pretty upset about latency spikes, lack of throughput, very high CPU > usage and other problems. I will note that without offload hardware, I see about 30% of one CPU core being used when transferring a large file over a 1Gbps LAN connection x86-64 with AESNI). This is fine for clients, but means a server with a lot of NFS over TLS active client connections will either need a lot of cores or hardware offload, I think? > > > In addition, the trend is towards always-on > > encryption (QUICv1). IMO you will not be able to avoid encryption-in- > > transit in the future. > > > > > > > Exacerbated by Linux's crazy desire to only support hyper-secure > > > post-quantum encryption method (so no fast arcfour, because that is > > > "insecure", and everyone is expected to only work with AMD > > > Threadripper 3995WX), lack of good threading through the TLS eye of > > > the needle, and LACK of support in NFS clients. > > > > I believe the IETF has also broadly discouraged the use of easy-to- > > defeat encryption algorithms. > > Sometimes I think the IETF decisions are a) from Mars or b) done by > marketing, and not rooted in reality. I know Elon Musk wants to go to Mars, but I don't think he participates in IETF meetings. > > > Perhaps this desire is not limited to only > > Linux. > > And how should the "small shops" (below what Oracle would call > multi-million enterprises) should deploy an usable NFS over TLS > configuration then? Basically you force them to either have ZERO > security, or a security they CANNOT AFFORD. I deployed it here at home with nothing except command line openssl. (There are easier ways to do a site local CA, but it works.) You can look at: https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt or for Kerberos https://people.freebsd.org/~rmacklem/nfs-krb5-setup.txt rick > No middle ground. > > Which brings us back to the debate of the secure/insecure export > option, and overkill Krb5 configs like sssd. > > > > > Using a deprecated encryption algorithm means you get very little > > real security in addition to worse performance, so it's not a good > > choice. > > Then please forget about using TLS. and don't recommend a > broken-by-design solution. > > Thanks, > Martin >
