Hi Darrick, On Tue, Jul 29 2025, Darrick J. Wong wrote: > On Tue, Jul 29, 2025 at 02:56:02PM +0100, Luis Henriques wrote: >> Hi! >> >> I know this has been discussed several times in several places, and the >> recent(ish) addition of NOTIFY_RESEND is an important step towards being >> able to restart a user-space FUSE server. >> >> While looking at how to restart a server that uses the libfuse lowlevel >> API, I've created an RFC pull request [1] to understand whether adding >> support for this operation would be something acceptable in the project. > > Just speaking for fuse2fs here -- that would be kinda nifty if libfuse > could restart itself. It's unclear if doing so will actually enable us > to clear the condition that caused the failure in the first place, but I > suppose fuse2fs /does/ have e2fsck -fy at hand. So maybe restarts > aren't totally crazy. Maybe my PR lacks a bit of ambition -- it's goal wasn't to have libfuse do the restart itself. Instead, it simply adds some visibility into the opaque data structures so that a FUSE server could re-initialise a session without having to go through a full remount. But sure, there are other things that could be added to the library as well. For example, in my current experiments, the FUSE server needs start some sort of "file descriptor server" to keep the fd alive for the restart. This daemon could be optionally provided in libfuse itself, which could also be used to store all sorts of blobs needed by the file system after recovery is done. >> The PR doesn't do anything sophisticated, it simply hacks into the opaque >> libfuse data structures so that a server could set some of the sessions' >> fields. >> >> So, a FUSE server simply has to save the /dev/fuse file descriptor and >> pass it to libfuse while recovering, after a restart or a crash. The >> mentioned NOTIFY_RESEND should be used so that no requests are lost, of >> course. And there are probably other data structures that user-space file >> systems will have to keep track as well, so that everything can be >> restored. (The parameters set in the INIT phase, for example.) > > Yeah, I don't know how that would work in practice. Would the kernel > send back the old connection flags and whatnot via some sort of > FUSE_REINIT request, and the fuse server can either decide that it will > try to recover, or just bail out? That would be an option. But my current idea would be that the server would need to store those somewhere and simply assume they are still OK after reconnecting. The kernel wouldn't need to know the user-space was replaced by another server, potentially different, after an upgrade for example. Right now, AFAIU, restarting a FUSE server *can* be done without any help from the kernel side, as long as the fd is kept alive. The NOTIFY_RESEND is used only for resending FUSE requests for which the kernel is currently waiting replies for. So, for example if the kernel sends a FUSE_READ to user-space and the server crashes while trying to serve it, the kernel will still be waiting for that reply. However, a new server trying to recover from the crash will have no way to know that. And this is where the NOTIFY_RESEND is useful. >> But, from the discussion with Bernd in the PR, one of the things that >> would be good to have is for the kernel to send back to user-space the >> information about the inodes it already knows about. >> >> I have been playing with this idea with a patch that simply sends out >> LOOKUPs for each of these inodes. This could be done through a new >> NOTIFY_RESEND_INODES, or maybe it could be an extra operation added to the >> already existing NOTIFY_RESEND. > > I have no idea if NOTIFY_RESEND already does this, but you'd probably > want to purge all the unreferenced dentries/inodes to reduce the amount > of re-querying. No, NOTIFY_RESEND doesn't purge any of those; currently it simply resend all the requests. > I gather that any fuse server that wants to reboot itself would either > have to persist what the nodeids map to, or otherwise stabilize them? > For example, fuse2fs could set the nodeid to match the ext2 inode > numbers. Then reconnecting them wouldn't be too hard. Right, that's my understanding as well -- restarting a server requires stable nodeids. IIRC most (all?) examples shipped with libfuse can't be restarted because they cast a pointer (the memory address to some sort of inode data struct) and use that as the nodeid. >> Anyway, before spending any more time with this, I wanted to ask whether >> this is something that could be acceptable in the kernel, if people think >> a different approach should be followed, or if I'm simply trying to solve >> the wrong problem. >> >> Thanks in advance for any feedback on this. >> >> [1] https://github.com/libfuse/libfuse/pull/1219 > > Who calls fuse_session_reinitialize() ? Ah! Good question! So, my idea was that a FUSE server would do something like this: fuse_session_new() if (do_recovery) { get_old_fd() fuse_session_reinitialize() fuse_lowlevel_notify_resend() } else fuse_session_mount() fuse_daemonize() fuse_session_loop_mt() Anyway, my initial concerns with restartability started because it is currently not possible to restart a server that uses libfuse without hacking into it's internal data structures. The idea of resending all LOOKUPs just came from the discussion in the PR. Cheers, -- Luís
