mach: Add a cross-platform filesystem layer #906

New issue

Open

opened 2023-07-31 20:48:10 +00:00 by spindlebink · 1 comment

spindlebink commented

2023-07-31 20:48:10 +00:00

(Migrated from github.com)

Prior to working on a full asset pipeline (to be discussed), Mach needs an interface for loading file streams from a project. I'm most familiar with PhysFS as an off-the-shelf solution, but per discussion in https://github.com/hexops/mach/issues/864, we'd want a Mach solution to be a fresh project written in Zig. I'm happy to get started here. Loading game resources is a foundational part of any engine, so the sooner there's a way to do it, the better.

The design I had in mind (feedback would be very helpful) involves a file "server" of sorts which depends on a list of asset sources. When you request a file from the server, it polls each source in turn to see if it can find the appropriate file. If it can't, it moves to the next in the list; otherwise, it opens the data as a stream and returns it. PhysFS does something similar: you can "mount" directories or archives into the PhysFS instance, then when looking for a file, the filesystem polls each one.

fs.addSource(FallbackSource.init(...)) // etc., naming not the point
fs.addSource(ArchiveSource.init(...))
fs.addSource(ArchiveSource.init(...)) // Assets from this archive overwrite those from the previous one, since they're polled first

Doing it this way would enable developers to read from multiple archive sources (https://github.com/hexops/mach/issues/864) or e.g. the executable base directory at runtime, and it would also make polling for mods that overwrite assets transparent. In extreme (read: non MVP) cases, this approach could even be used as a means to download game assets as-needed, with an asset source pointing to the project's save directory and another higher up to retrieve assets from a server if they're not yet cached.

Concerns:

This layer should be implemented on top of the standard library's IO, such that getting a file handle from the Mach filesystem should return an io.StreamSource (or equivalent; I need to do some research here)
Loading resources should be distinct from reading file contents: the project in this issue should be specifically how to get access to file streams (= search paths and sources, not an asset pipeline) cross-platform.
There should be as little user-side platform-specific stuff as necessary. The filesystem should work transparently on web and desktop.

Other projects that I'm planning on looking into for inspiration:

PhysFS
Bevy's AssetServer+AssetIo combination--it implements this approach exactly as far as I can tell
Godot's ResourceLoader

Related: https://github.com/hexops/mach/issues/357, https://github.com/hexops/mach/issues/864 Prior to working on a full asset pipeline (to be discussed), Mach needs an interface for loading file streams from a project. I'm most familiar with PhysFS as an off-the-shelf solution, but per discussion in https://github.com/hexops/mach/issues/864, we'd want a Mach solution to be a fresh project written in Zig. I'm happy to get started here. Loading game resources is a foundational part of any engine, so the sooner there's a way to do it, the better. The design I had in mind (feedback would be very helpful) involves a file "server" of sorts which depends on a list of asset sources. When you request a file from the server, it polls each source in turn to see if it can find the appropriate file. If it can't, it moves to the next in the list; otherwise, it opens the data as a stream and returns it. PhysFS does something similar: you can "mount" directories or archives into the PhysFS instance, then when looking for a file, the filesystem polls each one. ```zig fs.addSource(FallbackSource.init(...)) // etc., naming not the point fs.addSource(ArchiveSource.init(...)) fs.addSource(ArchiveSource.init(...)) // Assets from this archive overwrite those from the previous one, since they're polled first ``` Doing it this way would enable developers to read from multiple archive sources (https://github.com/hexops/mach/issues/864) or e.g. the executable base directory at runtime, and it would also make polling for mods that overwrite assets transparent. In extreme (read: non MVP) cases, this approach could even be used as a means to download game assets as-needed, with an asset source pointing to the project's save directory and another higher up to retrieve assets from a server if they're not yet cached. Concerns: * This layer should be implemented on top of the standard library's IO, such that getting a file handle from the Mach filesystem should return an `io.StreamSource` (or equivalent; I need to do some research here) * Loading resources should be distinct from reading file contents: the project in this issue should be specifically how to *get access to file streams* (= search paths and sources, not an asset pipeline) cross-platform. * There should be as little user-side platform-specific stuff as necessary. The filesystem should work transparently on web and desktop. Other projects that I'm planning on looking into for inspiration: * PhysFS * Bevy's AssetServer+AssetIo combination--it implements this approach exactly as far as I can tell * Godot's ResourceLoader

spindlebink commented

2023-09-08 01:48:18 +00:00

(Migrated from github.com)

Relevant: Bevy just merged this massive rewrite of their asset system that we could probably learn a lot from. https://github.com/bevyengine/bevy/pull/8624