proposal: Resource management system #357
Labels
No labels
CI
all
basisu
blog
bug
build
contributor-friendly
core
correctness
deferred
dev
direct3d-headers
docs
driver-os-issue
duplicate
dxcompiler
editor
examples
experiment
feature-idea
feedback
flac
freetype
gamemode
gkurve
glfw
gpu
gpu-dawn
harfbuzz
help welcome
in-progress
infrastructure
invalid
libmach
linux-audio-headers
long-term
mach
mach.gfx
mach.math
mach.physics
mach.testing
model3d
needs-triage
object
opengl-headers
opus
os/linux
os/macos
os/wasm
os/windows
package-manager
priority
proposal
proposal-accepted
question
roadmap
slipped
stability
sysaudio
sysgpu
sysjs
validating-fix
vulkan-zig-generated
wayland-headers
website
wontfix
wrench
www
x11-headers
xcode-frameworks
zig-update
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
hexops/mach#357
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
General purpose resource management engine which is meant to be cross platform (works on desktop, wasm, android etc.)
The general idea is that in order to access resources, we will use URIs instead of path. All resources will be accessed from the provided API and std.fs.* stuffs should not be used (but the library may use it internally).
The URI will internally evaluate to a path (for desktops) or to an URL (for web). On desktops it will use std.fs APIs and on web it will stream resources which are served on your web server.
Mechanism
Let's say the projects needs a.png, b.png and c.mp3. Then the resources have to be arranged in your directory as such:
This arrangement can be done automatically by some code, which takes a config file (json/yaml/toml) as input and sort the files by their names. The file can look like as such (using json for example):
The resources arrangement/installation can be part of a build step for seamless building. This step may just install or optionally generate an archive file from all the resources.
The resources will be referred inside the application as such:
textures/a.png,textures/b.png,audio/c.mp3. But why notdata/textures/a.png? (Answered below in API)Remember that the names
textures,audioetc above are totally cosmetic and has no significance. It can be arbitrarily named anything. [1]API
The library will provide the following API:
Open Questions
[]const ResourceTypebe taken as comptime parameter? as it is known what type of resources the application will use beforehand.Future TODOs
EDIT: Oversights
Note: In all instances where multi-threading is mentioned, its about loading resources in parallel and not about the thread safety of the API.
I like the design overall. Thanks for starting this!
Questions
When can
loadCollectionbe called? It seems to returnvoidso presumably it must be called before any other methods?If I understand correctly,
ResourceGroupcould be used to describe a scene for example. In this context: Let's say scene A depends on textures (a, b, c) and you want to transition to scene B which requires textures (b, c, d) - ideally (b, c) are not unloaded, while (a) is unloaded and (d) is now loaded. Does theloadResourceandpopGroupAPI support this? (it's unclear to me)In games there is often a need to load resources which are used to compute/build something else. For example, you might load a PNG texture into memory in order to upload it and create a
gpu.Texture. In this situation, you ideally do not want the PNG texture to remain in memory after upload to the GPU finishes on a separate thread. Additionally, if the GPU device context is lost (can happen at any point on mobile and web), the GPU may just "lose all data entirely" and we'd need to be able to handle this event by loading such resources from disk (PNG) and uploading them to the GPU again. How can this system support this?How would a resource loader for textures get access to the GPU, say to upload the texture to the GPU? Would the resource manager be responsible for that, or does it merely handle loading of resources from disk?
Have you given thought to what custom resources might look like? For example, I can imagine some applications needing to define their own resource formats like Minecraft block chunks which are expensive to compute/produce, require some information from the game (ECS), and get saved in a custom binary format they define.
Can you give some real-world examples of how you envision "Collections" being used? I see
texturesandaudioused as examples right now, it's not fully clear to me what benefits collections give usThoughts
For things like this, I suspect it is mostly useful in the context of Mach only, and should be deeply integrated. So this can live in a library at
src/resourceperhaps, and accessible via@import("mach").resource. What do you think?I'll have more thoughts on other questions you posed once I learn a bit more from your responses to my questions I think.
For the sake of explaining, lets just assume that Collection is a single archive. Actually having multiple collections only makes sense in case of archives and thats why its there. But it can be used with local directory based structures. Lets take an example, you have a game and its data is contained in a archive called
gamedata.res. Since the assets are isolated you can easily havegamedataas a collection. Take another case in which game directory istextures.res audio.res some.dll game.exe. In this case you need to have different collections (textures and audio). Note that this can be a deliberate design decision. Take for instance its a large game and storing all resources in separate archives makes more sense. This is answer to your last question. (Q6)Do note that you do not need to have multiple collections in your game. You can perfectly store all resources in one collection/archive (but it may not scale well for large application organization). A collection itself internally follow tree structure, so it can have directories inside.
loadCollection should ideally be called before you create any groups. Collections aren't meant to be different for different scenes/chunks. That would be very inefficient. So you should call it just after ResourceManager.init(). If youre thinking about just putting it in init() as a param, it can work I guess.
(it's unclear to me)Thats exactly how it will work. ResourceGroups themselves don't store resources. They just ask the engine that they want to use it. loadResource and popGroup do take care about this. (But now that I m thinking, naming should not be pushGroup and popGroup. Since the system can get more complicated as you load resources for a different scene when the current scene is currently running/about to end)While preparing the ResourceManager, you pass it some callbacks (load() and unload()) which will provide you with the raw png data and you are free to use this data inside the function to generate a gpu.Texture. One small oversight here was that we also need to pass in some context (an additional
context: *anyopaqueparam).To handle, reloads, I think there should be a function to force reload a resource (i.e just reload it even if its already loaded, free and discard previously present data). This action can be signaled by the ECS. The question now is API: a function like
fn ResourceGroup.reloadResourcesimilar to howloadResourcelooks (i.e individual to each resource) or a more generalfn ResourceGroup.reloadResourceType(resource_type_name: []const u8) !voidAnswer in last paragraph with context. But I am open to better suggestions.
Techincally speaking, all resources are custom in the eyes of this system. The system just loads a chunk of data (from a files/files), pass it to your provided callback (fn load()) and stores whatever it returns. So I m unsure how anything would be different for the case you mentioned. All ecs data and such can be easily accessed with the context parameter. How the file is structured is not a problem of this system. You should have your custom functions to parse that file.
Answered in first para.
Sounds right. I didnt had any opinion on this so decided its better to ask, just in case.
The json collection example I initially created didn't take into consideration having multiple collections (in fact it used the word collection for just any folder). So here's an updated one:
Json is not good for manually creating trees.
OK this makes a lot of sense.
data/and it findsdata/textures,data/audioand produces the config file for those two collections?texturesandaudioare only examples, in reality you'd most likely have a single collection for all assets in your game, except in cases like:Yes and no. I think its better to say that its optional. So if someone wants, they can provide a list and structure of resources. What are the advantages? Well I think it would make sense in case of editor where you are too busy so you just throw your .psd files beside the exported .tga but selectively only import the .tga file in your editor asset menu.
The one I mentioned in the matrix chat was an unrelated feature which I didnt added to this proposal because I m not sure yet. See, archives would need to maintain a record of what files they have anyways. For plain directories, I dont think its much needed. The system can just error.FileNotFound. Generating the structure of collection wont give us any advantage because we cant validate what assets will be loaded in future at comptime, nor can we validate their types.
What I was actually referring to on matrix is a way to automatically generate the bunch of
loadResourcefunction calls instead of manually typing it one by one. The problem is that we need to provide a list of resources which are going to be used in that scene. I m not sure if this is a good idea. Ofcourse this will be completely optional, so with editors we can just do it easily. This has the additional disadvantage that it can break the resource streaming system. So I m unsure until we come up with a decent plan. In either case I think its better to push this to a future proposal/plan since its just a convenience function and not a functional one.Yes
root. See above.Collections is the abstraction / part of this proposal I feel least confident about. I think it enforces a certain way of working with your game data that may not be very clear, and the benefits are not always obvious.
Being very critical of it, most of what it solves can be resolved in other ways:
Modding
I think the best-case scenario for modding would be "Here's my mod directory/archive, it wants to override very specific resources, oh and i might have a few others of those.. just use a.png from one of the mod folders if you find it there first, otherwise fallback to the game's a.png file"
Collections don't seem like they would do this at all: they would only let us override specific resources, and only in aggregate. If you wanted to override a few audio files and texture files in a game, for example, you'd need to provide an entirely new
audioandtexturecollection with all game files in it, rather than just overriding the files you want to modify. Additionally, it's not clear that this system could support multiple mods wanting to override multiple different files.Chunking
We had discussed this benefit of collections:
But, actually I think it's not a big benefit necessarily. A single file can manage all assets in a reasonable way, so long as the file format of that file is reasonable. A good example of this is Guild Wars, where both the original game and newer version 2 game are distributed as a single exe file which downloads a single
gw2.datfile with all files in it. As you navigate the game and need new content, it updates that.datfile with more assets.Excluding of certain files
We could support this with a
.gitignore-type file easily.Suggestion: update the proposal with a solidified set of goals and non-goals. We could start with this set:
Goals
Non-goals (to be handled at a later date)
Suggestion: We could remove the idea of collections entirely, and instead add explicit support to the proposal for exclusion of assets and modding:
In order to access a resource, you use a URI instead of a file path:
data://textures/a.pngdata://audio/c.mp3data://junk.txtDepending on where the application is running, and in what mode (release/debug), behavior will differ by default (but you can choose):
std.fs.*APIs will be used to access a single-file archivedata.resfor example, which contains all assets for the game packed into a single file. The file will contain a header which describes where to locate files within the single-file archive, so we can e.g. seek to a specific file in the archive to read it. We can chat more about the specifics of this file format, but I think it's safe to assume in general we can come up with a good way to pack multiple files into a single one and manage that in a way that is performant and can support incremental updates (adding new files, updating existing ones, deleting ones, etc.)data.resfile is produced from your game'sdata/directory viabuild.zigat build time. The only constraint the system poses is that you provide a single directory where your assets will live.Rangerequests to query byte ranges of thedata.resfile. Works in the same way as file seeking natively, effectively.data/directory (this enables swapping out assets at runtime without rebuilding/updating thedata.resarchive file.)Exclusion of assets
Generally speaking, you put all game assets under a folder called
data/. In some cases, it may make sense to have files you want to live alongside your game assets such as.psdor.blendfiles excluded from being included in your finaldata.resarchive. There will thus be a way (TBD, maybe similar to.gitignore, maybe via build.zig options, maybe something else) to exclude files using patterns.When excluded, they will not end up in the final
data.resand will also not be accessible via the API in debug builds either (to prevent accidentally relying on assets which get excluded in release builds.)Modding
To enable resource modding of Mach games/applications generally, the following will occur:
If running natively (not supported in wasm for now), then a
modsfolder can live alongsidedata.res:game.exedata.resmods/mytexturepack.resnewmod/a.pngmods/can either be a.resfile (same format asdata.res), or just plain directories (newmod/).When loading a file, say
a.png, first each mod is checked in alphanumeric order for ana.pngfile to override the game's resource with. If none is found, thena.pngis loaded fromdata.res.Problem: I think the scoping logic may not be sufficient, OR I don't exactly understand how it should work. I see a few use cases we should support with scopes:
How could the API support all 3?
Answers:
I agree, not needed. Detecting if a file is binary (and what that actually means) is notoriously difficult/annoying.
We need a way to register
loadfunctions, right? As in, "here are the bytes of the file, now turn it into a type T(like PNG ->gpu.Texture`) - but I guess even if we had a bunch of these functions registered, we also don't know based on a given file/bytes, which one to call either.We could require that such a function be provided to the
getResourcefunction (so you pass it the function that knows how to turn PNG bytes ->gpu.Texturewith context.) That's the first thing that comes to mind for me, and doesn't seem too bad. Thoughts?I think file extension-based would be bad, because some resources with the same extension need to be interpreted differently (e.g.
.pngcould be agpu.Texture, or it could really be a PNG image someone wants to load and handle themselves (such as for a heightmap, or to do something else funky with.) Similarly,.jsonfiles might go into different data structures)This may be quite important to sort out.
Agreed.
I think we've gotten all of the major discussion points out on the table, so we can do one of two things (whatever you're comfortable with):
I'm OK with either at this point, I don't want to place a burden of writing out more stuff here on you just for the sake of it.
1 and 2 is solved by the function
ResourceManager.prepareGroup(). With this you register what resources are needed for the upcoming scene. This function is called when youre about to end the current scene. The whole system will work in a different thread, so this function will not block. When you do popGroup() on the current group, the system already has one more group on top of this, so it will be careful what to remove.Do note that pushGroup and popGroup are slightly misleading names as mentioned earlier.
3 ) I dont know if i understand it correctly. Is it about being able to individually load any resource? Well then it will be covered with the low level API. But let's say if the resource is a registered one, maybe we can provide an additional function to directly load with the URI?
Alright, I agree with everything here
I in general like this part but would like to propose minor changes. Let the resource manager initializer take a list of paths. It will try to load the resource from 1st, if not present then try in 2nd and so on. The actual resource archive being last in this list. This is similar to how
PATHenv var works. The reason for this proposal is that I think we shouldn't dictate what installing mods should look like to the end application, so basically it can just disallow mods if it wants.The job to convert PNG bytes -> gpu.Texture and so one is performed automatically behind the scenes by the load() function which we provide in ResourceManager.init. What I think can be done here that let say if load() function returns a particular error like
error.IncorrectResourceTypethen it will move on and try using a different resource loader. Plus a function calledcheckMagic()can be added besides load() which checks file magic to figure out the type, if it returns false, try with the next type and so on. This is a trick used in SDL's helper libraries.Yes, and sounds good. The point is just being able to load/free resources manually, without groups (think basically "I want to implement my own grouping logic on top, can I?")
The problem with this is we can't handle resources in different ways. Let's say my application needs to do two things:
gpu.Texturesgpu.Textureand I don't want it to be uploaded to the GPU anyway.We need some sort of way to support "handle the same resource type in different ways" I think.
The problem is more noticeable when you talk about e.g. loading
.jsonfiles: you don't want one genericloadfunction for this, you want to be able to handle JSON decoding based on say the resource URI using different functions.Can we use the URI scheme for this, since we aren't using it for anything else right now? Like
texture://images/player.pngandsheet://images/player_anim.pngwhere "texture" and "sheet" must be the name of any one ResourceType we added in ResourceManager.init().That sounds like a great solution! Then we don't have to rely on extension or
checkMagiceither!In the future (not part of library), we could also allow for loader functions to be defined as part of
mach.Modules, specifying a list of URI schemes and loader functions that implement that.I have some audio-centric questions that I didn't see answered.
It's very difficult to ensure stutter-free streaming, since you can't predict where the file comes from.
If a user has the game stored on a slow HDD, it may not be able to keep up. Similarly for a WASM game trying to stream over a slow mobile data connection.
Right, those are factors that can affect streaming. Obviously it would be impossible to guarantee stutter-free streaming in those situations. There are plenty of other places where it is possible to stream the audio - consoles, most modern computers, phones (I assume? I actually don't know how fast storage is on mobile), etc. I guess the question will then come down each individual game, and what your target audience is. Do you want to target older computers to increase your possible reach, or are you wanting to push the limits of modern hardware, or something in-between?
I mainly wanted to know if that use case had been planned for or bring attention to the possibility if it hadn't.