hexops/mach

Fork 0

all: accelerate long-term development #1166

New issue

Closed

opened 2024-02-28 18:37:44 +00:00 by emidoots · 5 comments

emidoots commented

2024-02-28 18:37:44 +00:00

(Migrated from github.com)

Context

I'd personally like to do better about focusing my efforts on higher-level APIs and 'here's what you can actually do using Mach today' use-cases in the future. i.e. I'd like to be able to focus more on our actual Zig code, and want to do better than I did in the v0.3 release.

I've been reflecting a lot on where we are / are not. Part of this is thinking about what types of Mach users exist today:

Those who believe in the overall long-term Mach vision..
a. ..and try to use it today for real projects: ~3 people today
b. ..and 'are waiting for it to be ready': probably sponsors (~40 people) is a decent signal
Those who see value in mach-core but not the rest of Mach: probably <50 people
Those who are just exploring Zig for the first time and looking for e.g. 'GLFW OpenGL Zig' tutorials or such. This is an untold, large portion of people.
Those who are just here for standalone libraries - again an untold portion of people.
Lurkers - folks who are just interested in Zig and keen to check things out, but probably don't plan to actually do anything with it (the overwhelming silent majority of people in the Discord I would say fit into this category.)

Mach is a pretty small project/community today! That's not super surprising - and we have a good trajectory. Still, I think it shows some of the division we have, and development effort is a bit fractured among those contributing to Mach:

Mach core

Mach core has actually seen less adoption/interest overall the past ~year than I thought it might. There are a few reasons for this:

WebGPU is a challenging API to use compared to say the old-school OpenGL API that many prefer to prototype/play with, the same is true of WGSL.
GLFW/SDL have years of momentum behind them, and mach-core doesn't embody our vision for it today (we view it as an actual competitor to the classical GLFW/SDL setup, but today it is in fact a wrapper over GLFW, even if that is changing rapidly.)

Even among those that I think would be a perfect fit for Mach core (people using GLFW and WebGPU), I hear of many preferring to 'I'll just wire them up myself' (possibly because of entrypoint differences on mobile/desktop/web), leading to some amount of support questions to projects that do not e.g. improve Mach or Mach core long-term.

Standalone library questions

Standalone libraries are fine and good, I am glad we offer them. However, it's probably something like 90% of people asking questions and needing support are just random Zig beginners asking basic setup questions around:

The wrong Zig version
How to get ZLS working (wrong version)
Using the wrong Zig version and it leading to weird build errors.
Someone screwing up their Zig package cache somehow
Failure to write build.zig code correctly
etc.

Most questions aren't about using Mach, or using Mach core even - and it takes a good deal of time to support such users.

Upcoming challenges

I think we've always known the following migrations were coming for us, it was always just a question of 'when', 'how', and 'what exactly will it look like?' - and the progress towards these have been substantial the past year:

WebGPU -> sysgpu
WGSL -> Zig shading language
GLFW -> custom windowing backends

Not all of these are in the same state.

sysgpu
- Still has a great deal of bugs and needs polish/testing.
- The Metal backend is largely fine, running all examples.
- The Vulkan and D3D12 backends largely runs all examples fine, but have some serious validation errors we need to investigate/fix.
- The WGSL shader compiler, although surprisingly robust/capable, has many missing features.
- We plan to replace WGSL with Zig shading language, and make major alternations to the public sysgpu API.
- In short, sysgpu is really close - but not there yet.
Zig shading language
- This is still further away, but is surprisingly close to being able to compile e.g. a basic triangle shader, and once it does things will go much quicker after that.
GLFW -> custom windowing backends
- A WIP X11 backend, and fairly solid Wayland backend, are already under way.

None of the above progress is thanks to myself, it's all thanks to temporary and long-term contributors like pdoane, ali, poltixe - people like foxxne trying it on their projects - and others giving advice.

We know these 3 migrations are going to happen at some point, and we know it's not going to be a buttery-smooth 'just upgrade' migration (sysgpu API will be different than WebGPU's, WGSL->Zig will require rewriting your shaders, GLFW -> custom windowing backends will have some major bugs, etc.) It will be painful.

Maintaining both paths is costly

Our dependency tree is not simple today, and that's in part because we increasingly maintain two totally different descriptions of what our graphics API (and in the near future, windowing backends and shading language) is:

Keeping Dawn in a good/solid/up-to-date state a lot of very focused time (I don't think people in general realize how much time I spend keeping Dawn in a nice state for everyone so that eating the Google build system is not necessary.)

With v0.3, we also weren't able to ship with sysgpu enabled even as an option (because we needed some time to sort out optional dependencies for some sysgpu dependencies) - meaning that the folks who really truly embrace where we're going with Mach (those who have embraced sysgpu) haven't been able to build there projects with the v0.3 release. For me, that cuts a bit deep and tells me something is wrong! If someone really embraces where the project is going, they should benefit from latest-and-greatest, not suffer for it.

The decision we face today

As I see it, we have two choices:

Continue down the current path of 'waiting until these projects are "solid enough"' - i.e. wait until we are confident users can upgrade if they put in elbow grease to do so.
Declare bankruptcy: say that we've seen the light of these new approaches, and fully embrace them ASAP.

Proposal

De-emphasize standalone libraries (minor point)

Today the website homepage emphasizes standalone libraries. Pages like https://machengine.org/pkg/ also have 'dedicated docs pages' with getting-started guides, etc. for them. Additionally, we direct people with any problem using a standalone library to our Discord.

My intention would be to de-emphasize standalone libraries on the website, making them more like 'just a GitHub repository with some code'. The goal would be to still allow standalone library usage, of course, but have it be more 'these are just the packages Mach uses' rather than 'these are a main feature of the Mach project'

Sound vague? It is :) The specifics would need to be sorted out a bit. This is the least important part of this proposal.

Aggressively embrace instability

My proposal is to aggressively embrace instability. This will harm us and be very painful in the short term, but will be better long-term.

sysgpu

As mentioned earlier, sysgpu is not stable. We expect it's API and shading language to change in some major ways. However, once it is usable as the only backend by a real-world project (like foxxne's projects, which we're already very close to) then I think that's sufficient to say we can switch to it wholeheartedly - even if we expect major changes in the future.

Zig shading language

This one is tricky. There is an initial ramp, where getting e.g. a basic triangle shader working is going to be a bit challenging. Ali and Robin are working on it, and I suspect we'll have that in the next few months.

Once that happens, i.e. once we have a basic triangle rendering using it, I think we should aggressively start switching to it. This will probably be a somewhat incremental process, where we start switching code over to it with WGSL still temporarily available as an option - before WGSL is completely removed.

Custom windowing backends

In short, my proposal here is the same as the above: as soon as we have keyboard+mouse input, and a triangle on screen, we should switch over to custom backends. Even if there are some major outstanding issues (resizing not functional, can't set the window title, etc.) - effectively embrace instability here.

Again, this will be painful short-term, but will open some very obvious contribution opportunities to those who might wish to contribute - and I think long-term it will be better for effort to go to these problems rather than the now-recognized-legacy systems we have in place today.

What happens to mach-gpu, mach-gpu-dawn?

I'd like to put development effort I would normally put to these towards sysgpu, etc. instead. That means I would officially deprecate and unsupport these packages, they would quickly fall into a state of unusable decay. I doubt anyone else will have the skillset/knowledge/time to pick up maintenance of these, so I fully expect they will die.

What happens to mach-glfw?

This project would move into my personal GitHub account and no longer be a Mach project. I would add a note to the README saying it is community-maintained.

I think there are enough Zig community members using this package to keep it in a good/functional state, but as Mach would no longer be using it this would be very low on my list of priorities.

Feedback & Timeline

I am preparing for the SYCL workshop, so none of this will be decided on until after May 17th at the soonest. That gives us a few months to discuss the tradeoffs of this proposal.

The current mach-core setup, using Dawn+WGSL+GLFW, will continue being supported at least until then.

I understand some of the changes in here will be controversial, and depending on what kind of Mach user you are - possibly quite frustrating! I hope you're able to read it in good faith.

## Context I'd personally like to do better about focusing my efforts on higher-level APIs and 'here's what you can actually do using Mach today' use-cases in the future. i.e. I'd like to be able to focus more on our actual Zig code, and want to do better than I did in the v0.3 release. I've been reflecting a lot on where we are / are not. Part of this is thinking about what types of Mach users exist today: 1. Those who believe in the overall long-term Mach vision.. a. **..and** try to use it today for real projects: ~3 people today b. **..and** 'are waiting for it to be ready': probably sponsors (~40 people) is a decent signal 2. Those who see value in mach-core but not the rest of Mach: probably <50 people 3. Those who are just exploring Zig for the first time and looking for e.g. 'GLFW OpenGL Zig' tutorials or such. This is an untold, large portion of people. 4. Those who are just here for standalone libraries - again an untold portion of people. 5. Lurkers - folks who are just interested in Zig and keen to check things out, but probably don't plan to actually do anything with it (the overwhelming silent majority of people in the Discord I would say fit into this category.) Mach is a pretty small project/community today! That's not super surprising - and we have a good trajectory. Still, I think it shows some of the division we have, and development effort is a bit fractured among those contributing to Mach: ![image](https://github.com/hexops/mach/assets/3173176/210d3e8a-8cf0-4a38-b4e9-e028dd9d823d) ### Mach core Mach core has actually seen less adoption/interest overall the past ~year than I thought it might. There are a few reasons for this: 1. WebGPU is a challenging API to use compared to say the old-school OpenGL API that many prefer to prototype/play with, the same is true of WGSL. 2. GLFW/SDL have years of momentum behind them, and mach-core doesn't embody our vision for it today (we view it as an actual competitor to the classical GLFW/SDL setup, but today it is in fact a wrapper over GLFW, even if that is changing rapidly.) Even among those that I think would be a perfect fit for Mach core (people using GLFW and WebGPU), I hear of many preferring to 'I'll just wire them up myself' (possibly because of entrypoint differences on mobile/desktop/web), leading to some amount of support questions to projects that do not e.g. improve Mach or Mach core long-term. ### Standalone library questions Standalone libraries are fine and good, I am glad we offer them. However, it's probably something like 90% of people asking questions and needing support are just random Zig beginners asking basic setup questions around: * The wrong Zig version * How to get ZLS working (wrong version) * Using the wrong Zig version and it leading to weird build errors. * Someone screwing up their Zig package cache somehow * Failure to write build.zig code correctly * etc. Most questions aren't about using Mach, or using Mach core even - and it takes a good deal of time to support such users. ### Upcoming challenges I think we've always known the following migrations were coming for us, it was always just a question of 'when', 'how', and 'what exactly will it look like?' - and the progress towards these have been substantial the past year: * WebGPU -> sysgpu * WGSL -> Zig shading language * GLFW -> custom windowing backends Not all of these are in the same state. * sysgpu * Still has a great deal of bugs and needs polish/testing. * The Metal backend is largely fine, running all examples. * The Vulkan and D3D12 backends largely runs all examples fine, but have some serious validation errors we need to investigate/fix. * The WGSL shader compiler, although surprisingly robust/capable, has many missing features. * We plan to replace WGSL with Zig shading language, and make major alternations to the public sysgpu API. * In short, sysgpu is really close - but not there yet. * Zig shading language * This is still further away, but is surprisingly close to being able to compile e.g. a basic triangle shader, and once it does things will go much quicker after that. * GLFW -> custom windowing backends * A WIP X11 backend, and fairly solid Wayland backend, are already under way. None of the above progress is thanks to myself, it's all thanks to temporary and long-term contributors like pdoane, ali, poltixe - people like foxxne trying it on their projects - and others giving advice. We know these 3 migrations are going to happen _at some point_, and we know it's _not_ going to be a buttery-smooth 'just upgrade' migration (sysgpu API will be different than WebGPU's, WGSL->Zig will require rewriting your shaders, GLFW -> custom windowing backends will have some major bugs, etc.) It will be painful. ### Maintaining both paths is costly Our dependency tree is not simple today, and that's in part because we increasingly maintain two totally different descriptions of what our graphics API (and in the near future, windowing backends and shading language) is: [![image](https://github.com/hexops/mach/assets/3173176/9dfcd103-43ce-4cf3-9f84-ecbaed8bb1b9)](https://github.com/hexops/mach/assets/3173176/9dfcd103-43ce-4cf3-9f84-ecbaed8bb1b9) Keeping Dawn in a good/solid/up-to-date state _a lot of very focused time_ (I don't think people in general realize how much time I spend keeping Dawn in a nice state for everyone so that eating the Google build system is not necessary.) With v0.3, we also weren't able to ship with sysgpu enabled _even as an option_ (because we needed some time to sort out optional dependencies for some sysgpu dependencies) - meaning that the folks who really truly embrace where we're going with Mach (those who have embraced sysgpu) haven't been able to build there projects with the v0.3 release. For me, that cuts a bit deep and tells me something is wrong! If someone really embraces where the project is going, they should benefit from latest-and-greatest, not suffer for it. ### The decision we face today As I see it, we have two choices: 1. Continue down the current path of 'waiting until these projects are "solid enough"' - i.e. wait until we are confident users _can_ upgrade if they put in elbow grease to do so. 2. Declare bankruptcy: say that we've seen the light of these new approaches, and fully embrace them ASAP. ## Proposal ## De-emphasize standalone libraries (minor point) Today the [website homepage](https://machengine.org/) emphasizes standalone libraries. Pages like https://machengine.org/pkg/ also have 'dedicated docs pages' with getting-started guides, etc. for them. Additionally, we direct people with any problem using a standalone library to our Discord. My intention would be to de-emphasize standalone libraries on the website, making them more like 'just a GitHub repository with some code'. The goal would be to still allow standalone library usage, of course, but have it be more 'these are just the packages Mach uses' rather than 'these are a main feature of the Mach project' Sound vague? It is :) The specifics would need to be sorted out a bit. This is the _least_ important part of this proposal. ## Aggressively embrace instability My proposal is to aggressively embrace instability. This will harm us and be very painful in the short term, but will be better long-term. ### sysgpu As mentioned earlier, sysgpu is not stable. We expect it's API and shading language to change in some major ways. However, once it is usable as the only backend by a real-world project (like foxxne's projects, which we're already very close to) then I think that's sufficient to say we can switch to it wholeheartedly - even if we expect major changes in the future. ### Zig shading language This one is tricky. There is an initial ramp, where getting e.g. a basic triangle shader working is going to be a bit challenging. Ali and Robin are working on it, and I suspect we'll have that in the next few months. Once that happens, i.e. once we have a basic triangle rendering using it, I think we should aggressively start switching to it. This will probably be a somewhat incremental process, where we start switching code over to it with WGSL still temporarily available as an option - before WGSL is completely removed. ### Custom windowing backends In short, my proposal here is the same as the above: as soon as we have keyboard+mouse input, and a triangle on screen, we should switch over to custom backends. Even if there are some major outstanding issues (resizing not functional, can't set the window title, etc.) - effectively embrace instability here. Again, this will be painful short-term, but will open some very obvious contribution opportunities to those who might wish to contribute - and I think long-term it will be better for effort to go to these problems rather than the now-recognized-legacy systems we have in place today. ### What happens to mach-gpu, mach-gpu-dawn? I'd like to put development effort I would normally put to these towards sysgpu, etc. instead. That means I would officially deprecate and unsupport these packages, they would quickly fall into a state of unusable decay. I doubt anyone else will have the skillset/knowledge/time to pick up maintenance of these, so I fully expect they will die. ### What happens to mach-glfw? This project would move into my personal GitHub account and no longer be a Mach project. I would add a note to the README saying it is community-maintained. I think there are enough Zig community members using this package to keep it in a good/functional state, but as Mach would no longer be using it this would be very low on my list of priorities. ## Feedback & Timeline I am preparing for the SYCL workshop, so none of this will be decided on until after May 17th at the soonest. That gives us a few months to discuss the tradeoffs of this proposal. The current mach-core setup, using Dawn+WGSL+GLFW, will continue being supported at least until then. I understand some of the changes in here will be controversial, and depending on what kind of Mach user you are - possibly quite frustrating! I hope you're able to read it in good faith.

❤️ 42

emidoots commented

2024-06-02 16:44:29 +00:00

(Migrated from github.com)

Progress towards repository reductions:

mach-model3d has been removed as an official Mach project: https://github.com/hexops/mach/issues/969
mach-sysjs lives in the main repository https://github.com/hexops/mach/tree/main/src/sysjs
mach-editor repository has been archived; this will come back in a different form later.
mach-objc-generator lives in the main repository https://github.com/hexops/mach/tree/main/src/mach/objc-generator
mach-core-starter-project has been deprecated in favor of a simpler build API and docs on https://machengine.org/core/getting-started/
basisu, mach-basisu -> https://github.com/hexops/mach/issues/965#issuecomment-2143986899

Progress towards repository reductions: * [x] `mach-model3d` has been removed as an official Mach project: https://github.com/hexops/mach/issues/969 * [x] `mach-sysjs` lives in the main repository https://github.com/hexops/mach/tree/main/src/sysjs * [x] `mach-editor` repository has been archived; this will come back in a different form later. * [x] `mach-objc-generator` lives in the main repository https://github.com/hexops/mach/tree/main/src/mach/objc-generator * [x] `mach-core-starter-project` has been deprecated in favor of a simpler build API and docs on https://machengine.org/core/getting-started/ * [x] `basisu`, `mach-basisu` -> https://github.com/hexops/mach/issues/965#issuecomment-2143986899

emidoots commented

2024-07-08 03:58:13 +00:00

(Migrated from github.com)

mach-glfw has now been moved to 'community maintained' status and now lives at https://github.com/slimsag/mach-glfw and https://github.com/slimsag/glfw

emidoots commented

2024-07-08 04:14:12 +00:00

(Migrated from github.com)

These projects have been deprecated / moved / are no longer maintained by the mach project:

These projects have been deprecated / moved / are no longer maintained by the mach project: * https://github.com/hexops-graveyard/mach-gpu * https://github.com/hexops-graveyard/mach-gpu-dawn * https://github.com/hexops-graveyard/dawn

PaperPrototype commented

2024-07-13 13:09:17 +00:00

(Migrated from github.com)

Sad to see no one commented on this issue, I love the plan here!
I just wanted to say thank you on behalf of the ProwlEngine team, your work with DXC is going to save us a lot of time, thank you!

Sad to see no one commented on this issue, I love the plan here! I just wanted to say thank you on behalf of the ProwlEngine team, your work with DXC is going to save us a lot of time, thank you!

YiraSan commented

2024-07-16 09:56:08 +00:00

(Migrated from github.com)

Just read those proposals and I love them!!! Thank you all contributors, i've effectively been the one "who's waiting for it to be ready" and now seeing all of that makes me wondering if I could maybe try to contribute at my level.