hexops/mach

Fork 0

libmach: Compile to shared library? #391

New issue

Closed

opened 2022-07-06 23:53:47 +00:00 by zack466 · 8 comments

zack466 commented

2022-07-06 23:53:47 +00:00

(Migrated from github.com)

Firstly, I want to say that this is a great project, and I really appreciate the effort being put into a truly cross platform graphics toolkit. Are there any plans (or is it even possible) to compile mach into a shared library? I think this would allow for easy integration with any programming language that supports a C FFI.

emidoots commented

2022-07-09 22:14:01 +00:00

(Migrated from github.com)

I've thought about this a bit, I think it would be good to have something like this, yes.

Supporting a broad range of bindings in other languages would be an explicit non-goal (i.e., you'd be on your own, because we just don't have time to support that) but I think that having a well-defined C ABI could be a good way to enable adoption of Mach from other languages.

I also think that having an option to script Mach using WASM modules (so, whichever languages can compile to WASM and use the Mach API through that) is compelling to me.

How this would look exactly? That's a bit up in the air. Since Mach is composed of a bunch of smaller projects, and isn't one big API, I think what we would need to do is create a subproject (let's call it libmach) which defines a C ABI, and imports all of the standard Mach libraries in order to implement it.

How far we go with it, what gets supported vs. what does not.. that gets trickier. I don't know. I think ultimately, the thought here would be: is someone interested in contributing/maintaining such a library?

I've thought about this a bit, I think it would be good to have something like this, yes. Supporting a broad range of bindings in other languages would be an explicit non-goal (i.e., you'd be on your own, because we just don't have time to support that) but I think that having a well-defined C ABI could be a good way to enable adoption of Mach from other languages. I also think that having an option to script Mach using WASM modules (so, whichever languages can compile to WASM and use the Mach API through that) is compelling to me. How this would look exactly? That's a bit up in the air. Since Mach is composed of a bunch of smaller projects, and isn't one big API, I think what we would need to do is create a subproject (let's call it `libmach`) which defines a C ABI, and imports all of the standard Mach libraries in order to implement it. How far we go with it, what gets supported vs. what does not.. that gets trickier. I don't know. I think ultimately, the thought here would be: is someone interested in contributing/maintaining such a library?

zack466 commented

2022-07-11 04:16:25 +00:00

(Migrated from github.com)

I'd be willing to contribute.

I made a hacky proof of concept here. If you run zig build, cd into libmach, then run make run, it should run the boids example. All it took was adding src/exports.zig and then generating a shared library in build.zig. I think that exporting mach internals should be relatively easy.

However, I'm not exactly sure how exactly the mach API should be exposed. From what I can tell, mach currently requires a Zig package called "app" to be linked at compile time, but I don't think this is desirable in the context of dynamic linking. Exposing the high-level API would probably require App to be more decoupled from mach internals. Otherwise, I don't think runtime linking would really make any sense (unless only low-level API functions are exposed?).

Also, Zig currently can't automatically emit C header files (but it should be supported by stage 2), so those would have to be written by hand for the moment.

edit: oops, for any future readers, the above link is no longer accurate (thanks, Mr.git push --force)

I'd be willing to contribute. I made a hacky proof of concept [here](https://github.com/zack466/mach/tree/libmach). If you run `zig build`, `cd` into `libmach`, then run `make run`, it should run the boids example. All it took was adding `src/exports.zig` and then generating a shared library in `build.zig`. I think that exporting mach internals should be relatively easy. However, I'm not exactly sure how exactly the mach API should be exposed. From what I can tell, mach currently requires a Zig package called "app" to be linked at compile time, but I don't think this is desirable in the context of dynamic linking. Exposing the high-level API would probably require `App` to be more decoupled from mach internals. Otherwise, I don't think runtime linking would really make any sense (unless only low-level API functions are exposed?). Also, Zig currently can't automatically emit C header files (but it should be supported by stage 2), so those would have to be written by hand for the moment. edit: oops, for any future readers, the above link is no longer accurate (thanks, Mr.`git push --force`)

emidoots commented

2022-07-12 14:29:06 +00:00

(Migrated from github.com)

cool!

The reason Mach requires an App is because we aim to support browser and android/ios. In these, you don't really get a main function - so the standard idea of "I will call into the library" doesn't really work. Instead, you get callbacks (basically) where your app can do things, render, etc. App reflects this constraint, I don't think we can change it.

So there are a few ways we can do this for libmach..

We could copy the ECS app, like this, and define an app that has no ECS modules at all (they'll all have to be added at runtime via libmach API). In this model, when you link against libmach it would have it's own pub fn init(engine: *ecs.World(modules)) !void { which calls your exported mach_init callback. Therefor, your C code must export such a function so it can call it.
- Benefit: It works on (at least) Desktop, Android, and iOS. Might work on WebAssembly too.
- Benefit: Supports Mach engine apps ("I want everything")
- Con: Doesn't support Mach core apps ("I want a window, input, and WebGPU API. That's it")
We could define an app that exposes init, deinit and update functions similar to the triangle example but again, just have them directly call a C function mach_init, mach_deinit, mach_update and expect that those are defined by the user linking libmach into their program.
- Benefit: It works on (at least) Desktop, Android, and iOS. Might work on WebAssembly too.
- Benefit: Supports Mach core apps ("I want a window, input, and WebGPU API. That's it")
- Con: Doesn't support Mach engine apps ("I want everything")
We could say "Truly? I don't care about iOS/Android/WebAssembly." - In this model, we could define a custom platform type for "library" mode, in which main is not defined for you but rather it is exported as mach_main or similar.
- Benefit: Supports both Mach engine and Mach core apps.
- Con: Can't ever work on Android/iOS/WebAssembly.

My advice would be that we go with both #1 and #2, naming them libmachengine and libmachcore. Depending on which one you link against, you choose whether it expects to find just a single init function (Mach engine apps), or init, deinit and update (Mach core apps.)

Then the rest of this is just sorting out how to write/expose C ABIs for Mach APIs. Shouldn't be too hard, but definitely some work to do there. Stage2 will help with headers in the future as you mentioned.

cool! The reason Mach requires an App is because we aim to support browser and android/ios. In these, you don't really get a main function - so the standard idea of "I will call into the library" doesn't really work. Instead, you get callbacks (basically) where your app can do things, render, etc. App reflects this constraint, I don't think we can change it. So there are a few ways we can do this for `libmach`.. 1. We could copy the ECS app, [like this](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/examples/ecs-app/main.zig#L33), and define an app that has no ECS modules at all (they'll all have to be added at runtime via `libmach` API). In this model, when you link against libmach it would have it's own `pub fn init(engine: *ecs.World(modules)) !void {` which calls **your** exported `mach_init` callback. Therefor, your C code must export such a function so it can call it. * Benefit: It works on (at least) Desktop, Android, and iOS. Might work on WebAssembly too. * Benefit: Supports _Mach engine_ apps ("I want everything") * Con: Doesn't support _Mach core_ apps ("I want a window, input, and WebGPU API. That's it") 2. We could define an app that exposes `init`, `deinit` and `update` functions similar to the [triangle example](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/examples/triangle/main.zig#L10) but again, just have them directly call a C function `mach_init`, `mach_deinit`, `mach_update` and expect that those are defined by the user linking `libmach` into their program. * Benefit: It works on (at least) Desktop, Android, and iOS. Might work on WebAssembly too. * Benefit: Supports _Mach core_ apps ("I want a window, input, and WebGPU API. That's it") * Con: Doesn't support _Mach engine_ apps ("I want everything") 3. We could say "Truly? I don't care about iOS/Android/WebAssembly." - In this model, we could define a custom platform type for "library" mode, in which [`main` is not defined for you](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/src/platform/native.zig#L590-L646) but rather it is exported as `mach_main` or similar. * Benefit: Supports both _Mach engine_ and _Mach core_ apps. * Con: Can't ever work on Android/iOS/WebAssembly. My advice would be that we go with both #1 and #2, naming them `libmachengine` and `libmachcore`. Depending on which one you link against, you choose whether it expects to find just a single `init` function (_Mach engine_ apps), or `init`, `deinit` and `update` (_Mach core_ apps.) Then the rest of this is just sorting out how to write/expose C ABIs for Mach APIs. Shouldn't be too hard, but definitely some work to do there. Stage2 will help with headers in the future as you mentioned.

zack466 commented

2022-07-12 22:59:14 +00:00

(Migrated from github.com)

Ok, thanks for the explanation.

However, I'm a little confused. When you say "C code exporting a function", what exactly do you mean? Do you mean that a C file can define something like extern void mach_init() and then libmach will figure out the right function(s) to call through some linker magic? (I haven't been able to get this to work yet). Or do you mean having C code explicitly pass function pointer(s) to libmach at runtime to use as callbacks?

I was thinking the second approach makes more sense, since then other languages with a C FFI could load libmach and then provide their own callback function(s) for libmach to call in its internal core loop. That's the sort of use case I was envisioning (similar to libraries like Raylib, but you use mach's core loop instead of your own). But would this satisfy the constraints of iOS/Android/WASM?

If we go with the first approach instead, then I just don't really see how C bindings would be all that useful - I thought the whole point was so that Mach could sort of be used like a library from multiple languages. Is there something that I'm missing?

Ok, thanks for the explanation. However, I'm a little confused. When you say "C code exporting a function", what exactly do you mean? Do you mean that a C file can define something like `extern void mach_init()` and then `libmach` will figure out the right function(s) to call through some linker magic? (I haven't been able to get this to work yet). Or do you mean having C code explicitly pass function pointer(s) to `libmach` at runtime to use as callbacks? I was thinking the second approach makes more sense, since then other languages with a C FFI could load `libmach` and then provide their own callback function(s) for `libmach` to call in its internal core loop. That's the sort of use case I was envisioning (similar to libraries like Raylib, but you use mach's core loop instead of your own). But would this satisfy the constraints of iOS/Android/WASM? If we go with the first approach instead, then I just don't really see how C bindings would be all that useful - I thought the whole point was so that Mach could sort of be used like a library from multiple languages. Is there something that I'm missing?

emidoots commented

2022-07-13 00:07:34 +00:00

(Migrated from github.com)

OK, so I think what we'll need to do in order to support this then is something like this:

libmach (single library)
Rename platform/native.zig to something like native_common.zig and rename main in there to run_main.
Add a new platform/native.zig which exposes main and just calls run_main.
Add a new platform/libmach.zig which will expose our main C ABI. It can be used in two ways:
- Mach engine apps:
  - mach_engine_set_init (takes a single argument, a function pointer like this). If you use this, you're writing a Mach engine app.
  - mach_run - takes control from here on out, and calls native_common.zig's run_main function.
- Mach core apps:
  - mach_core_set_init (function pointer like this)
  - mach_core_set_deinit (like this)
  - mach_core_set_update (like this)
  - mach_run - takes control from here on out, and calls native_common.zig's run_main function.

In theory, I think that libmach.zig could maybe look something like this, except it would also need to expose those mach_* functions above.

Then inside of the mach_run implementation, it would look at whether you used mach_engine_set_init or mach_core_set_init and decide what to do from there:

If you've used mach_core_set_init, it can just delegate to those function pointers you've set.
If you use mach_engine_set_init, then it just needs to delegate those 3 function calls to these 3 functions and call the user's init

Let me know if any of that doesn't make sense, there might be other things I've not thought about here. We can start small, maybe with just a PR to do the file renaming + expose mach_run when building a shared library?

Feel free to join the Matrix chat BTW, we do a lot of collaboration/discussion like this in there: https://matrix.to/#/#hexops:matrix.org

Ooh, sorry, I guess my brain kind of glazed over on that part. For some reason I was thinking of languages that can export C functions as part of their FFI approach (e.g. Go can do this) but not languages that use e.g. dlopen-based FFI. OK, so I think what we'll need to do in order to support this then is something like this: * `libmach` (single library) * Rename [platform/native.zig](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/src/platform/native.zig#L590-L646) to something like `native_common.zig` and rename `main` in there to `run_main`. * Add a new `platform/native.zig` which exposes `main` and just calls `run_main`. * Add a new `platform/libmach.zig` which will expose our main C ABI. It can be used in two ways: * _Mach engine_ apps: * `mach_engine_set_init` (takes a single argument, a function pointer [like this](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/examples/ecs-app/main.zig#L33)). If you use this, you're writing a _Mach engine_ app. * `mach_run` - takes control from here on out, and calls `native_common.zig`'s `run_main` function. * _Mach core_ apps: * `mach_core_set_init` (function pointer [like this](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/examples/triangle/main.zig#L10)) * `mach_core_set_deinit` ([like this](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/examples/triangle/main.zig#L74)) * `mach_core_set_update` ([like this](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/examples/triangle/main.zig#L76)) * `mach_run` - takes control from here on out, and calls `native_common.zig`'s `run_main` function. In theory, I think that `libmach.zig` could maybe look [*something* like this](https://gist.github.com/slimsag/17785280b93c8e9a79823dace9f9567c), except it would also need to expose those `mach_*` functions above. Then inside of the `mach_run` implementation, it would look at whether you used `mach_engine_set_init` or `mach_core_set_init` and decide what to do from there: * If you've used `mach_core_set_init`, it can just delegate to those function pointers you've set. * If you use `mach_engine_set_init`, then it just needs to delegate those 3 function calls to [these 3 functions](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/src/engine.zig#L25-L40) and call the user's `init` Let me know if any of that doesn't make sense, there might be other things I've not thought about here. We can start small, maybe with just a PR to do the file renaming + expose `mach_run` when building a shared library? Feel free to join the Matrix chat BTW, we do a lot of collaboration/discussion like this in there: https://matrix.to/#/#hexops:matrix.org

zack466 commented

2022-07-19 04:37:58 +00:00

(Migrated from github.com)

Just to track progress:
#406
#420
#423

Just to track progress: #406 #420 #423

silversquirl commented

2022-07-22 12:59:12 +00:00

(Migrated from github.com)

Rename platform/native.zig to something like native_common.zig and rename main in there to run_main.

Why is this necessary? The name main doesn't mean anything in Zig, it's just what std/start.zig looks for in @import("root") when you compile an executable.

> Rename [platform/native.zig](https://github.com/hexops/mach/blob/05b0df052d6723284e92b0c9df9485186dac2095/src/platform/native.zig#L590-L646) to something like native_common.zig and rename main in there to run_main. Why is this necessary? The name `main` doesn't mean anything in Zig, it's just what `std/start.zig` looks for in `@import("root")` when you compile an executable.

emidoots commented

2023-07-13 00:37:20 +00:00

(Migrated from github.com)

new approach hexops/mach#858