hexops/mach

Fork 0

make static Dawn library easy to consume in other projects #109

New issue

Closed

opened 2021-11-28 22:38:01 +00:00 by emidoots · 7 comments

emidoots commented

2021-11-28 22:38:01 +00:00

(Migrated from github.com)

Spawned from Twitter convo https://twitter.com/rsms/status/1465073396861992961

We are already doing a fair amount of work to enable effortless compilation (and cross-compilation) of Google's Dawn WebGPU implementation, it makes sense to have this exist in a standalone project in which anyone interested in building static Dawn without all the hassle of Google tools, dependencies, etc. can consume.

This should also include the C shims needed to access the portions of dawn_native's API needed to e.g. bind Dawn to a GLFW window and similar from C.

cc @rsms

Spawned from Twitter convo https://twitter.com/rsms/status/1465073396861992961 We are already doing a fair amount of work to enable effortless compilation (and cross-compilation) of Google's Dawn WebGPU implementation, it makes sense to have this exist in a standalone project in which anyone interested in building static Dawn without all the hassle of Google tools, dependencies, etc. can consume. This should also include the C shims needed to access the portions of dawn_native's API needed to e.g. bind Dawn to a GLFW window and similar from C. cc @rsms

❤️ 1

emidoots commented

2021-11-28 22:46:09 +00:00

(Migrated from github.com)

@rsms a few questions for you:

(1) Preference for single big libdawn.a static lib, or one for each component? Right now we build one for each component:

ls zig-out/lib
libabseil-cpp.a		libdawn-native.a	libdawn-utils.a		libglfw.a		libspirv-tools.a
libdawn-native-mach.a	libdawn-platform.a	libdawn-wire.a		libgpu.a		libtint.a

(2) Clone the repo and build from source (would require zig and git), or download some prebuilt binary release?

(3) You mentioned on Twitter noticing Dawn's binary size being quite huge, I also noticed that - it was one of the reasons I switched to per-component static libs (was looking to debug it), right now we have:

% du -sh zig-out/lib/*
 18M	zig-out/lib/libabseil-cpp.a
2.4M	zig-out/lib/libdawn-native-mach.a
 66M	zig-out/lib/libdawn-native.a
940K	zig-out/lib/libdawn-platform.a
2.4M	zig-out/lib/libdawn-utils.a
 26M	zig-out/lib/libdawn-wire.a
1.6M	zig-out/lib/libglfw.a
903M	zig-out/lib/libgpu.a (all linked together)
539M	zig-out/lib/libspirv-tools.a
247M	zig-out/lib/libtint.a

Of course an actual triangle example binary linked against them is much smaller:

% du -sh zig-out/bin/dawn-example 
 20M	zig-out/bin/dawn-example

But curious if you have any thoughts/concerns around this

@rsms a few questions for you: (1) Preference for single big `libdawn.a` static lib, or one for each component? Right now we build one for each component: ``` ls zig-out/lib libabseil-cpp.a libdawn-native.a libdawn-utils.a libglfw.a libspirv-tools.a libdawn-native-mach.a libdawn-platform.a libdawn-wire.a libgpu.a libtint.a ``` (2) Clone the repo and build from source (would require `zig` and `git`), or download some prebuilt binary release? (3) You mentioned on Twitter noticing Dawn's binary size being quite huge, I also noticed that - it was one of the reasons I switched to per-component static libs (was looking to debug it), right now we have: ``` % du -sh zig-out/lib/* 18M zig-out/lib/libabseil-cpp.a 2.4M zig-out/lib/libdawn-native-mach.a 66M zig-out/lib/libdawn-native.a 940K zig-out/lib/libdawn-platform.a 2.4M zig-out/lib/libdawn-utils.a 26M zig-out/lib/libdawn-wire.a 1.6M zig-out/lib/libglfw.a 903M zig-out/lib/libgpu.a (all linked together) 539M zig-out/lib/libspirv-tools.a 247M zig-out/lib/libtint.a ``` Of course an actual triangle example binary linked against them is much smaller: ``` % du -sh zig-out/bin/dawn-example 20M zig-out/bin/dawn-example ``` But curious if you have any thoughts/concerns around this

rsms commented

2021-11-29 18:30:18 +00:00

(Migrated from github.com)

Thank you Stephen for making this effort!

(1) Preference for single big libdawn.a static lib, or one for each component?
One for each component seems like a better approach. In this case it would be useful to include a text file documenting the dependencies (e.g. libdawn-native.a requires libspirv-tools.a) Separate lib files also makes distribution more streamlined.

(2) Build from source or download prebuilt binary release?
Build from source seems like the most pragmatic solution. Zig being so readily available via binary dist itself makes it a small ask from users. Might be best to start that way and then consider adding github actions to produce release binaries later on if there's demand.

(3) You mentioned on Twitter noticing Dawn's binary size being quite huge
Indeed quite thicc. It's from all the C++ and the fact that no stripping or DCE-esque optimizations can be done on generic object code. However I've got it in my todo list to experiment with -fvisibility and a linker version script to explicitly define public symbols which should in theory allow the compiler to spit out smaller object files.

Would you consider separating the mach code from the "static dawn lib" project? I.e. making mach a "consumer"/"user" of the dawn libraries?

Thank you Stephen for making this effort! > (1) Preference for single big libdawn.a static lib, or one for each component? One for each component seems like a better approach. In this case it would be useful to include a text file documenting the dependencies (e.g. `libdawn-native.a requires libspirv-tools.a`) Separate lib files also makes distribution more streamlined. > (2) Build from source or download prebuilt binary release? Build from source seems like the most pragmatic solution. Zig being so readily available via binary dist itself makes it a small ask from users. Might be best to start that way and then consider adding github actions to produce release binaries later on if there's demand. > (3) You mentioned on Twitter noticing Dawn's binary size being quite huge Indeed quite thicc. It's from all the C++ and the fact that no stripping or DCE-esque optimizations can be done on generic object code. However I've got it in my todo list to experiment with `-fvisibility` and a linker version script to explicitly define public symbols which should in theory allow the compiler to spit out smaller object files. Would you consider separating the mach code from the "static dawn lib" project? I.e. making mach a "consumer"/"user" of the dawn libraries?

emidoots commented

2021-11-29 19:50:57 +00:00

(Migrated from github.com)

However I've got it in my todo list to experiment with -fvisibility and a linker version script to explicitly define public symbols which should in theory allow the compiler to spit out smaller object files.

Whoa that's a cool idea!

Would you consider separating the mach code from the "static dawn lib" project? I.e. making mach a "consumer"/"user" of the dawn libraries?

Absolutely, yes, that's the primary thing this issue is for. A few thoughts:

For Mach itself there are a lot of good reasons to prefer monorepo development, so here's how I envision this working in a way that is compatible with both your and my goals:

There will be a separate repo https://github.com/hexops/mach-dawn which would let anyone consume the Dawn static lib stuff entirely separately from the rest of Mach.
That repository would be an auto-sync'd copy of the dawn subdirectory in this repository: issues, PRs, etc. would be sent to this repository and once any change is merged to this repo it'll get auto-pushed to the https://github.com/hexops/mach-dawn repository.
Only downside of this is that PRs have to be sent to this repository, not to the mach-dawn repository, to avoid some complex merge conflicts that can arise.
An example of how we already do this today: the glfw/ sub-directory of this repository is similarly published entirely separately from the rest of Mach so others can use it in their own projects/engines at https://github.com/hexops/mach-glfw

> However I've got it in my todo list to experiment with -fvisibility and a linker version script to explicitly define public symbols which should in theory allow the compiler to spit out smaller object files. Whoa that's a cool idea! > Would you consider separating the mach code from the "static dawn lib" project? I.e. making mach a "consumer"/"user" of the dawn libraries? Absolutely, yes, that's the primary thing this issue is for. A few thoughts: For Mach itself there are a lot of good reasons to prefer monorepo development, so here's how I envision this working in a way that is compatible with both your and my goals: - There will be a separate repo https://github.com/hexops/mach-dawn which would let anyone consume the Dawn static lib stuff entirely separately from the rest of Mach. - That repository would be an auto-sync'd copy of the `dawn` subdirectory in this repository: issues, PRs, etc. would be sent to this repository and once any change is merged to this repo it'll get auto-pushed to the https://github.com/hexops/mach-dawn repository. - Only downside of this is that PRs have to be sent to this repository, not to the mach-dawn repository, to avoid some complex merge conflicts that can arise. - An example of how we already do this today: the `glfw/` sub-directory of this repository is similarly published entirely separately from the rest of Mach so others can use it in their own projects/engines at https://github.com/hexops/mach-glfw

meshula commented

2021-11-30 00:55:21 +00:00

(Migrated from github.com)

The dawn_utils part of the dawn library has various glfw interoperation bits in it.

Is dawn_utils part of the effort? If so it seems like it would be a good idea to have it be a third library, so that there's no hard dependency from dawn to glfw. dawn without glfw would be good...

The dawn_utils part of the dawn library has various glfw interoperation bits in it. Is dawn_utils part of the effort? If so it seems like it would be a good idea to have it be a third library, so that there's no hard dependency from dawn to glfw. dawn without glfw would be good...

emidoots commented

2021-11-30 01:48:20 +00:00

(Migrated from github.com)

I can't say how easy Dawn is to use without dawn_utils, of course - it also implements the swap chain logic bindings to Metal/DirectX/Vulkan which you would likely want even if not using GLFW - that'd be an issue to file upstream on dawn.googlesource.com

@meshula Yeah, I'd like for you to be able to choose (via build.zig, and CLI parameters) which of the components to build. You may still pay the cost of downloading GLFW (not sure), but it wouldn't build. I can't say how easy Dawn is to use without `dawn_utils`, of course - it also implements the swap chain logic bindings to Metal/DirectX/Vulkan which you would likely want even if not using GLFW - that'd be an issue to file upstream on dawn.googlesource.com

emidoots commented

2022-02-28 03:18:16 +00:00

(Migrated from github.com)

We've gotten a lot better here:

All the code is now in a separate repository for others to consume: https://github.com/hexops/mach-gpu-dawn
We now have the CI system building static libraries, a single libdawn.a with everything, and publishing them as GitHub releases. You can download the static lib, headers as JSON file, or tarball of everything.
Building from source and cross-compiling is as easy as e.g. zig build -Dtarget=aarch64-macos.12 -Ddawn-from-source=true - everything is built from source with Zig build system if you pass -Ddawn-from-source=true flag.
Our build scripts, etc. are far more polished than before.
Our fork of Dawn is much more formalized and clearly denotes how it differs from upstream (all patches will be sent upstream and really only apply to building Dawn with Zig.)
Windows support + binaries landing very soon. Probably less than a week away. https://github.com/hexops/mach/issues/86

There's still more to do, documentation to add, etc. but overall you can consume this from other projects now without much trouble I believe.

I'll close this issue, feel free to create new ones with any feedback / issues, etc.

We've gotten a lot better here: * All the code is now in a separate repository for others to consume: https://github.com/hexops/mach-gpu-dawn * We now have the CI system building static libraries, a single `libdawn.a` with everything, and publishing them as GitHub releases. You can download the static lib, headers as JSON file, or tarball of everything. * Building from source and cross-compiling is as easy as e.g. `zig build -Dtarget=aarch64-macos.12 -Ddawn-from-source=true` - everything is built from source with Zig build system if you pass `-Ddawn-from-source=true` flag. * Our build scripts, etc. are far more polished than before. * Our fork of Dawn is [much more formalized](https://github.com/hexops/dawn/tree/main/mach#mach-engine-fork-of-dawn) and clearly denotes [how it differs from upstream](https://github.com/hexops/dawn/pulls) (all patches will be sent upstream and really only apply to building Dawn with Zig.) * Windows support + binaries landing very soon. Probably less than a week away. https://github.com/hexops/mach/issues/86 There's still more to do, documentation to add, etc. but overall you can consume this from other projects now without much trouble I believe. I'll close this issue, feel free to create new ones with any feedback / issues, etc.

emidoots commented

2022-02-28 03:26:34 +00:00

(Migrated from github.com)

Latest binary release: https://github.com/hexops/mach-gpu-dawn/releases/tag/release-6b59025

Latest binary release: https://github.com/hexops/mach-gpu-dawn/releases/tag/release-6b59025 <img width="1282" alt="image" src="https://user-images.githubusercontent.com/3173176/155919001-dd075fe3-bf0b-4ce9-874b-eee0c9b165df.png">