dawn: source bundle trimming #758

New issue

Closed

opened 2023-04-17 16:30:31 +00:00 by emidoots · 1 comment

emidoots commented

2023-04-17 16:30:31 +00:00

(Migrated from github.com)

Most users don't have to download a Dawn source tarball. Still, we should try to trim it down further.

Downloading a source tarball for Dawn + all its dependencies is 91MB, right now. Noting some things for myself to follow up on to reduce that.

The uncompressed archive is 856M:

du -sh ../dawn-9e2a193db129ebaf42eccfb9d6bb40055645c5d9 | gsort -h
856M	../dawn-9e2a193db129ebaf42eccfb9d6bb40055645c5d9

We should be able to reduce that to ~124M if we do the following:

% du -sh ./* | gsort -h
...
1.2M	./tools
 10M	./out
 28M	./src
159M	./test
655M	./third_party

test can likely be removed:

slimsag@hexops-ci dawn-9e2a193db129ebaf42eccfb9d6bb40055645c5d9 % du -sh ./test/* | gsort -h
4.0K	./test/OWNERS
159M	./test/tint
slimsag@hexops-ci dawn-9e2a193db129ebaf42eccfb9d6bb40055645c5d9 % du -sh ./test/tint/* | gsort -h
4.0K	./test/tint/extract-spvasm.py
4.0K	./test/tint/parse_hlsl_errors.py
4.0K	./test/tint/test-all.sh
 28K	./test/tint/let
 32K	./test/tint/struct
 56K	./test/tint/const
 56K	./test/tint/unicode
 60K	./test/tint/layout
 84K	./test/tint/extensions
184K	./test/tint/benchmark
192K	./test/tint/samples
224K	./test/tint/access
224K	./test/tint/loops
288K	./test/tint/out_of_order_decls
312K	./test/tint/array
532K	./test/tint/diagnostic_filtering
560K	./test/tint/identifiers
560K	./test/tint/shadowing
752K	./test/tint/types
1.2M	./test/tint/shader_io
1.4M	./test/tint/var
2.7M	./test/tint/ptr_ref
2.8M	./test/tint/statements
4.5M	./test/tint/unittest
6.3M	./test/tint/vk-gl-cts
7.4M	./test/tint/bug
 13M	./test/tint/buffer
 24M	./test/tint/expressions
 92M	./test/tint/builtins

We could also remove swiftshader (software vulkan implementation), we don't use anything from it afaik (same for angle the opengl es software impl?):

 % du -sh ./third_party/* | gsort -h
4.0K	./third_party/go.mod
364K	./third_party/ninja
1.6M	./third_party/vulkan_memory_allocator
1.9M	./third_party/zlib
2.7M	./third_party/khronos
7.4M	./third_party/gn
 12M	./third_party/abseil-cpp
 52M	./third_party/angle
 67M	./third_party/vulkan-deps
510M	./third_party/swiftshader

We should be able to remove gn, or at least webgpu-cts:

du -sh ./third_party/gn/* | gsort -h
8.0K	./third_party/gn/abseil-cpp
8.0K	./third_party/gn/glfw
7.4M	./third_party/gn/webgpu-cts

Lastly, we can remove vulkan-headers as it is duplicative of https://github.com/hexops/vulkan-headers

% du -sh ./third_party/vulkan-deps/* | gsort -h
4.0K	./third_party/vulkan-deps/DEPS
4.0K	./third_party/vulkan-deps/LICENSE
4.0K	./third_party/vulkan-deps/OWNERS
4.0K	./third_party/vulkan-deps/README.chromium
4.0K	./third_party/vulkan-deps/README.md
4.0K	./third_party/vulkan-deps/additional_readme_paths.json
8.0K	./third_party/vulkan-deps/update-commit-message.py
1.2M	./third_party/vulkan-deps/vulkan-tools
2.6M	./third_party/vulkan-deps/vulkan-loader
3.2M	./third_party/vulkan-deps/spirv-headers
6.1M	./third_party/vulkan-deps/glslang
7.5M	./third_party/vulkan-deps/spirv-tools
 22M	./third_party/vulkan-deps/vulkan-headers
 24M	./third_party/vulkan-deps/vulkan-validation-layers

Most users don't have to download a Dawn source tarball. Still, we should try to trim it down further. Downloading a source tarball for Dawn + all its dependencies is 91MB, right now. Noting some things for myself to follow up on to reduce that. The uncompressed archive is 856M: ``` du -sh ../dawn-9e2a193db129ebaf42eccfb9d6bb40055645c5d9 | gsort -h 856M ../dawn-9e2a193db129ebaf42eccfb9d6bb40055645c5d9 ``` We should be able to **reduce that to ~124M** if we do the following: ``` % du -sh ./* | gsort -h ... 1.2M ./tools 10M ./out 28M ./src 159M ./test 655M ./third_party ``` test can likely be removed: ``` slimsag@hexops-ci dawn-9e2a193db129ebaf42eccfb9d6bb40055645c5d9 % du -sh ./test/* | gsort -h 4.0K ./test/OWNERS 159M ./test/tint slimsag@hexops-ci dawn-9e2a193db129ebaf42eccfb9d6bb40055645c5d9 % du -sh ./test/tint/* | gsort -h 4.0K ./test/tint/extract-spvasm.py 4.0K ./test/tint/parse_hlsl_errors.py 4.0K ./test/tint/test-all.sh 28K ./test/tint/let 32K ./test/tint/struct 56K ./test/tint/const 56K ./test/tint/unicode 60K ./test/tint/layout 84K ./test/tint/extensions 184K ./test/tint/benchmark 192K ./test/tint/samples 224K ./test/tint/access 224K ./test/tint/loops 288K ./test/tint/out_of_order_decls 312K ./test/tint/array 532K ./test/tint/diagnostic_filtering 560K ./test/tint/identifiers 560K ./test/tint/shadowing 752K ./test/tint/types 1.2M ./test/tint/shader_io 1.4M ./test/tint/var 2.7M ./test/tint/ptr_ref 2.8M ./test/tint/statements 4.5M ./test/tint/unittest 6.3M ./test/tint/vk-gl-cts 7.4M ./test/tint/bug 13M ./test/tint/buffer 24M ./test/tint/expressions 92M ./test/tint/builtins ``` We could also remove swiftshader (software vulkan implementation), we don't use anything from it afaik (same for angle the opengl es software impl?): ``` % du -sh ./third_party/* | gsort -h 4.0K ./third_party/go.mod 364K ./third_party/ninja 1.6M ./third_party/vulkan_memory_allocator 1.9M ./third_party/zlib 2.7M ./third_party/khronos 7.4M ./third_party/gn 12M ./third_party/abseil-cpp 52M ./third_party/angle 67M ./third_party/vulkan-deps 510M ./third_party/swiftshader ``` We should be able to remove `gn`, or at least webgpu-cts: ``` du -sh ./third_party/gn/* | gsort -h 8.0K ./third_party/gn/abseil-cpp 8.0K ./third_party/gn/glfw 7.4M ./third_party/gn/webgpu-cts ``` Lastly, we can remove vulkan-headers as it is duplicative of https://github.com/hexops/vulkan-headers ``` % du -sh ./third_party/vulkan-deps/* | gsort -h 4.0K ./third_party/vulkan-deps/DEPS 4.0K ./third_party/vulkan-deps/LICENSE 4.0K ./third_party/vulkan-deps/OWNERS 4.0K ./third_party/vulkan-deps/README.chromium 4.0K ./third_party/vulkan-deps/README.md 4.0K ./third_party/vulkan-deps/additional_readme_paths.json 8.0K ./third_party/vulkan-deps/update-commit-message.py 1.2M ./third_party/vulkan-deps/vulkan-tools 2.6M ./third_party/vulkan-deps/vulkan-loader 3.2M ./third_party/vulkan-deps/spirv-headers 6.1M ./third_party/vulkan-deps/glslang 7.5M ./third_party/vulkan-deps/spirv-tools 22M ./third_party/vulkan-deps/vulkan-headers 24M ./third_party/vulkan-deps/vulkan-validation-layers ```

emidoots commented

2024-07-08 13:05:50 +00:00

(Migrated from github.com)

https://github.com/hexops/mach/issues/1166#issuecomment-2212990680