build: self-hosted compiler progress #180
Labels
No labels
CI
all
basisu
blog
bug
build
contributor-friendly
core
correctness
deferred
dev
direct3d-headers
docs
driver-os-issue
duplicate
dxcompiler
editor
examples
experiment
feature-idea
feedback
flac
freetype
gamemode
gkurve
glfw
gpu
gpu-dawn
harfbuzz
help welcome
in-progress
infrastructure
invalid
libmach
linux-audio-headers
long-term
mach
mach.gfx
mach.math
mach.physics
mach.testing
model3d
needs-triage
object
opengl-headers
opus
os/linux
os/macos
os/wasm
os/windows
package-manager
priority
proposal
proposal-accepted
question
roadmap
slipped
stability
sysaudio
sysgpu
sysjs
validating-fix
vulkan-zig-generated
wayland-headers
website
wontfix
wrench
www
x11-headers
xcode-frameworks
zig-update
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
hexops/mach#180
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This is the tracking issue for getting all Mach examples/libraries working with the new self-hosted Zig compiler, which will be enabled by default in Zig v0.11 and is already enabled by default in all nightly Zig builds.
Status
cd libs/ecs && zig build test -fno-stage1cd libs/ecs && zig build test -fstage1cd libs/gpu && zig build testcd libs/gpu && zig build run-examplecd libs/glfw && zig build testcd libs/sysaudio && zig build testcd libs/sysaudio && zig build run-example-soundio-sine-wavecd libs/freetype && zig build testcd libs/freetype && zig build run-example-single-glyph -- 'a'cd libs/freetype && zig build run-example-glyph-to-svgcd libs/gpu-dawn && zig buildzig build-exe --main-pkg-path . ./tools/html-generator.zigzig build run-example-triangle -fstage1 -Dtarget=wasm32-freestanding-muslzig build run-shaderexpzig build run-example-advanced-gen-texture-lightzig build run-example-boidszig build run-example-ecs-appzig build run-example-fractal-cubezig build run-example-gkurvezig build run-example-image-blurzig build run-example-instanced-cubezig build run-example-map-asynczig build run-example-rotating-cubezig build run-example-textured-cubezig build run-example-trianglezig build run-example-triangle-msaazig build run-example-two-cubeszig build run-example-cubemapzig build example-triangle -Dtarget=wasm32-freestanding-muslzig build run-example-triangle -fstage1 -Dtarget=wasm32-freestanding-muslzig build example-sysaudio -Dtarget=wasm32-freestanding-muslzig build run-example-sysaudio -fstage1 -Dtarget=wasm32-freestanding-muslzig buildzig build testKnown issues
Zig issues we are blocked on
Zig issues we have hacky workarounds for
Search for
TODO(self-hosted)Zig issues that are problems for us, but haven't been filed yet
NOT YET FILED:
map-asyncexample (which does not use zig async) required adding a useless field to theAppfile struct (*Appis treated as*const Appfor some reason otherwise, not sure why. Need to make a minimal repro and file a bug.error: expected type '*main.main', found '*const main.main')github.com/hexops/mach@a4ddfb6bdaI made a quick test on M1 and Intel Macs. Stage2 has made a lot of progress! Running boids is now down to a single compile error.
M1 Max
Intel Macbook Pro 2019
Wow, nice!
That bitcast warning is interesting ... so the reason the bitcast is there is because we have the same exact code symlink'd into the
libs/directory (so$REPO/gpu/libs/mach-glfwis a symlink to$REPO/glfw) to workaround the fact that dependencies must be below thebuild.zigfile (relative imports not allowed), and since there's not a package manager we can use to avoid this.Without this, we would have to duplicate all code from
glfw/in this repository (which gets published as a standalone library in a separate repo via some git-fu github.com/hexops/mach-glfw) into thegpu/libs/mach-glfwdirectory OR make that a Git submodule... to our own repository 🤮I would hope that there is a way we can tell the Zig compiler still "hey, trust me, I really know what I'm doing - just unsafely cast type A to type B", but maybe not.
For a case like this that would be nice. I don't know how it's supposed to work now. Stage2/3 is moving really fast, so perhaps some cornercase just isn't implemented yet. I wonder if we would get the same error on Linux and Windows?
This is some stuff I found out trying to test mach/glfw with -fno-stage1.
To compile the build.zig we just need to change
thisDir()to(comptime thisDir()). This is maybe due to the fact that the stage2 compiler can't infer as well when something should be resolved at comptime or at runtime.When running
zig build test -fno-stage1we get many errors:Not sure of the above.
Probably an error when trying to solve namespacing? Basically we have already
const Window = @import("Window.zig");but we are callingglfw.Window..., the fix is to remove glfw. and just use Window.The problem above, is probably due to a similar problem as
thisDir()only this time it's the opposite, it tries to resolve at comptimestd.debug.assert()but glfw_initialized is a runtime value, (maybe since it's a global value the compiler assumes it should be comptime?). A similar problem happens when building gpu-dawn. That gives this errorThen we have:
The easy fix for this is to just change
c.glfwExtensionSupported(extension)toc.glfwExtensionSupported(extension.ptr), though this still leaves the problem that zig translate-c transforms char* into many item pointers instead of null terminated pointers.The allowzero... is the result of a simple
const slice = try allocator.alloc(Monitor, @intCast(u32, count));that returns a type that the compiler can't interpret as a slice, hence whyslice[i]is non-indexable.Again, this is a problem of zig translate-c and strings, solved with
return @ptrCast([*:0]const u8, name).Same thing and code as with Monitor and alloc before, this time the line with the error is above the line indexing, so maybe there is also a bug with error messages.
Same as the first error, couldn't figure out where it comes from.
Same problem with namespacing as glfw.Window probably, the fix is to remove joystic. and use just the function names.
Same as the first error.
There is also to note that there may be more errors but that weren't caught since the compiler failed before it could get there
One more issue with compiling to stage2 is that mach-gpu uses a lot of function pointers, but the semantics of fp are changed in stage2.
fn (args) ret->*const fn(args) retUpdate to solve this error, the type given when calling a C function that returns a string is
*const allowzero u8, this failsstd.mem.span()used inglfw/src/{clipboard.zig,key.zig,main.zig,Joystick.zig, casting the C strings with@ptrCast([*:0]const u8, name)solves the compilation errorThis was not a problem with
alloc(), it was monitors, gotten from a C function, that was the error, same as vidmode.To fix it we cast the C pointer to this:
@ptrCast([*c]const ?*c.GLFWmonitor, monitors)and@ptrCast([*c]const c.GLFWvidmode, modes).This leaves:
as the last error to solve
Solved the last error for compiling mach with glfw, basically
_ = glfw.getProcAddress("foobar")was being called at compile time because GLproc, the return type of getProcAddress used the old function declaration.Will test with the latest version of mach now and send a PR
Found a stage2 bug while compiling gpu/ with -fno-stage1:
This is in gpu/src/NaticeInstance,zig, descriptor is defined as:
The problem here is that the switch uses the wrong type for wgsl
Found out why the union is bugged.
This is how it is now, and also the code that swaps the union types.
But by changing the order in
CodeTag:We get the correct types. I suspect the stage2 compiler uses the order instead of the names to assign the types.
@PiergiorgioZagaria nice work, thanks so much for continuing to dig into this. I updated the issue description just now with what (I think?) the status quo is w.r.t. stage2 support, let me know if I should update anything there
This is a bug I found in mach-freetype.
Basically we have 4 packages:
The problem is that when trying to compile without adding
utils_pkgto our executable like this:We get the error:
Adding
main_tests.addPackage(utils_pkg)and using:To actually import it (without it the compiler doesn't evaluate because of lazy evaluation), seems to solve the problem.
Could this be related to https://github.com/ziglang/zig/issues/9204
aarch64-macos now mostly working.
Workarounds employed:
map-asyncexample (which does not use zig async) required adding a useless field to theAppfile struct (*Appis treated as*const Appfor some reason otherwise, not sure why. Need to make a minimal repro and file a bug.error: expected type '*main.main', found '*const main.main')github.com/hexops/mach@a4ddfb6bda@cImports to workaround https://github.com/ziglang/zig/issues/12483 (e.g.github.com/hexops/mach@80e127b5aeandgithub.com/hexops/mach@51932241c1)@embedFileseems to have regressed?github.com/hexops/mach@bc5e2fe9bf@cImport/translated file manually:Known issues
Failing even with stage1 and
-fstage1!image-blurfails with obscureerror: FileNotFound(even with-fstage1!)textured-cubefails with obscureerror: FileNotFound(even with-fstage1!)gkurvedoesn't work (depends on freetype which has@cImportissues)zig build run-example-triangle -Dtarget=wasm32-freestanding-muslFailing with self-hosted/stage3
ecs-appcrashes (compiler bug, need to file)image-blurfails with obscureerror: FileNotFound(even with-fstage1!)textured-cubefails with obscureerror: FileNotFound(even with-fstage1!)gkurvedoesn't work (depends on freetype which has@cImportissues)@cImportissues:sysaudiofreetype@cImportworkaround to Darwin hostszig build run-example-triangle -Dtarget=wasm32-freestanding-musl-fstage1on aarch64-macos host, it doesn't work. Obscureerror: FileNotFoundConfirmed working
mach/glfwtestsmach/gpuexampleshaderexpapple_pie has since been replaced with a web server dedicated to WASM serving: https://github.com/hexops/mach/tree/main/tools/wasmserve
We're now fully on the self-hosted compiler, so closing.