(regression) x86_64-macos -> aarch64-macos cross compilation failing after update to Zig master

emidoots commented

2021-11-27 07:06:12 +00:00

(Migrated from github.com)

After updating to latest Zig master https://github.com/hexops/mach/pull/107 we find an intriguing failure when cross compiling x86_64-macos -> aarch64-macos.

Notably, linux/windows -> aarch64-macos do not fail.

https://github.com/hexops/mach/runs/4339864442?check_suite_focus=true

After updating to latest Zig master https://github.com/hexops/mach/pull/107 we find an intriguing failure when cross compiling x86_64-macos -> aarch64-macos. Notably, linux/windows -> aarch64-macos do not fail. <img width="1051" alt="image" src="https://user-images.githubusercontent.com/3173176/143671835-bf3227bd-428e-4220-a2d1-81586adc075d.png"> https://github.com/hexops/mach/runs/4339864442?check_suite_focus=true

emidoots commented

2021-11-27 16:13:42 +00:00

(Migrated from github.com)

Huh, strangely with (nearly) the same Zig version on my Intel MBP everything is fine:

$ zig build test -Dtarget=aarch64-macos
warning(link): unable to resolve dependency /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
warning(link): unable to resolve dependency /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
warning: created /Users/slimsag/Desktop/hexops/mach/glfw/zig-cache/o/83c2edab7d4451bb28605a31fc98a00d/test but skipping execution because it is non-native

$ zig build test -Dtarget=aarch64-macos.11
warning(link): unable to resolve dependency /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
warning(link): unable to resolve dependency /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
warning: created /Users/slimsag/Desktop/hexops/mach/glfw/zig-cache/o/b1636399f5403b7ee98b9329a4d4703f/test but skipping execution because it is non-native

$ zig build test -Dtarget=aarch64-macos.12
warning(link): unable to resolve dependency /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
warning(link): unable to resolve dependency /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
warning: created /Users/slimsag/Desktop/hexops/mach/glfw/zig-cache/o/ca237d10517e108bb0bc18d25be67e1b/test but skipping execution because it is non-native

Huh, strangely with (nearly) the same Zig version on my Intel MBP everything is fine: ``` $ zig build test -Dtarget=aarch64-macos warning(link): unable to resolve dependency /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork warning(link): unable to resolve dependency /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices warning: created /Users/slimsag/Desktop/hexops/mach/glfw/zig-cache/o/83c2edab7d4451bb28605a31fc98a00d/test but skipping execution because it is non-native $ zig build test -Dtarget=aarch64-macos.11 warning(link): unable to resolve dependency /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork warning(link): unable to resolve dependency /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices warning: created /Users/slimsag/Desktop/hexops/mach/glfw/zig-cache/o/b1636399f5403b7ee98b9329a4d4703f/test but skipping execution because it is non-native $ zig build test -Dtarget=aarch64-macos.12 warning(link): unable to resolve dependency /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork warning(link): unable to resolve dependency /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices warning: created /Users/slimsag/Desktop/hexops/mach/glfw/zig-cache/o/ca237d10517e108bb0bc18d25be67e1b/test but skipping execution because it is non-native ```

emidoots commented

2021-11-27 16:15:32 +00:00

(Migrated from github.com)

Could be a difference in the XCode versions.

Or possibly the macOS version even? GitHub actions aren't running Monterey yet (but maybe we can opt in? see https://github.com/github/roadmap/issues/240 ) while my system is on Monterey.

Could be a difference in the XCode versions. Or possibly the macOS version even? GitHub actions aren't running Monterey yet (but maybe we can opt in? see https://github.com/github/roadmap/issues/240 ) while my system is on Monterey.

emidoots commented

2021-11-27 17:29:39 +00:00

(Migrated from github.com)

Prior tests were with latest XCode for macOS 12, and CLT 11.3 SDK (surprisingly updating XCode doesn't update CLT)

Same thing with no XCode/CLT, though: it just works.

Prior tests were with latest XCode for macOS 12, and CLT 11.3 SDK (surprisingly updating XCode doesn't update CLT) Same thing with no XCode/CLT, though: it just works.

emidoots commented

2021-11-27 18:09:43 +00:00

(Migrated from github.com)

Looks like macOS 12 on GitHub Actions won't be supported until at least early 2022, yuck.

https://github.com/actions/virtual-environments/issues/3649

Also tried what is described in https://github.com/github/roadmap/issues/240 but no dice, if there is a beta program for Monterey we're not in it.

Looks like macOS 12 on GitHub Actions won't be supported until at least early 2022, yuck. https://github.com/actions/virtual-environments/issues/3649 Also tried what is described in https://github.com/github/roadmap/issues/240 but no dice, if there is a beta program for Monterey we're not in it. <img width="1144" alt="image" src="https://user-images.githubusercontent.com/3173176/143695024-d429e550-6761-43ca-b9ab-2f6deff4f0ed.png">

emidoots commented

2021-11-27 20:04:32 +00:00

(Migrated from github.com)

With set_sysroot = true we don't run into the issue. This implies that Zig is picking up system x86_64 libraries (maybe during dependency resolution?) and trying to link those in by accident, instead of properly resolving the ones in the -iframework and -F parameters first.

With `set_sysroot = true` we don't run into the issue. This implies that Zig is picking up system x86_64 libraries (maybe during dependency resolution?) and trying to link those in by accident, instead of properly resolving the ones in the `-iframework` and `-F` parameters first.

emidoots commented

2021-11-27 20:10:57 +00:00

(Migrated from github.com)

Looks like macos-latest refers to macOS 11 with GitHub actions, and includes XCode 13.1 / CLT 13.0.0

https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md

Looks like `macos-latest` refers to macOS 11 with GitHub actions, and includes XCode 13.1 / CLT 13.0.0 https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md

emidoots commented

2021-11-27 20:15:34 +00:00

(Migrated from github.com)

What we know so far..

When cross compiling x86_64-macos -> aarch64-macos:

On GitHub actions (macOS 11, XCode 13) if we do not set sysroot, we get mismatched cpu architecture: expected Arch.aarch64, found Arch.x86_64
On Intel MBP (macOS 12, with both XCode 13 AND no XCode) if we do not set sysroot, linking works fine.

The only fundamental difference here is macOS 11 vs macOS 12.

What we know so far.. When cross compiling x86_64-macos -> aarch64-macos: 1. On GitHub actions (macOS 11, XCode 13) if we do not set sysroot, we get `mismatched cpu architecture: expected Arch.aarch64, found Arch.x86_64` 2. On Intel MBP (macOS 12, with both XCode 13 AND no XCode) if we do not set sysroot, linking works fine. **The only fundamental difference here is macOS 11 vs macOS 12.**

emidoots commented

2021-11-28 12:28:38 +00:00

(Migrated from github.com)

OK, finally have a complete explanation.

Context: We do not wish to set sysroot, doing so prevents someone wishing to use e.g. mach-glfw alone from including libraries not found in the mach native SDKs (e.g. Vulkan is not included in our sdk-macos-11.3), we just want to use the sdk-macos-11.3 only for libraries mach-glfw needs by default - things not present on the native system.

Framework .tbd stubs define re-exported libraries:

github.com/hexops/sdk-macos-11.3@6d49d06e12/root/System/Library/Frameworks/Foundation.framework/Foundation.tbd (L24-L25)

When we link -framework Foundation without setting sysroot, it finds our sdk-macos-11.3 copy because we've added it to the framework include path.

On macOS 12+, it simply fails to find these reexported libraries:

warning(link): unable to resolve dependency /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
warning(link): unable to resolve dependency /usr/lib/libobjc.A.dylib

This is fine on macOS 12+, because we also explicitly link CoreFoundation and objc separately (not with absolute paths like in those reexported libraries.)

But on macOS 11 and earlier, these libraries in fact exist on the system. /usr/lib/libobjc.A.dylib is a real x86_64 dylib, and that's a mismatch.

OK, finally have a complete explanation. Context: We do not wish to set `sysroot`, doing so prevents someone wishing to use e.g. mach-glfw alone from including libraries not found in the mach native SDKs (e.g. Vulkan is not included in our sdk-macos-11.3), we just want to use the sdk-macos-11.3 only for libraries mach-glfw needs by default - things not present on the native system. Framework .tbd stubs define re-exported libraries: https://github.com/hexops/sdk-macos-11.3/blob/6d49d06e1280f0f4a207720cf67df44eed41e4db/root/System/Library/Frameworks/Foundation.framework/Foundation.tbd#L24-L25 When we link `-framework Foundation` without setting sysroot, it finds our sdk-macos-11.3 copy because we've added it to the framework include path. On macOS 12+, it simply fails to find these reexported libraries: ``` warning(link): unable to resolve dependency /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation warning(link): unable to resolve dependency /usr/lib/libobjc.A.dylib ``` This is fine on macOS 12+, because we also explicitly link `CoreFoundation` and `objc` separately (not with absolute paths like in those reexported libraries.) But on macOS 11 and earlier, these libraries in fact exist on the system. `/usr/lib/libobjc.A.dylib` is a real x86_64 dylib, and that's a mismatch.

emidoots commented

2021-11-28 12:36:04 +00:00

(Migrated from github.com)

Here is where in the Zig linker code that these re-exported libraries are resolved: https://sourcegraph.com/github.com/ziglang/zig@aa61e03f244a72ea01f05c3ceea7c5fb5aadf1ff/-/blob/src/link/MachO.zig?L1267:4

One option I looked into here: what if we made these paths relative (/usr/lib/libobjc.A.dylib -> usr/lib/libobjc.A.dylib) in our copy only, then included sdk-macos-11.3/root on the lib path. The hope being that usr/lib/libobjc.A.dylib doesn't resolve the system copy, but it would resolve to the sdk-macos-11.3/root/usr/lib/libobjc.A.dylib copy.

The problem with that is that when resolving dependencies, Zig's linker doesn't consider the framework/library search path - it expects absolute paths (rightfully so perhaps?) so unless we could specify fully qualified paths, that's a no-go.

Here is where in the Zig linker code that these re-exported libraries are resolved: https://sourcegraph.com/github.com/ziglang/zig@aa61e03f244a72ea01f05c3ceea7c5fb5aadf1ff/-/blob/src/link/MachO.zig?L1267:4 One option I looked into here: what if we made these paths relative (`/usr/lib/libobjc.A.dylib` -> `usr/lib/libobjc.A.dylib`) in our copy only, then included `sdk-macos-11.3/root` on the lib path. The hope being that `usr/lib/libobjc.A.dylib` doesn't resolve the system copy, but it would resolve to the `sdk-macos-11.3/root/usr/lib/libobjc.A.dylib` copy. The problem with that is that when resolving dependencies, Zig's linker doesn't consider the framework/library search path - it expects absolute paths (rightfully so perhaps?) so unless we could specify fully qualified paths, that's a no-go.

emidoots commented

2021-11-28 12:37:21 +00:00

(Migrated from github.com)

Another option is to say we don't support cross compilation to Mac unless you're on macOS 12+. But this is icky, particularly because we're relying on some subtle behavior here which could change (that these libs do not exist on the system.)

emidoots commented

2021-11-28 18:30:39 +00:00

(Migrated from github.com)

Another option to consider: maybe we send a PR to zld to have it respect framework/lib include dirs when resolving dependencies, if an absolute filepath is not found. At first thought, that would seem specific to what Mach is doing but I think Zig would need to solve this too when it starts distributing tbd stubs with the package manager?

emidoots commented

2021-11-28 18:45:16 +00:00

(Migrated from github.com)

Another valid relatively low-effort option: strip all dependency sections from the .tbd files we distribute.

Pro: Fixes the issue on macOS 11
Pro: Doesn't rely on side effects on macOS 12.
Con: Requires specifying all dependencies manually.

Maybe this option is not too bad - the number of dependencies for system libs seems small in general, there's no harm in specifying all dependencies manually, and if you're using system_sdk.zig you are already specifying libs to link.

Another valid relatively low-effort option: strip all dependency sections from the .tbd files we distribute. * Pro: Fixes the issue on macOS 11 * Pro: Doesn't rely on side effects on macOS 12. * Con: Requires specifying all dependencies manually. Maybe this option is not too bad - the number of dependencies for system libs seems small in general, there's no harm in specifying all dependencies manually, and if you're using `system_sdk.zig` you are already specifying libs to link.

emidoots commented

2021-11-29 01:15:13 +00:00

(Migrated from github.com)

Another valid relatively low-effort option: strip all dependency sections from the .tbd files we distribute.

I plan to go this route. However, this is a non-backwards-compatible change to the sdk-macos repos. Thus, I am first going to ensure we have SDK version pinning and automatic updates for SDKs.

> Another valid relatively low-effort option: strip all dependency sections from the .tbd files we distribute. I plan to go this route. However, this is a non-backwards-compatible change to the sdk-macos repos. Thus, I am first going to ensure we have SDK version pinning and automatic updates for SDKs.

Rows
Columns

(regression) x86_64-macos -> aarch64-macos cross compilation failing after update to Zig master #108