mach: text rendering ECS module #877

New issue

Closed

opened 2023-07-23 18:12:49 +00:00 by emidoots · 1 comment

emidoots commented

2023-07-23 18:12:49 +00:00

(Migrated from github.com)

No description provided.

emidoots commented

2023-09-25 02:09:48 +00:00

(Migrated from github.com)

There now exists a Text ECS module which is rather experienced/needs a lot of work.

Learnings

OpenType fonts are represented as a bunch of SFNT tables, which have (a) all the metadata needed for proper shaping/etc - and aside from that are just (b) the beziers making up glyphs.

We need (a) for proper unicode / text shaping support, and that makes up the majority of freetype (as a dependency)'s size. You also cannot separate harfbuzz from freetype, because SFNT tables are really what we would need to compose for harfbuzz. With the gkurve experiments experiments it seems likely we can render beziers directly, so if gkurve did have a font format of it's own it would look... exactly like (a) and (b), i.e. exactly how OpenType fonts (ttf and otf) look today. Thus, logically, we should just use those directly most likely. The only difference would be that TrueType Level 2 fonts can contain cubics, which we would need to convert to quadratics on-the-fly.. but that's not a big deal.

The downside is primarily browser support, though: when combined, harfbuzz and freetype dependencies are rather chunky and will suck in a WASM context. The developer of photopea.com for example has been using a WASM compilation of harfbuzz, but even as stripped-down/minimal as they could make it.. it's still the largest part of their application. Not great.

In a browser/WASM context, Ghostty takes a different approach here which we also planned to take: delegating to a HTML5-canvas-based backend for fonts, which seems logical and should be fine for us. The browser ships these font rendering stacks themselves, we just use their API for it. The downside with this approach is we cannot use gkurve/GPU-accelerated beziers for font rendering, because the browser APIs do not provide access to the underlying bezier information.

Decision

So we have a decision to make:

Option 0: Ship harfbuzz+freetype to the web (likely adds at least 1MB to bundle size)
Option 1: Use a sprite-based font-texture-atlas text rendering implementation in BOTH the browser and natively.
- Using HTML5 canvas APIs for browsers for text measuring/shaping/rasterization.
- Using freetype/harfbuzz APIs natively for text measuring/shaping/rasterization.
Option 2: Use a different text rendering implementation per platform:
- Use a sprite-based font-texture-atlas approach in the browser.
- Use a gkurve actual-vector-graphics approach natively.

Option 0 is not great for Mach's goals of supporting browsers as a platform.

Option 2 necessitates having two entirely different text renderers / ECS modules entirely, which is a lot of added complexity. Additionally, it is likely there would be performance differences and characteristic differences between the two (you could rely on scalable text natively, but it'd be blurry in the browser - or you'd need to handle resizing rasterization via a platform-specific API.)

So really, we're asking: do we want two different text rendering stacks, one for native and one for web?

I think at this stage, we could do with less complexity & differences between platforms. Not just complexity we have to understand, but complexity our users would have to understand when e.g. writing shaders, too.

Decision: We won't use gkurve for font rendering for now. We'll have a very nice sprite-based texture-atlas text renderer that is integrated very nicely with the ECS, has good browser support, and call it a day for now.

Plan

Implement a glyph renderer which is platform-specific:
- Native: use freetype to render glyphs into bitmaps
- Browser: use CanvasRenderingContext2D: fillText() to render glyphs into bitmaps
Implement a text shaper which is platform-specific:
- Native: use harfbuzz to shape text runs
- Browser: use CanvasRenderingContext2D: measureText() to shape text runs
Implement a text layout interface, which can be provided to the Text ECS module to determine how text should be laid out. e.g. to enable flowing text around complex geometry, breaking at specific lengths instead of just when a newline is encountered, etc.
Integrate the above with the existing Text ECS module (mentioned at the top of this post), which would handle font atlas management and other text rendering nuances.

There now exists a [Text](https://github.com/hexops/mach/blob/c77619045f82066cb39b4fe254bd4d32b25ff40d/src/gfx/Text.zig) ECS module which is rather experienced/needs a lot of work. ## Learnings OpenType fonts are represented as a bunch of SFNT tables, which have (a) all the metadata needed for proper shaping/etc - and aside from that are just (b) the beziers making up glyphs. We need (a) for proper unicode / text shaping support, and that makes up the majority of freetype (as a dependency)'s size. You also cannot separate harfbuzz from freetype, because SFNT tables are really what we would need to compose for harfbuzz. With the gkurve experiments experiments it seems likely we can render beziers directly, so if gkurve did have a font format of it's own it would look... exactly like (a) and (b), i.e. exactly how OpenType fonts (ttf and otf) look today. Thus, logically, we should just use those directly most likely. The only difference would be that TrueType Level 2 fonts can contain cubics, which we would need to convert to quadratics on-the-fly.. but that's not a big deal. The downside is primarily browser support, though: when combined, harfbuzz and freetype dependencies are rather chunky and will suck in a WASM context. The developer of photopea.com for example has been using a WASM compilation of harfbuzz, but even as stripped-down/minimal as they could make it.. it's still the largest part of their application. Not great. In a browser/WASM context, Ghostty takes a different approach here which we also planned to take: delegating to a HTML5-canvas-based backend for fonts, which seems logical and should be fine for us. The browser ships these font rendering stacks themselves, we just use their API for it. The downside with this approach is we cannot use gkurve/GPU-accelerated beziers for font rendering, because the browser APIs do not provide access to the underlying bezier information. ## Decision So we have a decision to make: * Option 0: Ship harfbuzz+freetype to the web (likely adds at least 1MB to bundle size) * Option 1: Use a sprite-based font-texture-atlas text rendering implementation in BOTH the browser and natively. * Using HTML5 canvas APIs for browsers for text measuring/shaping/rasterization. * Using freetype/harfbuzz APIs natively for text measuring/shaping/rasterization. * Option 2: Use a different text rendering implementation per platform: * Use a sprite-based font-texture-atlas approach in the browser. * Use a gkurve actual-vector-graphics approach natively. Option 0 is not great for Mach's goals of supporting browsers as a platform. Option 2 necessitates having two entirely different text renderers / ECS modules entirely, which is a lot of added complexity. Additionally, it is likely there would be performance differences and characteristic differences between the two (you could rely on scalable text natively, but it'd be blurry in the browser - or you'd need to handle resizing rasterization via a platform-specific API.) So really, we're asking: do we want two different text rendering stacks, one for native and one for web? I think at this stage, we could do with less complexity & differences between platforms. Not just complexity we have to understand, but complexity our users would have to understand when e.g. writing shaders, too. **Decision:** We won't use gkurve for font rendering for now. We'll have a very nice sprite-based texture-atlas text renderer that is integrated very nicely with the ECS, has good browser support, and call it a day for now. ## Plan 1. Implement a _glyph renderer_ which is platform-specific: * Native: use freetype to render glyphs into bitmaps * Browser: use [CanvasRenderingContext2D: fillText()](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/fillText) to render glyphs into bitmaps 2. Implement a _text shaper_ which is platform-specific: * Native: use harfbuzz to shape text runs * Browser: use [CanvasRenderingContext2D: measureText()](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/measureText) to shape text runs 3. Implement a _text layout_ interface, which can be provided to the Text ECS module to determine how text should be laid out. e.g. to enable flowing text around complex geometry, breaking at specific lengths instead of just when a newline is encountered, etc. 3. Integrate the above with the existing Text ECS module (mentioned at the top of this post), which would handle font atlas management and other text rendering nuances.