ecs: ECS research

emidoots commented

2021-12-14 17:58:31 +00:00

(Migrated from github.com)

The point of this issue is to collect material / research the best way to write an ECS in Zig leveraging DOD and unique language features like comptime. If anyone has thoughts, please do chime in!

Things I need to look into more / form opinions about:

https://bevyengine.org/learn/book/getting-started/ecs/
ECS back and forth blog
https://github.com/SanderMertens/flecs
https://github.com/Leopotam/ecs
prime31: "i did a loose port of EnTT to zig." https://github.com/prime31/zig-ecs
Bitron: "by far the best experience I had was with bevy ecs. the way you write systems feels just right."

kbm:

flecs. [...] It seems like a great library and I love the features in theory. But I'm not sure if I'm ever going to actually use all of those. Coming from that C# ECS library I used (Entitas) I try to avoid bloat as much as possible from now on.

Entitas actually had code-generation. So you had to write components and then each time run a script that generates some glue-code. After ~3 years and 1000 components it became a complete nightmare.

prime31:

Flecs is exact opposite of bloat. compared to entitas its tiny
its also module based, so you can just use the root storage
when i use it, i basically just use sorting and its queries. it lets you do whatever you want with systems. you can make your own system scheduler, iterate multiple queries simultaneously, etc. Very flexible

slimsag:

The mention about Entitas having code generation is interesting, one thought I had is that if we do not need entities of different types to be in the same e.g. global list, we can leverage comptime to build entity types with components as dedicated fields which would likely be quite efficient

kbm:

yeah Entitas used code generation to generate a bunch of interfaces, to facilitate "better readable" code.

so you could make a PositionComponent {x,y} and hit code generation to generate helper methods which give you stuff like entity.ReplacePosition(x, y) or entity.RemovePosition(x, y)

Bitron:

in bevy it looks something like this:

struct Velocity {..}
struct Position {..} 

fn main() {
  App::new().add_system(physics).run();
}

fn physics(mut query: Query<(&mut Position, &Velocity)>) {
  for (mut pos, vel) in query.iter_mut() {
    pos += vel;
  }
}

prime31: "you can get almost exactly that with Flecs too 😉"

people I might be able to run ideas by later:

prime31
kbm
Bitron
idhank (iddev5)

The point of this issue is to collect material / research the best way to write an ECS in Zig leveraging DOD and unique language features like comptime. If anyone has thoughts, please do chime in! --- Things I need to look into more / form opinions about: * https://bevyengine.org/learn/book/getting-started/ecs/ * ECS back and forth blog * https://github.com/SanderMertens/flecs * https://github.com/Leopotam/ecs * prime31: "i did a loose port of EnTT to zig." https://github.com/prime31/zig-ecs * Bitron: "by far the best experience I had was with bevy ecs. the way you write systems feels just right." kbm: > flecs. [...] It seems like a great library and I love the features in theory. But I'm not sure if I'm ever going to actually use all of those. Coming from that C# ECS library I used (Entitas) I try to avoid bloat as much as possible from now on. > Entitas actually had code-generation. So you had to write components and then each time run a script that generates some glue-code. After ~3 years and 1000 components it became a complete nightmare. prime31: > Flecs is exact opposite of bloat. compared to entitas its tiny > its also module based, so you can just use the root storage > when i use it, i basically just use sorting and its queries. it lets you do whatever you want with systems. you can make your own system scheduler, iterate multiple queries simultaneously, etc. Very flexible slimsag: > The mention about Entitas having code generation is interesting, one thought I had is that if we do not need entities of different types to be in the same e.g. global list, we can leverage comptime to build entity types with components as dedicated fields which would likely be quite efficient kbm: > yeah Entitas used code generation to generate a bunch of interfaces, to facilitate "better readable" code. > > so you could make a `PositionComponent {x,y}` and hit code generation to generate helper methods which give you stuff like `entity.ReplacePosition(x, y)` or `entity.RemovePosition(x, y)` Bitron: > in bevy it looks something like this: ```zig struct Velocity {..} struct Position {..} fn main() { App::new().add_system(physics).run(); } fn physics(mut query: Query<(&mut Position, &Velocity)>) { for (mut pos, vel) in query.iter_mut() { pos += vel; } } ``` prime31: "you can get almost exactly that with Flecs too 😉" people I might be able to run ideas by later: * prime31 * kbm * Bitron * idhank (iddev5)

❤️ 2

InKryption commented

2021-12-14 18:14:23 +00:00

(Migrated from github.com)

Exciting stuff. I will say, I wrote a pretty short and shitty "Entity Manager/Registry" data structure, and it was really painless, leveraging std.MultiArrayList (no benchmarks on it or anything, just an educational experiment to understand it better).

One of the things I've understood from reading about ECS, in particular EnTT, is that it's at least similar on a surface level to making a hash map, in the sense that the design and implementation are all based on the balance in trade-offs between memory and speed you want, and that it's nigh impossible to make completely general-purpose ECS software, since it should always cater to the specific problem, unless you heavily leverage a lot of compile-time configuration in respect to those details (which you also have to answer, to what degree do you make that configurable before it becomes a complete mess?).

Whatever the case, looking forward to see Mach come up with something like this, you bet I'll be using it as soon as it's usable. And hopefully I'll be able to learn from more while seeing it progress.

Exciting stuff. I will say, I wrote a pretty short and shitty "Entity Manager/Registry" data structure, and it was really painless, leveraging `std.MultiArrayList` (no benchmarks on it or anything, just an educational experiment to understand it better). One of the things I've understood from reading about ECS, in particular EnTT, is that it's at least similar on a surface level to making a hash map, in the sense that the design and implementation are all based on the balance in trade-offs between memory and speed you want, and that it's nigh impossible to make completely general-purpose ECS software, since it should always cater to the specific problem, unless you heavily leverage a lot of compile-time configuration in respect to those details (which you also have to answer, to what degree do you make that configurable before it becomes a complete mess?). Whatever the case, looking forward to see Mach come up with something like this, you bet I'll be using it as soon as it's usable. And hopefully I'll be able to learn from more while seeing it progress.

emidoots commented

2021-12-14 18:23:14 +00:00

(Migrated from github.com)

Could you share your std.MultiArrayList stuff? I'd be curious to take a look (even if it was just an experiment / bad code / etc)

balance in trade-offs between memory and speed you want, and that it's nigh impossible to make completely general-purpose ECS software

Agreed, that is my (uninformed) opinion currently as well.

unless you heavily leverage a lot of compile-time configuration in respect to those details (which you also have to answer, to what degree do you make that configurable before it becomes a complete mess?).

I think that Zig's comptime can give us some nice ways to express this without getting too far into the weeds. That's my hope, at least. Will see how this pans out as I learn more about the constraints of ECS.

Could you share your `std.MultiArrayList` stuff? I'd be curious to take a look (even if it was just an experiment / bad code / etc) > balance in trade-offs between memory and speed you want, and that it's nigh impossible to make completely general-purpose ECS software Agreed, that is my (uninformed) opinion currently as well. > unless you heavily leverage a lot of compile-time configuration in respect to those details (which you also have to answer, to what degree do you make that configurable before it becomes a complete mess?). I think that Zig's comptime can give us some nice ways to express this without getting too far into the weeds. That's my hope, at least. Will see how this pans out as I learn more about the constraints of ECS.

👍 1

InKryption commented

2021-12-14 18:35:34 +00:00

(Migrated from github.com)

Could you share your std.MultiArrayList stuff? I'd be curious to take a look (even if it was just an experiment / bad code / etc)

Sure thing, though I'm not sure how readable it is, or if all of the intentions are clear enough to discern; I didn't add many (any?) comments, since like I said, it was purely exploratory: https://github.com/InKryption/small-zig-ecs/blob/main/src/main.zig
Edit: also just noticed, I haven't updated it in a while, so it won't compile without a few tweaks (the Allocgate change e.g.).

I think that Zig's comptime can give us some nice ways to express this without getting too far into the weeds. That's my hope, at least. Will see how this pans out as I learn more about the constraints of ECS.

Absolutely; it is just my personal opinion that overall, if push comes to shove, over-engineering should be avoided, since it would be better to have a simple, drop-in ECS that can be switched out for a better-suited, personalized system in the future of any project that happens to use Mach (that isn't to say to compromise on speed or efficiency though, those are obviously still important).

> Could you share your `std.MultiArrayList` stuff? I'd be curious to take a look (even if it was just an experiment / bad code / etc) Sure thing, though I'm not sure how readable it is, or if all of the intentions are clear enough to discern; I didn't add many (any?) comments, since like I said, it was purely exploratory: https://github.com/InKryption/small-zig-ecs/blob/main/src/main.zig Edit: also just noticed, I haven't updated it in a while, so it won't compile without a few tweaks (the Allocgate change e.g.). > I think that Zig's comptime can give us some nice ways to express this without getting too far into the weeds. That's my hope, at least. Will see how this pans out as I learn more about the constraints of ECS. Absolutely; it is just my personal opinion that overall, if push comes to shove, over-engineering should be avoided, since it would be better to have a simple, drop-in ECS that can be switched out for a better-suited, personalized system in the future of any project that happens to use Mach (that isn't to say to compromise on speed or efficiency though, those are obviously still important).

👍 1 ❤️ 1

meshula commented

2021-12-14 18:56:36 +00:00

(Migrated from github.com)

I used EnTT in one of my projects for quite some time, but stripped it out in favor of a boring hashmap as I found myself battling its semantics more and more. In particular, managing entity lifecycle was far more cumbersome than the value I as getting. I ran into concurrency issues as well. I also needed reverse look ups so I had my own maps of what entt was tracking...

so the summary would be, I needed:

concurrency,
reverse look ups,
simple treatment of null entities,
simple semantics

I ended up just stripping it all out in favor of a trivial set of concurrent hashmaps and an allocator:

        uint64_t create_entity() {
             static std::atomic<uint64_t> id = 0;
             return ++id;
         }

and life was so much simpler thereafter.

I used EnTT in one of my projects for quite some time, but stripped it out in favor of a boring hashmap as I found myself battling its semantics more and more. In particular, managing entity lifecycle was far more cumbersome than the value I as getting. I ran into concurrency issues as well. I also needed reverse look ups so I had my own maps of what entt was tracking... so the summary would be, I needed: - concurrency, - reverse look ups, - simple treatment of null entities, - simple semantics I ended up just stripping it all out in favor of a trivial set of concurrent hashmaps and an allocator: ```cpp uint64_t create_entity() { static std::atomic<uint64_t> id = 0; return ++id; } ``` and life was so much simpler thereafter.

👍 1 👀 1

emidoots commented

2021-12-14 19:01:39 +00:00

(Migrated from github.com)

@meshula Very interesting. How'd you manage concurrency and entity <-> component relations after doing that?

Can you elaborate on what "simple treatment of null entities" and "simple semantics" meant?

@meshula Very interesting. How'd you manage concurrency and entity <-> component relations after doing that? Can you elaborate on what "simple treatment of null entities" and "simple semantics" meant?

meshula commented

2021-12-14 19:39:21 +00:00

(Migrated from github.com)

concurrency.

I made a MPSC transactional queue, serviced at game loop frequency, but on its own thread. entity creation and deletion is managed via the queue's service routine.
the entity maps are treated as const outside of the transactional queue, any lookups return a locked referent object that guarantees that the fetched component and entity id stay alive until you release it. If a deletion is requested simultaneously, the service queue puts it into a destruction list until outstanding referents are released.

simple semantics.

entity management is done via a provider (which contains the transactional queue)
- no entity management is done by working with the entity system directly
const references via a referent lock. The component and its data are not const, but its participation in the provider is const if you see what I mean.
entities and components and the ECS are not bound via a type system.
- EnTT highly conflates what it does with how it's represented. i.e. It's a very type-centric approach, and those types spill over into your code because you need to instantiate objects versus EnTT's templates.
- I instead bound entity and component creation via closures, so creating a component bound to an entity involves stuffing closures in the queue that reify the construction and deletion.

null entities.

I wanted to have a trivial, and non-fragile, way to have undefined entity variables
* which I solved in EnTT by making sentinel entities. It always felt very heavyweight, and fragile, because it meant that I had to extremely diligent about assigning the sentinels. EnTT gets quite unhappy with invalid queries and reports its unhappiness in fabulous C++ template style at compile time, or at runtime, with similarly convoluted callstacks.

I worked with EnTT for a few years and can say all of it can be dealt with, and you learn how to think about it. But in the end, I felt the juice wasn't worth the squeeze, as it were, and came up with a much simpler pattern that makes me much happier. And concurrency became a deal breaker.

entt issue 470.

concurrency. - I made a MPSC transactional queue, serviced at game loop frequency, but on its own thread. entity creation and deletion is managed via the queue's service routine. - the entity maps are treated as const outside of the transactional queue, any lookups return a locked referent object that guarantees that the fetched component and entity id stay alive until you release it. If a deletion is requested simultaneously, the service queue puts it into a destruction list until outstanding referents are released. simple semantics. - entity management is done via a provider (which contains the transactional queue) * no entity management is done by working with the entity system directly - const references via a referent lock. The component and its data are not const, but its participation in the provider is const if you see what I mean. - entities and components and the ECS are not bound via a type system. * EnTT highly conflates what it does with how it's represented. i.e. It's a very type-centric approach, and those types spill over into your code because you need to instantiate objects versus EnTT's templates. * I instead bound entity and component creation via closures, so creating a component bound to an entity involves stuffing closures in the queue that reify the construction and deletion. null entities. - I wanted to have a trivial, and non-fragile, way to have undefined entity variables * which I solved in EnTT by making sentinel entities. It always felt very heavyweight, and fragile, because it meant that I had to extremely diligent about assigning the sentinels. EnTT gets quite unhappy with invalid queries and reports its unhappiness in fabulous C++ template style at compile time, or at runtime, with similarly convoluted callstacks. I worked with EnTT for a few years and can say all of it can be dealt with, and you learn how to think about it. But in the end, I felt the juice wasn't worth the squeeze, as it were, and came up with a much simpler pattern that makes me much happier. And concurrency became a deal breaker. entt issue 470.

👍 2

emidoots commented

2021-12-14 20:14:54 +00:00

(Migrated from github.com)

Really appreciate that write-up!

emidoots commented

2021-12-14 20:43:20 +00:00

(Migrated from github.com)

interesting resource: https://github.com/SanderMertens/ecs-faq/blob/master/README.md

👀 2

alichraghi commented

2021-12-20 21:50:07 +00:00

(Migrated from github.com)

http://entity-systems.wikidot.com

silversquirl commented

2021-12-20 22:26:55 +00:00

(Migrated from github.com)

In case it's useful.

I don't know a whole lot about ECS, so it's probably not the fastest thing in the world, but maybe it can help with inspiration :)

[In case it's useful](https://github.com/silversquirl/znt). I don't know a whole lot about ECS, so it's probably not the fastest thing in the world, but maybe it can help with inspiration :)

❤️ 1

emidoots commented

2021-12-20 23:13:53 +00:00

(Migrated from github.com)

Much appreciated @AliChraghi and @silversquirl !

emidoots commented

2021-12-31 07:55:42 +00:00

(Migrated from github.com)

Thought: One potentially large deciding factor in API design here may be how effectively the ECS, and what it would even look like to have the ECS, integrate with a GUI editor / visualizer of sorts.

👍 2

emidoots commented

2022-01-12 09:26:35 +00:00

(Migrated from github.com)

Looking at Bevy and Flecs, one truly striking thing to me is just how pervasive and complex their ECS systems are. When looking at them, I find it hard to not draw similarities between React in the earlier days (with extensive lifecycle hooks, etc.)

I'm not sure one could adopt ever adopt portions of these systems, they do seem like something you need to buy into 100%. I wonder if there is an opportunity for a simpler, but obviously more "limited", approach.

Just an observation, still a lot here I don't feel confident about.

Bevy

Bevy's ECS ultimately seems to include the following key concepts:

App builder
Worlds
Entities
Resources
Local resources
Components
Component bundles
Systems
- Custom run criterias
- Startup systems
- Per-frame systems
- State machine system
Queries
Query sets
Events
Plugins
System ordering / scheduler
System sets
Hierarchical entities
Change detection
Scheduler stages
Removal detection

Flecs

Flecs seems perhaps more complex, their docs have this interesting diagram:

Looking at Bevy and Flecs, one truly striking thing to me is just how pervasive and complex their ECS systems are. When looking at them, I find it hard to not draw similarities between React in the earlier days (with extensive lifecycle hooks, etc.) I'm not sure one could adopt ever adopt _portions_ of these systems, they do seem like something you need to buy into 100%. I wonder if there is an opportunity for a simpler, but obviously more "limited", approach. Just an observation, still a lot here I don't feel confident about. ## Bevy Bevy's ECS ultimately seems to include the following key concepts: * App builder * Worlds * Entities * Resources * Local resources * Components * Component bundles * Systems * Custom run criterias * Startup systems * Per-frame systems * State machine system * Queries * Query sets * Events * Plugins * System ordering / scheduler * System sets * Hierarchical entities * Change detection * Scheduler stages * Removal detection ## Flecs Flecs seems perhaps more complex, their docs have this interesting diagram: ![image](https://user-images.githubusercontent.com/3173176/149101053-41f99527-1f70-4c82-9afd-3df0f0f0f3e2.png)

👍 1 👀 2

emidoots commented

2022-01-17 02:17:17 +00:00

(Migrated from github.com)

I'll close this issue for now since it's served it's purpose (a collection of people for me to chat with, things to research, etc.)

I sent a very very early stages implementation as well: https://github.com/hexops/mach/pull/156

I'll close this issue for now since it's served it's purpose (a collection of people for me to chat with, things to research, etc.) I sent a very very early stages implementation as well: https://github.com/hexops/mach/pull/156

meshula commented

2022-01-17 06:16:05 +00:00

(Migrated from github.com)

The concepts are much older than twenty years; worth noting in a research thread.

https://www.chrishecker.com/images/6/6f/ObjSys.ppt <- talk by Doug Church about the "Object System" (ECS) in Thief (1998).

Scott Bilas’ Dungeon Siege paper/article that (around 2002). https://www.gamedevs.org/uploads/data-driven-game-object-system.pdf

Sims used a variant. After 2000, they spread rapidly and became endemic through the industry. Diablo 3 used ECS. Star Wars the Force Unleashed used it (Ronin Engine).

The concepts are much older than twenty years; worth noting in a research thread. https://www.chrishecker.com/images/6/6f/ObjSys.ppt <- talk by Doug Church about the "Object System" (ECS) in Thief (1998). Scott Bilas’ Dungeon Siege paper/article that (around 2002). https://www.gamedevs.org/uploads/data-driven-game-object-system.pdf Sims used a variant. After 2000, they spread rapidly and became endemic through the industry. Diablo 3 used ECS. Star Wars the Force Unleashed used it (Ronin Engine).

👍 3 ❤️ 3

emidoots commented

2022-01-17 07:00:59 +00:00

(Migrated from github.com)

@meshula I appreciate that info a ton! I was aware of some but definitely not most of those so quite appreciated :) I'll update my blog post to link to your message as well so others know

Hsad commented

2022-05-04 14:21:24 +00:00

(Migrated from github.com)

Tim Fords GDC talk about ECS in Overwatch, also how their netcode works to appear zero latency
https://www.gdcvault.com/play/1024001/-Overwatch-Gameplay-Architecture-and

Other ECS stuff I bookmarked
https://github.com/skypjack/entt/wiki
https://skypjack.github.io/entt/md_docs_md_entity.html
https://gist.github.com/dakom/82551fff5d2b843cbe1601bbaff2acbf <- overview of how sparse-array backed ECS systems work

(I've spent so much time over so many years banging my head against a wall trying to get some version of open GL working, to be able to just download and build is like having the heavens open up with trumpets and blinding light pouring out. Thanks)

Tim Fords GDC talk about ECS in Overwatch, also how their netcode works to appear zero latency https://www.gdcvault.com/play/1024001/-Overwatch-Gameplay-Architecture-and Other ECS stuff I bookmarked https://github.com/skypjack/entt/wiki https://skypjack.github.io/entt/md_docs_md_entity.html https://gist.github.com/dakom/82551fff5d2b843cbe1601bbaff2acbf <- overview of how sparse-array backed ECS systems work (I've spent so much time over so many years banging my head against a wall trying to get some version of open GL working, to be able to just download and build is like having the heavens open up with trumpets and blinding light pouring out. Thanks)

Rows
Columns

ecs: ECS research #127

Bevy

Flecs