Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Implement lua-vec - vectors as native Lua datatype #1728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Pirulax wants to merge 7 commits into multitheftauto:master
base: master
Choose a base branch
Loading
from Pirulax:feature/native-vectors

Conversation

@Pirulax
Copy link
Contributor

@Pirulax Pirulax commented Oct 17, 2020
edited
Loading

This PR aims to fix the issue we have with userdata vectors. (#321)
As discussed on discord (and maybe even in the issue itself), using Lua tables would be only marginally better, and the performance benefit would probably be lost when doing table <=> conversion.
Note: I've tested this with bytecode as well (Compiled all default resources, and they all worked perfectly). So no backwards compatibility issues here.

This code(Lua-vec) was "stolen" from here.

I need opinions: Although I've already discussed this with @Woovie, I'd be great if others could give their feedback.
I think naming the datatype as vector would be totally okay (the function table will be named vector in lower case, just so we keep Lua's naming convention. We can just alias it to Vector in a form of a global variable)

Features in lua-vec:

  • Nice indexing of vectors, either with indecies, .x, or even .xxyz, .xx (like in hlsl)
  • Really fast (even compared to Lua table implementation). from 4x up to 10x faster than Vector4.
  • Very memory friendly. Our current Vector implementation is a memory hog([1]). Lua-vec uses no additional memory at all! Its free! [2]

[1]: A Vector4 needs 4 bytes for lua_newuserdata, another sizeof(unsigned long) + sizeof(void*) + sizeof(CLuaVector4D (which is 16 + 4 bytes) in CIdArray
[2]: Its value is stored in a GcObject union. This is where the userdata is stored as well, so, compared to the userdata implementation, we use no additional memory

Todo:

  • Make vec callable (add __call metamethod), right now vectors can only be created with vec.new
  • Make it possible for ArgumentParser and CScriptArgReader to read the new type as a Vector2/3/4D
  • Add support to construct Vector2/3/4 from native vector
  • Add support to construct native vector from existing Vector2/3/4 (only way to implement this cleanly is to use .x .y .z)
  • @qaisjp suggested that it would be nice to have a separate branch to keep track of our modifications to Lua.
  • Implement LUA_TVEC in CLuaArgument (WriteToBitStream and ReadFromBitStream)

Tests:
Resource used: vectest.zip
Scene rendering with smallpt.lua (from Lua-vec):

Command: runsmallpttests 250 250 4 false

Settings: w: 250, h: 250, samples: 4, do_force_gc_collect: false
============[vec]============
Writing to file... took 176 ticks
Tracing took 115081 ticks
Memory used: 3.3 gb
============[stdlua]============
Writing to file... took 291 ticks
Tracing took 371085 ticks
Memory used: <= 3.3 gb
============[Vector4]============
Ran out of memory. Needs around 16 gigs.

Command: `runsmallpttests 250 250 4 true`
Settings: w: 250, h: 250, samples: 4, do_force_gc_collect: true
============[vec]============
Collect garbage total: 19830 ticks
Writing to file... took 165 ticks
Tracing took 103906 ticks
============[Smallpt - Vector4]============
Collect garbage: 25859 ticks
Writing to file: 338 ticks
Tracing: 475007 ticks
============[Smallpt - stdlua]============
Collect garbage: 15044 ticks
Writing to file: 344 ticks
Tracing: 360707 ticks

Syntethic add/create/mul/sub benchmark:

Iterations: 10000000
============[native-vector - GC: OFF]=======
=> create: 2694 ms
=> sub: 1692 ms
=> mul: 1681 ms
=> eg: 535 ms
=> add: 1716 ms
============[native-vector - GC: ON]========
=> create: 2228 ms
=> sub: 1338 ms
=> mul: 1313 ms
=> eg: 507 ms
=> add: 1327 ms
===========================================
============[Vector2 - GC: OFF]============
=> create: 9966 ms
=> sub: 10291 ms
=> mul: 10085 ms
=> eg: 2138 ms
=> add: 9660 ms
============[Vector2 - GC: ON]============
=> create: 30965 ms
=> sub: 21562 ms
=> mul: 21275 ms
=> eg: 5635 ms
=> add: 17737 ms
===========================================
============[Vector3 - GC: OFF]============
=> create: 22923 ms
=> sub: 11093 ms
=> mul: 12490 ms
=> eg: 2196 ms
=> add: 13755 ms
============[Vector3 - GC: ON]============
=> create: 47758 ms
=> sub: 15084 ms
=> mul: 15346 ms
=> eg: 4736 ms
=> add: 13413 ms
===========================================
============[Vector4 - GC: OFF]============
=> create: 14349 ms
=> sub: 31588 ms
=> mul: 24745 ms
=> eg: 2147 ms
=> add: 12531 ms
============[Vector4 - GC: ON]============
=> create: 50227 ms
=> sub: 16102 ms
=> mul: 15704 ms
=> eg: 4504 ms
=> add: 13207 ms
===========================================

4O4, jayceon123, YSAFE, and Fernando-A-Rocha reacted with heart emoji
@qaisjp qaisjp self-requested a review October 17, 2020 15:40
@StrixG StrixG added the enhancement New feature or request label Oct 17, 2020
@StrixG StrixG added this to the Backlog milestone Oct 17, 2020
@Pirulax Pirulax changed the title (削除) Implement lua-vec - vectors are lua datatype (削除ここまで) (追記) Implement lua-vec - vectors as native Lua datatype (追記ここまで) Oct 18, 2020
Copy link
Contributor Author

Pirulax commented Nov 13, 2020
edited
Loading

So, whats people's opinion on this?
I'd be really happy to see this in 1.6.
Also it seems like we could without problems add support for matrices as well. A GCObject is 180 bytes, a regular 3x3 matrix is.. 48, which means that the unions size would stay the same.

@qaisjp qaisjp removed this from the Confirmed Issues milestone Dec 7, 2020
Copy link

ghost commented Dec 9, 2020

I like the changes.

@qaisjp suggested that it would be nice to have a separate branch to keep track of our modifications to Lua.

We can have a submodule for Lua. In this way, it will be easy to track the changes.

Copy link
Contributor Author

Pirulax commented Dec 9, 2020

So, we should create a separate repo?

Copy link

ghost commented Dec 9, 2020

So, we should create a separate repo?

Hmm, that doesn't seem like a good idea. Let's just create a new branch for Lua. I'd do it like this:

  1. Download Lua 5.1 original source and commit the code to the branch.
  2. Copy the Lua source from MTA vendor to the new branch.
  3. And then create a PR for lua-vec changes.

Copy link
Contributor

qaisjp commented Dec 9, 2020
edited
Loading

My approach would be:

  1. Fork Lua repo to multitheftauto org. This keeps original history instead of destroying the Lua 5.1 history, and makes it easier to cherry-pick improvements later.
  2. Cherry-pick our custom 5.1 mods from mtasa-blue to a branch on the Lua repo
  3. Replace vendor/lua in mtasa-blue with a submodule that targets that Lua repo
  4. Create a PR for lua-vec changes in the Lua repo.

This approach is compatible with the submodule system proposed in #319.

Pirulax reacted with thumbs up emoji

Copy link
Contributor Author

Pirulax commented Mar 8, 2021

AFAIK There is no official Lua repo on GitHub, if that's what you meant by Fork Lua repo.
I dont think the Lua history is necessary. I'll just create a new repo with the 5.1 source code from Lua website

Copy link
Contributor Author

Pirulax commented Mar 8, 2021

Okay I think I did it.
Is this what you meant?

Copy link
Member

AlexTMjugador commented Mar 13, 2021
edited
Loading

AFAIK There is no official Lua repo on GitHub, if that's what you meant by Fork Lua repo.
I dont think the Lua history is necessary. I'll just create a new repo with the 5.1 source code from Lua website

There's an official mirror on GitHub of the Lua repo, here. It has a tag for every released Lua version, all the way back to 5.1. I agree that keeping the history may come in handy in the future for some cherry-picks, for updating Lua more easily, or whatever, but I would also understand not keeping the original commit history if it's deemed too verbose.

Copy link
Contributor Author

Pirulax commented Mar 14, 2021

Please look at MTA org's /lua repo.
I have filtered our commits to vendor/lua + added base Lua at the top (source code for 5.1.5 downloaded from Lua website).
Would that fit?

Copy link
Contributor

LosFaul commented Mar 16, 2021

i noticed you call the vector type in Lua only "vec" which is inconsistent to the other types: "number", "table", ...
the type should be "vector"

Copy link
Contributor

LosFaul commented Mar 16, 2021

since Lua uses doubles, shouldn't the vectors not use doubles too?

Copy link
Contributor Author

Pirulax commented Mar 16, 2021
edited
Loading

There would be no benefit of the additional memory required. MTA uses float vectors everywhere, the additiona precision would be lost.
Yeah, the type name will be changed, once this PR gets reviewed or something.. Tbh, we didnt really agree on a name yet. I think vector (perhaps with a capital V) is acceptable. Since it's not vanilla Lua, we might as well stick to our naming convention.

Copy link
Contributor

LosFaul commented Mar 17, 2021

maybe cVector
where c stands for custom
or if someone has better idea he should write it

Copy link
Contributor

LosFaul commented Mar 17, 2021

btw. since my knowledge is not too big about instruction sets
so, only asking if it's possible:
could it be possible that the vectors could somehow utilize something like sse2 for even higher performance?

Copy link
Member

Please look at MTA org's /lua repo.
I have filtered our commits to vendor/lua + added base Lua at the top (source code for 5.1.5 downloaded from Lua website).
Would that fit?

I'm not sure if you were talking to me, but yeah, it looks good to me. I was just pointing out that the original approach that @qaisjp suggested was possible due to the existence of that mirror.

Copy link
Contributor Author

Pirulax commented Mar 17, 2021

Anyways, the submodule stuff #2112

Copy link
Contributor Author

Pirulax commented Mar 17, 2021

The main bottleneck is probably still the Lua side, rather than the actual maths. Not worth the headache. But where I think we might benefit from SSE is our vector stuff.. although I wonder why MSVC doesn't use SSE by default there.. Should be pretty trivial I belive.

Copy link
Contributor Author

Pirulax commented Mar 23, 2021
edited
Loading

Edit: Turns out there's 2 lua-vec branches: One which is a first-class value like a number, bool, etc.. and the second one (this) treats it as GCObject's. Obviously the first one is faster, but comes at the expense of making TValue be 16 bytes instead of 8 (Which translates to basically 2x memory usage for Lua values) and probably performance loss as it would take 4 memory fetches to move the value into the register. The GC'd version is probably already fast enough IMHO. Although it uses some hackery (some kind of free list?).

Copy link
Member

botder commented Jul 25, 2023

@Pirulax A great amount of time has passed, but do you still want to get this pull request merged?

@TheNormalnij TheNormalnij mentioned this pull request Jan 2, 2024
Copy link
Member

This is very useful and would be a good update.
You don't plan to continue dealing with this PR in the future?

Copy link

YSAFE commented Jan 20, 2025

Bump?

Copy link

Rilot06 commented Apr 25, 2025

Any news on this? Would make my current project way easier and faster instead of countless float math operations each time.

Copy link
Contributor Author

Pirulax commented Oct 13, 2025
edited
Loading

I think it can be merged, no?

Edit: Actually nvm, this is just the veclib, i'd still have to change all the parser stuff and whatnot to return this type instead of Vector3...
I don't really have the time to deal with this PR to be honest.
So, if anyone has the time to work on it, then great, we can get this merged... Otherwise it is what it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@qaisjp qaisjp Awaiting requested review from qaisjp

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /