Re: [Experiment] Module system based on static hashing
[
Date Prev][
Date Next][
Thread Prev][
Thread Next]
[
Date Index]
[
Thread Index]
- Subject: Re: [Experiment] Module system based on static hashing
- From: Sean Conner <sean@...>
- Date: 2020年4月16日 20:26:09 -0400
It was thus said that the Great Sean Conner once stated:
> It was thus said that the Great Stefan once stated:
> > Hello,
> >
> > I've conducted an experiment to speed up Lua state creation with static
> > hash tables generated by GNU gperf and got some interesting results.
> >
> > Introduction
> > ============
> >
> > Lua states offer a very light-weight way to execute independent scripts,
> > which is a much desirable feature for programs that execute a large
> > number of them (e.g. web servers).
> >
> > Unfortunately the standard library is too small for many tasks and
> > adding modules by hand is quite a hassle. Furthermore are dynamic
> > libraries a platform-dependent mess.
> >
> > The goal of this experiment was to find a way to add many more functions
> > to Lua a) without using dynamic loading and b) without slowing down the
> > creation of new states.
> >
> > luaL_openlibs loads all functions, tables and values such as print,
> > string, _VERSION, math.pi etc. that make up the standard library into
> > the Lua state so that the script can access them via table lookups.
> >
> > But rarely does a script use ALL of them and the more functions get
> > added, the more unnecessary work luaL_openli has to do.
> > So, the less unused Lua values get loaded into RAM, the better.
> >
> > OK, so what if we don't actually load them and just set a metatable
> > with a __index metamethod that fetches the values as the script needs
> > them? The script won't notice absent values it doesn't use -- Great!
>
> I have a similar issue at work, although it's not speed reasons but
> installation reasons (the less files to install, the better). To that end,
> I created what I call the Kitchen Sink Lua executable, which includes *all*
> the modules we use at work. At startup, I load all the luaopen_*() calls
> into package.preload (these are listed in a luaL_Reg[] arraay) and add a
> special loader to package.searchers to load the modules in Lua (that are
> compiled via luac, compressed [1] and stored in the executable; the loader
> finds the appropriate data and decompresses it when lua_load() is called).
>
> I do call luaL_openlibs() but upon reflection, my Lua state initialization
> code (which is only called once---I don't create tons of multiple states)
> could just be (sans error checking):
>
> static const luaL_Reg preloadtable[] =
> {
> { "coroutine" , luaopen_coroutine },
> { "table" , luaopen_table },
> { "io" , luaopen_io },
> { "os" , luaopen_os },
> { "string" , luaopen_string },
> { "math" , luaopen_math },
> { "utf8" , luaopen_utf8 },
> { "debug" , luaopen_debug },
> /* other pre-installed C-based modules */
> { NULL , NULL }
> };
>
> /*-----------------------------------------------
> ; create our state with the bare minimum required
> ; for a Lua state.
> ;-----------------------------------------------*/
>
> L = luaL_newstate();
> luaL_requiref(L,"_G",luaopen_base,1);
> luaL_requiref(L,"package",luaopen_package,1);
> luaL_getsubtable(L, LUA_REGISTRYINDEX, LUA_LOADED_TABLE);
> luaL_setfuncs(L,preloadtable,0);
> lua_pop(L,3);
>
> This (not checked, just off the top of my head) populates a Lua state with
> the functions NOT in a module (like dofile(), pairs(), select, etc.) and the
> package module, which I believe is the minimum required [2]. This does
> assume that all scripts call require on basic modules like os and io.
> Preloading modules written in Lua is left as an exercise to the reader.
>
> Would something like this work?
So I decided to try this and get some timings. The code is very
simple---the base version basically does:
L = luaL_newstate();
lua_close(L);
The minmin version does:
L = luaL_newstate();
luaL_requiref(L,"package",luaopen_package,1);
luaL_getsubtable(L, LUA_REGISTRYINDEX, LUA_PRELOAD_TABLE); // oops
luaL_setfuncs(L,preloadtable,0);
lua_pop(L,2);
lua_close(L);
(this assumes we can require stuff like pairs(), next(), assert(), etc.)
And the min version is the code I'm quoting above (but loading
LUA_PRELOAD_TABLE instead of LUA_LOADED TABLE, my bad).
And the full version:
L = luaL_newstate();
luaL_openlibs(L);
lua_close(L);
The work was done with Lua 5.3.5 on a slightly older 32-bit Linux system and
the times are in nanoseconds (using clock_gettime(CLOCK_MONOTINIC)). So
with that said, the results of my simple test:
[spc]lucy:/tmp/fl>./bare 10000
new: 19999.100000
lib: 0.000000
close: 5302.000000
memory: 2.000000
[spc]lucy:/tmp/fl>./minmin 10000
new: 23124.800000
lib: 38415.200000
close: 11407.900000
memory: 5.000000
[spc]lucy:/tmp/fl>./min 10000
new: 21497.600000
lib: 60993.500000
close: 14968.400000
memory: 6.000000
[spc]lucy:/tmp/fl>./full 10000
new: 20842.000000
lib: 174061.400000
close: 35268.400000
memory: 14.000000
[spc]lucy:/tmp/fl>
Each was run 10,000 times and the runtime (and memory, obtained from
lua_gc()) averaged. No special compiler options were used but I think this
is enough to give us some ballpark figures. The base (just state creation
and deletion) used 2K and gives us a baseline to compare the rest. The
minmin case (just loading package and leaving the rest to be require()ed)
took almost three times as long and amost 3 times the momory. The min case
took almost four times as long, and 3 times the memory. And the full case
... well ... 9 times longer and 7 times the memory.
-spc (Code available upon request)
> [1] I would get better compression by not pre-compiling the Lua code,
> but the code I have works, and the difference is not enough to
> actually worry about it.
>
> [2] I'm not sure how to make functions like assert(), getmetatable(),
> etc. avaialble via require(). It could be done I suspect, but would
> take a bit of code.