lua-users home
lua-l archive

An update on the LEM Project

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index] 

 A few weeks have gone by since I announced the project, which has some
mild support. I've been working on it since then, and I have made some
progress, or at the very least, gained some new information.
 First off, I wrote the following modules [1]:
	org.conman.zip
	org.conman.zip.read
	org.conman.zip.write
 Those, with the addition of lzlib (and possibly with something like LFS or
org.conman.fsys) means we can read and write about 95% of all ZIP files. 
There is some stuff I don't support (encryption, any of the documented file
extra data, or the 64-bit extensions) but for now, what I have works.
 Now, the modules listed are low level routines, where knowledge of the ZIP
file format is required, but they're all written in straight Lua, with no
other dependencies since these modules don't bother with the
deflating/inflating or even reading/writing data. Nope, these modules just
deal with the ZIP file format.
 I also wrote a lem module [4], which allows you to specify a LEM file and
it will load modules out of said file (or files---you can add multiple LEM
files). There are, however, still some issues.
 First off, a sample LEM file:
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.base64
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.crc
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.env
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.errno
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.fsys
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.hash
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 1.1 MODULES/org.conman.iconv
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.math
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.net
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.pollset
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.process
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.strcore
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 1.0.0 MODULES/org.conman.sys
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.syslog
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.base64
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.crc
Linux x86 LGPL3+ Lua 5.1 5.1 1.0.0 MODULES/org.conman.env
Linux x86 LGPL3+ Lua 5.1 5.1 1.0.0 MODULES/org.conman.errno
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.fsys
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.fsys.magic
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.hash
Linux x86 LGPL3+ Lua 5.1 5.1 1.1.1 MODULES/org.conman.iconv
Linux x86 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.math
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.net
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.net.ipacl
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.pollset
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.process
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.strcore
Linux x86 LGPL3+ Lua 5.1 5.1 1.2.0 MODULES/org.conman.sys
Linux x86 LGPL3+ Lua 5.1 5.1 1.0.2 MODULES/org.conman.syslog
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.tcc
 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.cc
 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.date
 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.debug
 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.dns.resolv
 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.getopt
 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.string
 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.table
 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.unix
 MIT Lua 5.1 5.1 0.10 MODULES/lpeg
 MIT Lua 5.1 5.1 0.10 MODULES/re
Linux x86 MIT Lua 5.1 5.1 0.12 MODULES/lpeg
Linux x86 MIT Lua 5.2 5.2 0.12 MODULES/lpeg
 MIT Lua 5.1 5.2 0.12 MODULES/re
Linux x86 MIT/X11 Lua 5.1 5.1 0.4.work3 MODULES/zlib
 MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket
Linux x86 MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.core
 MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.ftp
 MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.http
 MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.smtp
 MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.tp
 MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.url
 MIT/X11 Lua 5.1 5.1 1.0.1 MODULES/ltn12
 FILES/APPNOTE.TXT
 FILES/COPYING
 FILES/README
 FILES/Miscellaneous Things About Nothing
The Good: I still support multiple architectures, and modules written in Lua
can be loaded directly from the LEM file. Also, only those modules that
match the OS, CPU and Lua versions are loaded, otherwise, they're ignored. 
I also include the version of the module, as well as the license.
The Bad: the modules written in C (so, a DLL or a shared object) *have* to
be written to disk before using. There is no way, short of writing my own
version of dlopen(), to support that. The limitiation is due, I suspect,
mainly due to "oh, we didn't think of that" with a minority of sysadmins
shouting from the wings "IT'S A @#$@#@#$@#$ SECURITY NIGHTMARE! DON'T ALLOW
IT!" [2]. So don't exepect that to change any time soon (Or ever).
The Ugly: I ended up monkey patching loadfile() and dofile(). That came up
as I was packaging LuaRocks into a LEM file:
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.add
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.admin_remove
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build.builtin
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build.cmake
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build.command
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build.make
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.cache
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.cfg
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.command_line
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.deps
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.dir
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.doc
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.download
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.cvs
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.git
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.git_file
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.hg
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.sscm
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.svn
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.lua
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.unix
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.unix.tools
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.win32
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.win32.tools
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.help
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.index
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.install
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.lint
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.list
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.loader
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.make
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.make_manifest
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.manif
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.manif_core
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.new_version
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.pack
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.path
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.persist
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.purge
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.refresh_cache
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.remove
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.repos
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.require
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.search
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.show
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.site_config
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.tools.patch
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.tools.tar
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.tools.zip
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.type_check
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.unpack
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.util
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.validate
 MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.write_rockspec
 MIT/X11 Lua 5.1 5.2 2.1.2 APP/luarocks-admin
 MIT/X11 Lua 5.1 5.2 2.1.2 APP/luarocks
 Lua 5.1 5.1 SCRIPTS/config-5.1.lua
 Lua 5.1 5.2 SCRIPTS/config.lua
 LuaRocks looks for config-VERSION.lua and config.lua, and to get around
that, I had to do the aforementioned monkeypatching. Well, had dofile() not
called the C version of loadfile(), I could have minimized the monkey
patching to just loadfile(), but seeing how dofile() uses the C API, I also
needed to override that function as well.
 But I'm lucky in that LuaRocks only reads these files, and can run quite
nicely from the LEM file [3]:
[spc]lucy:~/projects/LEM>./tstapp.lua search zlib
Search results:
===============
Rockspecs and source rocks:
---------------------------
lzlib
 0.4.work3-1 (rockspec) - http://rocks.moonscript.org
 0.4.work3-1 (src) - http://rocks.moonscript.org
 0.4.work3-1 (rockspec) - http://luarocks.org/repositories/rocks
 0.4.work3-1 (src) - http://luarocks.org/repositories/rocks
 0.4-1 (rockspec) - http://rocks.moonscript.org
 0.4-1 (src) - http://rocks.moonscript.org
 0.4-1 (rockspec) - http://luarocks.org/repositories/rocks
 0.4-1 (src) - http://luarocks.org/repositories/rocks
 0.3-3 (rockspec) - http://rocks.moonscript.org
 0.3-3 (src) - http://rocks.moonscript.org
 0.3-3 (rockspec) - http://luarocks.org/repositories/rocks
 0.3-3 (src) - http://luarocks.org/repositories/rocks
 0.3-2 (rockspec) - http://rocks.moonscript.org
 0.3-2 (src) - http://rocks.moonscript.org
 0.3-2 (rockspec) - http://luarocks.org/repositories/rocks
 0.3-2 (src) - http://luarocks.org/repositories/rocks
 0.3-1 (rockspec) - http://rocks.moonscript.org
 0.3-1 (src) - http://rocks.moonscript.org
 0.3-1 (rockspec) - http://luarocks.org/repositories/rocks
 0.3-1 (src) - http://luarocks.org/repositories/rocks
 Had LuaRocks written to any of the files, then what? And that's where I'm
stuck right now. Running Lua modules directly (for various values of
"direct") works, but trying to run a Lua application from a LEM file might
be asking too much, especially if there are persistent files. As a
distribution format, it's not bad, and perhaps if there was a command to
extract the various modules, scripts and files and write them to appropriate
places that would be the direction to go in.
 One other thing to mention: the meta data, the OS, CPU, license, versions
and what not, aren't stored in a separate file, but are, instead, stored
using an extention method described in the ZIP file format. This extra data
isn't Lua specific and it could be used for other scripting languages. The
fields of this extended file data are:
	version		- version number of the file
	license		- license 
	language	- language (perhaps application, like WoW?)
	lvmin		- minimum version of the language required
	lvmax		- maximum version of the language required
	cpu		- architecture required (for C-based modules)
	os		- operating system required
	osver		- operating system version
 The fields are free-form strings, not longer than 255 characters. You
could easily throw in scripts in other languages, and while I didn't start
out with a universal scripting language application distribution format,
there's nothing in the format that restricts it to just Lua.
 -spc (As always, comments and questions are welcome ... )
[1]	No rockspec yet, as this is still somewhat in flux, but
	https://github.com/spc476/lua-conmanorg/blob/master/lua/zip.lua
	https://github.com/spc476/lua-conmanorg/blob/master/lua/zip/read.lua
	https://github.com/spc476/lua-conmanorg/blob/master/lua/zip/write.lua
[2]	To which I reply, "the computer is running. Game over."
[3]	The code in question:
		lem = require "lem"
		-- ******************************************************
		-- Any module loaded past this point *WILL* come from the
		-- supplied LEM file. No ands, ifs or buts about it.
		-- ******************************************************
		package.path = ""
		package.cpath = ""
		local luarocks,err = lem("luarocks.lem","luarocks")
		if not luarocks then
		 io.stderr:write("luarocks: ",err)
		 os.exit(1)
		end
		luarocks(unpack(arg))
[4]	https://github.com/spc476/LEM/blob/master/lem.lua

AltStyle によって変換されたページ (->オリジナル) /