Luaste

A structured, strongly typed Lua dialect with Rust-inspired syntax extensions and tooling.

The name is a word play on the Finnish word ruoste, which translates rust, and is a nod to similarly named game engine, Luanti, which is based on the Finnish word Luonti which translates to creation, and which uses Lua as its scripting language. Both ruaste and luanti are common dialectic forms for the words.

If that's too quirky, you can pretend it's short for Lua Strongly Typed Extension (LuaSTE).

What is it?

Luaste is a dialect of Lua that adds useful syntax extensions while compiling down to standard Lua. It comes with a language server implementation for IDE integration and similar niceties.

Initially designed as a more robust replacement for the usual lua scripting backend often used in in-house game engines, its intention is simply to introduce some facilities for safer, stabler lua code that is easier to maintain and extend on larger code bases, while still reaping the benefits that often drive engine devs to lua. And also to get these niceties but also avoid the burden of maintaining either a fork of lua, a custom runtime, or an entirely new script language. This way we can benefit from all the advancements lua makes, without having to keep up in a laborous way.

The code you write compiles to standard lua, so it should work with most existing Lua runtimes and libraries.

Luaste uses AOT compilation (but note that the resulting compiled code is then interpreted or JIT depending on your runtime), so there should not be any runtime overhead. Startup time is slightly increased, but that's the trade-off for a little bit of more safety and a lot of convenience.

There's initial proof-of-concept design for compiling to a native binary via Rust, but this is not yet fully implemented. The end goal is to be able to facilitate optimizing some computationally heavier code paths by compiling them to a fully lua-bridged binary with automatic lua-side glue, though some restrictions and limitations on how the code is written have to be imposed, so this will likely require explicit annotation and some knowledge of the underlying code generation.

⚠️ Early Development ⚠️

This project is in the early stages of development. The language definition and features are still evolving. Expect breaking changes and incomplete functionality.

Features

• Strong Typing: Luaste introduces strong typing to Lua, allowing you to define enforced types for variables, function parameters and return values. This helps catch errors at compile time rather than runtime, and provides more robust real-time feedback to the developer while developing via the lsp implementation.
• Rust-inspired Syntax: The syntax is inspired by Rust, making it familiar to Rust developers while retaining compatibility with Lua ecosystem to some extent. While the syntax is somewhat different from Lua, it is much more sane.
• Native Compilation: The compiler can produce, when explicitly annotated, valid rust code, which can be compiled to an optimized binary. This is not yet fully implemented, but the end goal is to allow procedural codegen to rust with automagic mlua semantics and lua-side glue generation.
• IDE Integration: The language server provides IDE features like code completion, go-to definition, and error checking. Luaste employs a middleware design, which means we can directly benefit from all advancements made by LuaLS without introducing any significant maintenance burden or overhead.
• Compatibility: Luaste code compiles down to standard Lua, so you can, in theory, use it with any Lua runtime or library. As a bonus, there is no burden of maintaining a custom runtime, or a completely custom language server, for that matter.
• Standard Library: Luaste comes with an optional, but recommended standard library that provides useful functions and utilities for common tasks, that employ some minor but not insignificant optimizations and idioms that would not otherwise be feasible in Lua.

Why?

Personally, I was working on my thesis on a data-driven game engine specializing on community content, not just static, but rocedural and dynamic too, and got quickly frustrated with maintenance and extendability of lua code. Lua is a great backend for scripting, especially for familiarity which encourages community involvement. A lot of game modders are familiar with lua, and a lot of in-house engines do use it, like (at least the modern) Civilization games. Roblox is a big one, too, same as Luanti. This encourages and facilitates a much more broader spectrum and amount of community content, and also lowers the threshold to actually take a shot at it for newcomers.

This anecdote besides, there are some pain points with lua for community content creation and maintenance. For one, lacking any strong typing, it's easy to get a lot of different moving parts conflicting and breaking each other. Not only that, but without any kind of static analysis, this is all discoverable only at runtime, which is a pain to debug.

Big modpacks and especially modmods, which are submods intended for some larger mods, really very deeply would improve with interfaces. Without them, and without static analysis, one mod depending on another will unavoidably break when they aren't all maintained and updated in sync (which is of course never the case with community content).

Nullability is tolerable, but you shouldn't have to check on each function you write, whether or not the parameter you got passed in is a string, a table, a number or whatever. On the other hand, if you do not check, you'll get happy little panics at runtime, which is always a treat, and in case of community content, of which there almost always ends up being hundreds and hundreds running at the same time, this is a nightmare to debug.

The goal is not to replace any current scripting backends in play, but simply provide an alternative to those writing an engine, that are expecting their community to write large, complex mods, or maybe small ones, but hundreds of them simultaneously. It's just nicer to have all this, since it really encourages modularization and shared resources such as libraries, and also encourages collaboration on some of the more prominent mods that others depend on a lot.

You should not think of this as something groundbreaking or ambitious. In practice, at the time of writing, this is all just a part of my thesis work, and a small part of it at that. But maybe, eventually, this is something to build up from, and maybe it gets robust enough for some actual use. And, maybe, community content will become slightly less chaotic and just a bit more sane, perhaps.

Concept

Note that all this is subject to change. Initially, the language is designed to be a proof of concept, and as such, it will evolve over time. The goal is to have a working language that can be used in a game engine, but this is the long-term goal. The short-term goal is to have the structs, traits, enums, and other basic features working, the lsp sufficiently robust to be usable, and the compiler able to produce valid lua code.

Note also that this is the initial concept, not implicative of what is currently implemented. Once we reach 0.2 we can set actual examples of what is implemented and what is not, and how the language looks like in practice. For now it's a work in progress, rapidly developed and changing.

Basic structs and function definition:

-- this is intentionally very close to lua, but also to rust.
-- for those only familiar with lua, this can be thought of as a "schema" definition for a table
struct Point {
 -- this is a type annotation, required to ensure no mismatched types on runtime
 -- initially this isn't enforced so well, but ultimately the static analysis will 
 -- get to the point where this is enforced on compile time with good enough accuracy
 x: number,
 y: number,
}
struct Circle {
 -- note that a defined struct can be used as a type
 -- the builtins, like number, string etc. are also structs under the hood
 center: Point,
 radius: number,
}
-- implementation block for the struct
-- you can think of this as batch prepending the struct name to the function name, sort of
-- so the equivalent in a more lua-ish code would be:
-- ```lua
-- function Circle.new(center, radius) ... end
-- function Circle:area() ... end
-- ```
impl Circle 
 -- note the arrow, which is a bit different from lua
 -- it signifies that this method *has* to return a certain type
 -- which helps with stability and safety a lot.
 fn new(center: Point, radius: number) -> Circle
 return Circle { center, radius }
 end
 fn area(self) -> number 
 -- ultimately the goal is to infer this type and allow implicit typing,
 -- but for now (and probably for some time) we require explicit typing
 local radius: number = self.radius 
 return math.pi * radius * radius
 end
end

Traits and impl blocks:

-- traits are similar to interfaces in other languages
-- they define a set of methods that a struct must implement
trait Shape {
 -- note the lack of a definition body
 -- this means it only defines the method signature
 -- implementation is done in the impl block (see below)
 fn area(self) -> number
}
-- impl blocks are used to implement the trait for a struct
-- similar to simply implementing the methods in the struct definition
impl Shape for Circle 
 -- this matches the trait signature, and is as such, valid
 -- we want to enforce this best we can, but initially this can be a bit spotty
 fn area(self) -> number
 -- note that because we know self.radius is a number, and
 -- we know that math.pi is a number, we can infer the type
 -- of the return statement to be a number. nice!
 return math.pi * self.radius * self.radius
 end
end

Mutability and immutability:

-- everything is immutable by default, like in rust. but not as robustly tracked or enforced.
-- an immutable var can not be assigned to again, so it is sort of a const, but not really.
-- this is because it depends on our compiler to be smart enough to keep track of all this,
-- which, initially, isn't the case. consts are inlined and baked into the code as literals on
-- compilation, but this is not the case for immutables. they are just variables we try our
-- best not to allow assigning to again. this is a bit of a lie, but it is what it is.
local x: number = 42
x = 13 -- this causes an error, since x is immutable and can't be assigned to
local mut x: number = 42
x = 13 -- this is fine, since x is mutable, huzzah!

Enums and pattern matching:

-- an enum is not a struct, but a type that can be one of several variants
-- this is similar to rust enums, but not quite
-- to make pattern matching more useful, we allow tuple-like syntax
-- to define a payload for the enum variant
enum Shape {
 Circle(number),
 Rectangle(number, number),
}
fn area(shape: Shape) -> number
 -- this is a pattern match, similar to switches in other languages
 -- but much more powerful, not as robust but similar to how rust does it. 
 -- what we do here is destructure the enum variant,
 -- and _match_ and bind the values to variables
 --
 -- note that initially there has to be an explicit return statement, but ultimately
 -- we want to allow implicit returns, i.e similar-looking-but-not-really-same to rust, where
 -- everything is an expression
 return match shape by
 Circle => math.pi * radius * radius,
 Rectangle => width * height,
 _ => 0,
 end
end

Raw lua semantics (special keyword lua):

-- this is something that can be configured, but because it is basically free and easy,
-- luaste allows you to use raw lua code in your luaste code. Just use the
-- special keyword `lua` to indicate that this is raw lua code, and it will be
-- just passed as-is through compiler without processing
lua fn print_stuff() -- note no return type needed
 local x = 42 -- note no type needed
 local y = 13
 local result = lua return x + y end
 -- the above is redundant here, since the whole body is raw lua, 
 -- but this can be used in luaste code too, similar to rust's
 -- `unsafe` blocks
 print(result)
end

Consts and generics:

-- true constant, baked on compile time
const OFFSET: number = 2.0
-- assuming something like:
trait NumberLike
 fn add(self, other: NumberLike) -> NumberLike
end
-- generics use rust semantics and require trait bounds
-- this is a very much simplified and less powerful version of generics
-- than in rust though
struct Container<T: NumberLike> {
 items: array<T>,
}
-- note we use rust semantics here too, setting explicit bounds again on the impl
-- definition even though it could naively be inferred from the struct definition
impl<T: NumberLike> Container<T> 
 fn new() -> Container<T>
 return Container { items = {} }
 end
 fn add(self, item: T)
 -- note the bound defined again in the impl block definition
 -- ensures our compiler that the :add method exists for T, 
 -- otherwise this would not compile since this isn't safe to assume otherwise
 local _item: T = item:add(OFFSET)
 table.insert(self.items, _item)
 end
end

Private, public, internal, static

struct MyStruct {
 pub field1: number,
 field2: string,
 pub(mod) field3: boolean,
}
impl MyStruct 
 -- no self, so this is a static method
 fn new(field1: number, field2: string, field3: boolean) -> MyStruct
 return MyStruct { field1, field2, field3 }
 end
 -- public is accessible from outside the struct
 -- with no params, it also exports with the module
 pub fn get_field1(self) -> number
 return self.field1
 end
 -- private is accessible only within the struct
 -- note that we use rust semantics and default to private
 fn get_field2(self) -> string
 return self.field2
 end
 -- internal is accessible from the same module, but not exported
 -- note that we use the rust semantics pub(mod) instead of internal 
 pub(mod) fn get_field3(self) -> boolean
 return self.field3
 end
 
end

Native compilation, its limitations and ownership-lite:

-- this is a work in progress, but the idea is to allow native compilation
-- to rust, with some limitations and caveats. let's say we'd want to write this:
native fn process_data()
 local data = create_large_vector()
 
 -- First use of data - this would move ownership in Rust
 send_to_processor(data)
 
 -- Second use of data - this would fail in Rust since data was moved
 local length = data:len()
 
 -- Multiple mutable borrows would also fail
 local worker1 = Worker.new(data)
 local worker2 = Worker.new(data)
 
 worker1:process() -- Both workers try to mutate the same data
 worker2:process() -- This would fail in Rust
end
-- this would not fly in rust. in general, anything we'd write in lua, or in luaste,
-- will very likely just be too ambiguous. So directly compiling to rust is not going to be possible.
-- however, we can try to fill in the gaps as best we can, and hopefully give rust's compiler
-- enough to go from.
-- this means, in specifically native compiled code, the syntax is a bit more verbose.
-- for one, we need to explicitly define the ownership of the variables, and we need to
-- explicitly define the lifetimes of the variables. this is a bit of a pain, but
-- ultimately, you wouldn't want to write this for all of your scripting code. Only the 
-- most resource-intensive parts, where the extra performance is worth the hassle. So, example:
native fn process_data()
 -- variables implicitly own their values until moved
 local data = create_large_vector()
 -- immutable borrowing with & syntax
 send_to_processor(&data) 
 -- alternative method syntax, probably easier to grasp for lua devs
 send_to_processor(data:as_ref())
 
 local length = data:len()
 -- multiple immutable borrows are fine
 local worker1 = Worker.new(&data)
 local worker2 = Worker.new(&data)
 -- for mutable borrowing
 local data_clone = data:clone()
 local worker3 = Worker.new(&mut data) 
 -- or
 local worker4 = Worker.new(data_clone:as_mut_ref())
 -- explicit ownership transfer with move
 local processor = Processor.new(move data)
 
 -- after move, this would fail compilation
 -- local x = data:len()
end

Project Structure

This repo contains three main components:

• luaste-lang: The language definition of Luaste. Contains the language grammar, syntax tree, and compilation logic.
• luaste-lsp: A Language Server Protocol implementation for editor integration
• luaste-stdlib: A standard library for Luaste, providing an idiomatic, common base of useful functions and utilities frequently used in Lua.

Contributing

Contributions are welcome! Feel free to open issues for feature requests or bugs. Pull requests are appreciated.

If you're interested in helping, check out the open issues or just improve something that bothers you. The code is organized to be accessible even if you're new to language development.

Acknowledgements

Formerly known as "Oxilua".

License

This project is licensed under the MIT License. See the LICENSE file for details.