Macroid

What's the deal?

This project is a prototype. It's no longer in development, as it hit the wall it was expected to eventually. See this repository for the latest work on the Macroid project.

What is it?

It started as a macro-assembler for weird hardware but it can be used to generate any kind of text file you want. I think it's rather nice, and it's already helping tremendously in all sorts of use cases where I need to quickly specify a binary file or generate some repetitive code. It doesn't have thorough documentation beyond this yet, but I am working on it.

It also doesn't have great error messages yet, because I'm rushing on this prototype for a bootstrapping scenario. However, the error messages are at least not nonsense, and you can use WARN to spit out any error and warning messages you like on stderr. It supports a bitmask for enabled warnings, so you can be very granular about which messages you want to look at.

The basic, and only type is a bitstream. A bitstream can be specified with variables a through z, although if its specified in hexadecimal, you can only use g through z because a through f are digits under that regime. It will cause an unresolved bitstream error to do this outside of a macro, or with variables not specified as arguments for the macro.

A bitstream can be interpreted as a number, in which case the first bit is considered the least significant, and the last bit is assumed to repeat forever in a two's compliment scheme. Note this means #01 is 1 and #1 is -1. You can sign extend a bitstream to any number of bits using the = regime explained below, or zero extend with the # regime.

Macro arguments are always a single letter as stated just previously. A macro always begins with a capital letter. After this, any character not used for punctuation as expounded on in Features and Basic Usage below is allowable as part of a macro name. There are a few text files that give usage examples. Expect that to become more organized in the near future.

License

Copyright 2025 Croil (croil@pm.me)

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Overview

Getting Started

Use ./build.sh to build the program macroid or ./build.sh loud if you want a big spew of debugging information that mostly makes sense to this one right now. I promise it's not that hard to follow once you've spent some time bashing against this thing, but you probably don't want that by default.

The CLI is very simple. The input filename can be specified at the end of the arguments without qualification, or it can be specified with the -i option. An output file can be specified with the -o option, or if unspecified, it defaults to stdout. The -w parameter is called the warning mask which is written as a single bitstream formatted as it is in the macro language described below under Features and Basic Usage. This is used by the WARN builtin to filter out categories of messages you don't want in a flexible way.

Reading the below as well as the quick tests in tests/ will hopefully illuminate matters further. At some point soon those should be structured and full documentation will be written as things stabilize. This project is in active development so, knock on wood, I won't be eating those words.

Features and Basic Usage

• Maximum line length is 4 megabytes.
• Top level lines delimited by ;.
• Multiline remarks delimited by (...).
• Multiline macros % with arbitrary fixed number of arguments, each of which has a single lowercase letter name. Note that variables a-f cannot be used in hexadecimal substitutions. Macros end with ;.
• Primitive bit streams (max 2^32 bits under # and = regimes specified either in binary, octal, or hexadecimal):
  • Zero fill substitution under the # regime.
  • Sign extended substitution under the = regime.
• Signed numbers with no maximum number of digits specified either in binary, octal, decimal, or hexadecimal. the usual one's compliment C-like notation is used with - sign to negate the value, but hexadecimal values begin with 0h rather than 0x.
• Representation of signed numbers in any of these bases using the builtin REPR.
• Forwarding regime .. This takes a sequence of variable names in any order and forwards the entire bitstream associated with each, in any given order with any number of repetitions. The . can be omitted as of the time of this writing, but a macro cannot begin with a lowercase letter.
• String regime supporting escape sequences:
  • Hexadecimal escape sequences \ of one or two nibbles. If the next character after the hex escape sequence is also a hexadecimal digit, you can disambiguate by adding a zero, since the maximum number of nibbles is two.
  • Named escape sequences with a one byte name.
    • n or N line feed.
    • r or R carriage return.
    • t or T tab.
    • v or V vertical tab.
    • p or P form feed.
    • ? escape character (033).
    • ! bell character (07).
  • Other characters are literally escaped, including ".
• Invocations by just naming the name table entries of builtins or macros (or, in the near future, stored global values), followed by each substitution parameter prefixed with a colon :. Since all macros have a fixed number of substitution parameters, this is never ambiguous. See some of the examples in tests/ for now, I haven't fully written this up yet and it largely exists in correspondence between a friend of mine and myself. Please note that a valid invocation cannot have spaces before or after the colons separating the arguments. That can be remedied in the future if it is requested or I just feel like it.
• Sublines delimited by {...}. the entire subline is concatenated as one bitstream. sublines allow an entire line to be treated as a single token in a macro parameter list, and inherit their substitution variables if any from the line they are in.
• Conditional expansion with cond ? line-non-zero | line-zero. This works at top level lines and in sublines. This feature enables Turing complete recursion with sentinel conditions, and all sorts of flexible options.
• Labels of the form @"..." and labels with programmatically configurable names @{...}. None of the bytes in the data can be a delimiter, a whitespace character, or zero. This is a current limitation of the key-value store and will be fixed (probably). A label marks the exact position in the bitstream in which it was emitted. A label that is emitted inside of a parameter bitstream has no effect and will issue a warning on stderr. This is because a parameter does not yet have a location in the output bitstream. Positions are given in the number of bits emitted. A label can be recalled using the locate construct &"..." or &{...} from anywhere, including in a paramter bitstream.
• Masked stderr output using the WARN builtin and the -w warning mask option allows CLI level bitmask control over which warning or error messages to display.
• Input and output files specified at CLI using -i and -o options, defaulting to stdin and stdout. The -i file may be specified after the named options without an option name.

A note: I have no plans to implement tail recursion unless it's a popular feature request, so you can't recurse forever. I am more likely to add a loop primitive eventually with the same kind of special subline syntax as for conditional sublines.

Builtins

• COUNT:x number of bits in x.
• TAKE:n:x take the bits from x up to and not including the nth bit.
• DROP:n:x take the bits from x starting from the nth bit up to the end.
• REP:n:x repetition of x n times.
• NOT:x bitwise negation of x.
• REV:x reversed bits of x.
• TEST:x 1 if x is non-zero, otherwise 0.
• SAME:x:y 1 if bitstreams x and y are the same lengths and all bits match, otherwise 0.
• GET:i:x get the ith bit of x. If i is negative, bits are counted from the end. If the position represented by i is greater than the most significant bit, the most significant bit is returned. If the position represented by i is less than zero, zero is returned.
• SLICE:i:j:s:x take bits between index i and index j with step s from x. If i or j is negative, they count from the end of the bitstream. If s is negative, then bits are taken in reverse. If i refers to a greater index than j, no bits are taken. If any position taken is greater than the position of the most significant bit, the most significant bit is given for that position. If any position taken is less than zero, zero is returned.
• CMP:x:y 0 if x and y are numerically equal, -1 if x < y, and 1 if x > y.
• LT:x:y 1 if x is numerically less than y, otherwise 0.
• GT:x:y 1 if x is numerically greater than y, otherwise 0.
• LE:x:y 1 if x is numerically less than or equal to y, otherwise 0.
• GE:x:y 1 if x is numerically greater than or equal to y, otherwise 0.
• EQ:x:y 1 if x is numerically equal to y, otherwise 0.
• NE:x:y 1 if x is numerically not equal to y, otherwise 0.
• SHIFT:x:n x is arithmetically shifted by n bits, preserving sign. If n is negative, this is a right arithmetic shift by -n bits, otherwise it's a left arithmetic shift by n bits. n must be a no more than a 33-bit signed value, since the maximum length of a bitstream is the maximum value of an unsigned 32-bit integer.
• ADD:x:y the arithmetic sum of x and y, preserving sign.
• SUB:x:y the arithmetic difference of x and y, preserving sign.
• MUL:x:y the arithmetic product of x and y, preserving sign.
• DIV:x:y the euclidean quotient of x and y, preserving sign.
• REM:x:y the euclidean remainder of x and y preserving sign.
• AND:x:y the bitwise conjunction of x and y, extending sign.
• OR:x:y the bitwise disjunction of x and y, extending sign.
• XOR:x:y the bitwise exclusive disjunction of x and y, extending sign.
• NEG:x the arithmetic negation of x, preserving sign.
• INC:x the successor of x, preserving sign.
• DEC:x the predecessor of x, preserving sign.
• NUM:x the signed numerical value of x, discarding redundant sign bits.
• ABS:x the absolute numerical value of x, negating as necessary and discarding redundant zeroes.
• SIGNUM:x the numerical sign of x.
• REPR:b:x the sequence of ASCII bytes representing the signed numerical value of x in base b. this is homoiconic w.r.t. the numerical regimes.
• SHOW:x the sequence of ASCII bytes representing the given bitstream, including unresolved variables if any.
• WARN:m:x send x to stderr after packing the bits into a string of bytes. m is the user specified warning type mask. A user may specify a warning enable mask using the -w CLI option where the argument is a fully resolved bitstream of one of the forms #, #h, or #o. If the warning type mask and the warning enable mask share any set bits, the x is sent to stderr, otherwise it is not. WARN will not send any bits to the output bitstream.
• DIE:c immediately exit with exit code c without flushing the output bitstream. The current top level line will not be written.
• STOP:c exit with exit code c after ensuring all bits sent to the output bitstream when STOP is evaluated are written to the output.
• HERE takes no parameters and gives the current position in the output bitstream.

TODO

• Implement invocation namespacing (feature request).
• Reorganize the DIE macro and the global stop condition so that it can be set from any part of the program. This is needed to cleanly avoid a segfault on error.*
• Input regime <...> allowing an implementation defined input bitstream specification from arbitrary sources. On UNIX we will simply use files.
• Store regime !K:v for storing global variables in the name table.
• Pragmatic regime ?K:v regime for global behavior controls.
• Implement macro source modules with namespacing???

* It should be the case if I'm not mistaken that this issue is resolved, but I will need to more carefully investigate. If I'm not mistaken, it was always caused by the fact that the output bitstream could be reallocated if it grew too large, and because the previous implementation of extend_bits() did not guarantee that a bitstream's base address would never change unless it was deallocated or allocated for the first time (given an input of NULL), it was not certain that pq->bs pointed to a valid heap address in the event that the exit handler was called in the middle of parsing a line. Given that guarantee, by ensuring that the output stream exists from the start we can be certain that it is always okay to free the output stream.