Sorry, you are absolutely right. Getting back to the OP's question, the only issue seems to be how to determine when base64 encoding is needed, and I think OP is correct that you will have to scan the string to check whether it contains any invalid sequences for a UTF representation (either that, or just base64 encoding the data all the time regardless). I couldn't easily tell from
http://lua-users.org/wiki/LuaUnicode whether any of the unicode packages already provide a function to test whether or not a string is a valid UTF encoding.