Properly escaping very large strings

Question

On an ESP8266 I want to create a little webserver that shows me a dashboard with charts and stuff. For this I want to include the echarts-library as it works offline. To access it the only way I can see this working is that I write:

const char libData[784674] PROGMEM = "the entire 700 KB library goes here";

And then deliver it on request with server.on('./echarts.min.js', handleLibRequest) or something like that.

Now my problem is that in this 700KB of text there are ' and " characters all scattered arround. The C++11 way of R("hard escapable '"'''" string") isn't working in my environment. How would one go about this to include this file in my code?

Is there mybe some way utilize the preprocessor for this? Or maybe an online tool that converts "str" to ['s','t','r']?

• Juraj: serve static files from file system
• Cowboy_Patrick: How would I add these static files into the filesystem? Without having to use any c++ code where I would face the same issues mentioned in my question?
• Juraj: with the IDE plugin for esp8266 FS upload. arduino-esp8266.readthedocs.io/en/latest/filesystem.html

Answer 1

Its simple, paste all you page between "()" on the below line

const char live_html[] PROGMEM = R"rawliteral(your date here)rawliteral";

Answer 2

You can use "objcopy" to produce an object file from the file's contents, if you declare it as binary.

objcopy -I binary echarts.min.js -O <your-bfd-format> echarts.min.js.o

To find out which BFD format you need, run objdump -t <any object file> and look at its output.

The resulting object file contains an object named after the input file in the .data section. There are 3 symbols in the symbol table:

• _binary_echarts_min_js_start
• _binary_echarts_min_js_end
• _binary_echarts_min_js_size

The "start" symbol marks the beginning of the file's content, and the "end" symbol marks the address after the last byte. The "size" symbol is hard to use in C or C++, because it is an absolute value.

Commonly I just use "start" and "end" like this:

extern const char _binary_x_bin_start[];
extern const char _binary_x_bin_end[];

• Edgar Bonet: Instead of objcopy, you can use ld -r -b binary echarts.min.js -o echarts.min.js.o This way you do not need to know the BFD format, provided you use the ld binary appropriate for the target architecture.

Answer 3

You could write a program that escapes all the offending characters and use the output of that in your const char libData[]; see here for a list of escape sequences;

You could serve the library directly from a (pseudo) file system on the ESP8266, without having to copy it "manually" into a variable; you just copy the file to the filesystem;

You could Base64-encode the library and manually copy the result into const char libData[], then Base64-decode before sending;

What I like to do, if there is no file system but enough memory:

• Gzip the library;
• Base64-encode the .gz file;
• Copy the Base-64 encoded result by hand into your const char libData[].

The Base64 encoding allows you to copy/paste without problems.

Despite the Base64 encoding, the gzipping makes the end result smaller than the original, which helps with memory usage and transfer speed, and it will have no "quote problems".

When the libData needs to be served, Base64-decode it, and send the resulting (gzipped) data to the client; all browsers I know will accept that if you add the Content-Encoding: gzip header.

Example code on a raw client without a webserver library: unsigned int sendBase64Page() (mind the content type).

• Cowboy_Patrick: That's like copying it into a variable but unecessarily complicated in my opinion. But thanks for the idea.

Answer 4

I've just written myself now a python-script that creates a char-array with the numbers representing the characters. Maybe it helps someone else:

import sys
def arrayify(sourceFile, targetFile):
 with open(sourceFile, 'rb') as sf:
 contents = sf.read()
 charArray = ""
 for char in contents:
 charArray += str(char) + ","
 # end array
 with open(targetFile, 'w') as tf:
 tf.write(charArray)
if __name__ == "__main__":
 sourceFile = sys.argv[1]
 targetFile = sys.argv[2]
 arrayify(sourceFile, targetFile)