Reading a 10-byte buffer out of a PROGMEM string stored in a string table

Question

I'm storing a large snippet of text in three separate PROGMEM char arrays:

const char script1[] PROGMEM = "....";
const char script2[] PROGMEM = "....";
const char script3[] PROGMEM = "....";

The first two are 32766 characters, and the last one is 23867 characters (the whole string is 89400 characters in total). I store these in PROGMEM on an Arduino Mega 2560 (which then uses about 90K of 256K of available program memory).

I stored these three strings in a table, together with their lengths:

const int TABLE_SIZE = 3;
const char * const scriptTable[] PROGMEM = 
{ 
 script1,
 script2,
 script3
};
const int scriptLengths[] {32767, 32767, 23867};

I have some issues trying to chunk up the string into 10-byte buffers, and processng it that way. I want to iterate through the table, process the first part in 10-byte chunks first, then move on to the second part, and finally the last part.

For a single string (code), the following works fine:

const char code[] PROGMEM = "....";
// Chunk up large code string in PROGMEM.
const int CHUNKSIZE = 10; // CHUNKSIZE *must* be an even number!!
char workingSpace[CHUNKSIZE + 1];
int baseIndex = 0;
while (baseIndex < sizeof(code)) {
 int chunk;
 if (baseIndex + CHUNKSIZE > sizeof(code)) {
 chunk = sizeof(code) - baseIndex;
 }
 else {
 chunk = CHUNKSIZE;
 }
 memcpy_PF(workingSpace, ((uint_farptr_t)code) + baseIndex, chunk);
 workingSpace[chunk] = '0円';
 // for debugging
 Serial.print("#");
 Serial.print(baseIndex);
 Serial.print("– Got a chunk [");
 Serial.print(workingSpace);
 Serial.println("]"); 
 baseIndex += CHUNKSIZE;

However, I'm having some difficulties doing the memcpy_PF call using the table.

int tableIndex = 0;
// for each of the three script strings
for (tableIndex = 0; tableIndex < TABLE_SIZE; tableIndex++) { 
 // Chunk up large script strings in PROGMEM.
 const int CHUNKSIZE = 10; // CHUNKSIZE *must* be an even number!!
 char workingSpace[CHUNKSIZE + 1];
 int baseIndex = 0;
 // we can only use sizeof if it's a static var.
 // strlen should only be calculated once, expensive operation (loop until we encounter '0円').
 // now stored statically in scriptLengths table.
 Serial.println("tableIndex: ");
 Serial.println(tableIndex); 
 int strLength = scriptLengths[tableIndex];
 Serial.print("length script string: ");
 Serial.println(strLength);
 while (baseIndex < strLength) {
 int chunk;
 if (baseIndex + CHUNKSIZE > strLength) {
 chunk = strLength - baseIndex;
 }
 else {
 chunk = CHUNKSIZE;
 }
 // do memcpy_PF call here

I have tried several options here, mostly having difficulties with the second parameter (how to address the string stored in the table, and increase its indexing parameter). I read that the way to get a string out of a string table is to use pgm_read_word. However, with both of these below, I either get gibberish in the buffer string, or it seems to start in an incorrect location in the string.

memcpy_PF(workingSpace, ((uint_farptr_t) &scriptTable[tableIndex][baseIndex]), chunk);
memcpy_PF(workingSpace, ((uint_farptr_t) (char*)pgm_read_word(&(scriptTable[tableIndex])) + baseIndex), chunk); 

Any help would be greatly appreciated!

Thanks!

Answer 1

The first thing you need to fix is this:

/home/nick/Arduino/Large_Progmem_test/Large_Progmem_test.ino: In function 'foo()':
/home/nick/Arduino/Large_Progmem_test/Large_Progmem_test.ino:42:27: warning: iteration 3276 invokes undefined behavior [-Waggressive-loop-optimizations]
 baseIndex += CHUNKSIZE;

You are adding 10 to 32760 hoping to be>= sizeof(code) however now baseIndex will become negative as an int goes from -32768 to +32767.

---

There were some interesting subtleties to getting this working. :)

One of your major problems is this:

const char * const scriptTable[] PROGMEM = 

That will give you 2-byte pointers (const char *) which will not be large enough to index into 89400 characters.

So you need to have an array of 4-byte pointers like this:

uint_farptr_t scriptTable [TABLE_SIZE] =

However you can't just initialize those like you tried to. That has to be done procedurally with a macro that pulls in the full address. My working code is below:

#include "code.cpp"
const int TABLE_SIZE = 3;
uint_farptr_t scriptTable [TABLE_SIZE]; // values calculated at runtime
const int scriptLengths[TABLE_SIZE] 
 {
 sizeof (script1), 
 sizeof (script2), 
 sizeof (script3)
 };
void foo ()
{
 // for each of the three script strings
 for (byte tableIndex = 0; tableIndex < TABLE_SIZE; tableIndex++) 
 {
 // Chunk up large script strings in PROGMEM.
 const int CHUNKSIZE = 10; // CHUNKSIZE *must* be an even number!!
 char workingSpace[CHUNKSIZE + 1];
 long baseIndex = 0;
 Serial.println ("----------------");
 Serial.print("tableIndex: ");
 Serial.println(tableIndex);
 long strLength = scriptLengths[tableIndex];
 Serial.print("length script string: ");
 Serial.println(strLength);
 while (baseIndex < strLength) {
 int chunk;
 if (baseIndex + CHUNKSIZE > strLength) {
 chunk = strLength - baseIndex;
 }
 else {
 chunk = CHUNKSIZE;
 }
 memcpy_PF(workingSpace, scriptTable [tableIndex] + baseIndex, chunk);
 workingSpace[chunk] = '0円';
 // for debugging
 Serial.print("#");
 Serial.print(baseIndex);
 Serial.print("- Got a chunk [");
 Serial.print(workingSpace);
 Serial.println("]");
 delay (50);
 baseIndex += CHUNKSIZE;
 } // end of while
 } // end of for
} // end of foo
void setup() 
{
 // get proper pointers to the various pieces of progmem
 scriptTable [0] = pgm_get_far_address (script1);
 scriptTable [1] = pgm_get_far_address (script2);
 scriptTable [2] = pgm_get_far_address (script3);
 Serial.begin (115200);
 foo ();
}
void loop() 
{
}

The file "code.cpp" just has the actual data in it (generated at http://www.lipsum.com).

As in:

#include <Arduino.h>
const char script1[] PROGMEM = { 
"Lorem ipsum dolor sit amet, consectetur adipiscing elit. ... " };
const char script2[] PROGMEM = { 
"Aliquam erat volutpat. Praesent aliquet accumsan arcu, ... " };
const char script3[] PROGMEM = { 
"Ut neque dolor, condimentum ut odio iaculis, ... " };

Output:

----------------
tableIndex: 0
length script string: 32767
#0- Got a chunk [Lorem ipsu]
#10- Got a chunk [m dolor si]
#20- Got a chunk [t amet, co]
#30- Got a chunk [nsectetur ]
#40- Got a chunk [adipiscing]
#50- Got a chunk [ elit. Pro]
#60- Got a chunk [in sit ame]
#70- Got a chunk [t purus cu]
#80- Got a chunk [rsus, dict]
#90- Got a chunk [um magna q]
#100- Got a chunk [uis, place]
#110- Got a chunk [rat dui. I]
#120- Got a chunk [nteger ali]
...
#32700- Got a chunk [, eget tem]
#32710- Got a chunk [por dui te]
#32720- Got a chunk [mpor a. Do]
#32730- Got a chunk [nec malesu]
#32740- Got a chunk [ada mi et ]
#32750- Got a chunk [dolor cras]
#32760- Got a chunk [ amet.]
----------------
tableIndex: 1
length script string: 32767
#0- Got a chunk [Aliquam er]
#10- Got a chunk [at volutpa]
#20- Got a chunk [t. Praesen]
#30- Got a chunk [t aliquet ]

It successfully transitioned from the first array to the second one.

• jozilla: Good point, I missed that, thanks! Probably best to limit the size of script1,script2,script3, etc. to avoid overflows (e.g., to 30.000)?
• Nick Gammon: Or use a long as an index.
• Jot: Maybe the avr-gcc compiler can not handle a PROGMEM section beyond 64kbytes. With a test sketch I am able to declare a lot of data (150kbytes), but the pointers in the Table[] stay 16-bit. They should be forced to 32-bit, but I don't know how. Even then, the memcpy_PF might not read beyond the first 64kbytes of the PROGMEM segment. So far I can't even read script3 directly with a function (when it is at the end of 150kbytes data).
• Nick Gammon: See my amended answer.
• Jot: I have tried a few things, and it it possible to create a 32-bit list of pointers with "const uint_farptr_t Table[] PROGMEM = ", but filling them during initialiazation will not fill them with the proper 32-bit addresses. I can't find another way to make it work than the workaround by Nick Gammon. The memcpy_PF() can retrieve data from PROGMEM that is beyond the 64kbytes border. It is the table that causes the problem. Also sizeof() and declaring too much data in a single array will cause trouble beyond the 64kbytes border, sometimes there is not even a compiler warning.