How can a function/method determine if a const array passed in is PROGMEM (flash) or not (RAM)?

Question

Is it possible for a function/method to know if a constant array that has been passed in is in flash or RAM?

If I have a method or function that receives a constant array that is in RAM, the array is declared like this: const uint8_t MY_ARRAY[] = { 1, 2, 3, 4 };

gets passed into a method or function like this: MyFunction(MY_ARRAY);

and the method or function can handle it like this:

void MyFunction(const uint8_t *MY_ARRAY) {
 uint8_t secondElement = MY_ARRAY[1];
 ...

However, if the array is in Flash (because it's a constant array, so that's where it should go), then the declaration adds PROGMEM: const uint8_t PROGMEM MY_ARRAY[] = { 1, 2, 3, 4 };

The pass looks the same: MyFunction(MY_ARRAY);

And the function declaration looks the same but it needs to use pgm_read_byte_near to get the correct values back:

void MyFunction(const uint8_t *MY_ARRAY) {
 uint8_t secondElement = pgm_read_byte_near(MY_ARRAY+1);
 ...

How can the receiving method know if the array is in flash (PROGMEM) or RAM (no PROGMEM) so it knows to use pgm_read_byte_near or not? Ideally I'd like a compiler error, but the type is the same (both are const arrays of uint8_t).

If pgm_read_byte_near is used when it shouldn't be, or it's not used when it should be, the results are garbage.

Some relevant questions: How to pass a static const (progmem) array to a function

RAM usage question: PROGMEM vs const vs #define

Storing array in PROGMEM

PROGMEM: do I have to copy data from flash to RAM for reading?

UPDATE: It looks like what I want to do is not possible. The responses below are very interesting options though: using __flash in plain C, using OOP to make PROGMEM and non-PROGMEM data holders or trying to (somehow?) use non-deprecated progmem specific data types from avr/gpmspace.h.

The best answer to my original question is that the method can not make this determination. The best solution, from a comment by Delta_G, is to create regular (ram) and "_P" (flash/ progmem) versions of the functions and to be as obvious as possible in the naming of those functions.

• KIIV: You can use the same trick as the Arduino classes are using. class __FlashStringHelper; #define F(string_literal) (reinterpret_cast<const __FlashStringHelper *>(PSTR(string_literal))) and function overloading
• Casey: @KIIV That sounds very promising. Can you show me what the function declaration would look like or give some references on how to do that?
• Delta_G: Go look at the Print class in the Arduino core.
• KIIV: void someFunction(const __FlashStringHelper *str); and call it like someFunction(F("soometning"));
• Delta_G: That's what you end up having to do. You release with two versions of the code and a note that if the user wants to use PROGMEM they can but they have to call the _P version of the function or however you name it differently. Often that's easier than trying to have it both ways because it reminds the user right in the name of the function and that sort of forces them back to the docs to see how it needs to be set up.

Answer 1

I am afraid there is no good solution to this problem. One option I do like is to use the __flash qualifier instead of PROGMEM:

const uint8_t ram_array[] = { 1, 2, 3, 4 };
__flash const uint8_t flash_array[] = { 5, 6, 7, 8 };
void function_reading_ram(const uint8_t *array)
{
 uint8_t secondElement = array[1];
 // ...
}
void function_reading_flash(__flash const uint8_t *array)
{
 uint8_t secondElement = array[1];
 // ...
}
int main(void)
{
 function_reading_ram(ram_array); // OK
 function_reading_flash(flash_array); // OK
 function_reading_ram(flash_array); // Warning
 function_reading_flash(ram_array); // Warning
}

Note that, with __flash, you don't need pgm_read_byte_near() or anything similar. You just use the array like you would use any regular array, and the compiler is smart enough to generate the code required to access the Flash memory.

The warnings generated by gcc are:

warning: conversion from address space ‘__flash’ to address space ‘generic’ [-Waddr-space-convert]
 function_reading_ram(flash_array);
 ^
warning: conversion from address space ‘generic’ to address space ‘__flash’ [-Waddr-space-convert]
 function_reading_flash(ram_array);
 ^

Given how nice this solution is, you may wonder why I wrote "there is no good solution". Well, this comes with two caveats, one small and one huge. The small caveat is that you need to explicitly pass the option -Waddr-space-convert to the compiler if you want it to generate the warnings. It is not implied even with -Wall -Wextra. The huge caveat is that this only works in plain C. If you try to use this trick in C++ you get:

error: ‘__flash’ does not name a type

This is an implementation of the "named address spaces" extension to the standard C. Since nothing like this is standardized in C++, the authors of gcc assumed it would not be useful to users of C++. :-( You can mix C and C++ sources in Arduino, but you can't call class methods from C.

• Casey: That's good to know. I've not heard of _flash before. However, I've got classes on my classes, so I'm stuck with C++ on this one.

Answer 2

I will expand on KIIV's comment here.

This is a known issue to the people who wrote the Arduino framework. Essentially, the pointers to PROGMEM and RAM are both data pointers (think of a pointer like an array index, but for the entire RAM/Flash itself rather than one chunk of it that you manage) of the same size and type. They cannot be told apart since pointer types are only determined by the size of the data, not where the data is.

So, they used a clever solution: Make the pointer into a different pointer that isn't compatible with the regular RAM pointer. This is done by casting to the __FlashStringHelper class (and then casting back when it must be read). The FSH class doesn't actually have any methods or do anything; it's simply used to define a custom pointer type that is different from the normal const *.

You can directly go read one of the better guides I've seen on it here (Note that it's for the ESP8266, but this part works the same), or I can try to explain it. I suggest reading it anyway, since it has a lot of other info that's useful but not relevant to this question. Do note that they have a few examples near the end which don't de-allocate memory when they are done using it, though, so copying them directly could cause a memory leak (see my string processing example to see the proper method).

First, you need to cast the array to the FSH class:

const byte MyProgmemArray[] PROGMEM = {4,4,4,4,4,4,4,4}; //Fixes issue #82 (add randomness); see https://xkcd.com/221/ for details :D
__FlashStringHelper * MyArray = (__FlashStringHelper *) MyProgmemArray;

I am unaware if there is any method (and I did look) to declare these in-place, i.e. to make the original array declaration into a __FlashStringHelper * directly. So you still need, as the programmer/user, to know which strings are in flash when you add them, and add the conversion line. This means that what you wanted (fully automatic RAM/Flash detection) isn't quite possible, but this is as close as I think it gets. A further warning is that the original array still exists, meaning that it could accidentally be used instead (will still cause the original mismatch issue) and the new definition takes up slightly more memory for the additional pointer (if the compiler does not optimize it out?).

Oddly, it works fine if your want to store a string:

__FlashStringHelper * MyString = PSTR("yaddayaddayadda...");

Strings can be inlined, but I think it's because of both the PSTR macro and because string definitions might be treated differently than an array of values (even though they are physically the same thing). Perhaps you could experiment with however PSTR works (it's actually a macro that does something, and isn't just a keyword) and see if you can implement this in your own code after all.

---

Now, you still need to get the data OUT of the array in that function you mentioned. To do that, we actually use two versions of that function. This is called overloading (and I think it's C++ only, but this is Arduino, which is by definition C++, so it's safe).

The RAM function you declare as normal:

void MyFunction(const uint8_t *MY_ARRAY){
 byte index_two = MY_ARRAY[1];
 ...
}

The PROGMEM function you declare as follows (assuming that pgm_read_byte_near IS the right function. Also note that it gets more complicated if you need multi-byte values, like int or something...):

void MyFunction(__FlashStringHelper *MY_PROGMEM_ARRAY){
 const byte *MY_ARRAY = (const byte *)MY_PROGMEM_ARRAY; //convert back to normal
 byte index_two = pgm_read_byte_near(MY_ARRAY+1); //still have to read it as flash, of course
 ...
}

---

As a bonus bit of info that is unrelated to the question as asked, but that may interest you/be relevant later:

There are a few more notes if you use this for string data (or pretend it's a string for this part, which works if your data contains no zeros and ends in one). By the latter, I refer to the fact that strings in C (NOT the C++ std::string or Arduino String, though, so keep this in mind) are basically char arrays with a 0 tacked on the end. All string-processing functions run an infinite loop until they read a 0 at their current index, which allows them to work with strings of any length but still know when they are done. As a result, when reading it, you can treat your data like a string, with three caveats:

1. It can never contain a 0, except for the very last item, which won't be read.
2. When defining it, you'd need to still use {} instead of "" since most values don't have printable ASCII characters. (And even so, you'd have to write " " for 32, for instance.)
3. You can't actually print the alleged string, since the data won't actually contain printable characters. You'll get any mix of numbers, letters, punctuation, and fun things like backspace and newline all jumbled together. You would need to read each character and convert it (via, say, String num = <your value here>; or itoa() etc).

If you want to print it using an existing Arduino function, like Serial.print, and it is actually a string (and not data pretending to be a string), it will natively support __FlashStringHelper *, so you don't need the conversion step (this lets you do Serial.print(F("No static RAM used here!")); and similar).

There are two ways to read the data if it is for your own string-processing function, however. The first, the simple one, is not memory efficient and creates a full copy of the data or string in RAM while it's being used. As a result, it is not capable of processing strings that are too large for RAM and may have issues if it's called while a lot of RAM is already in use or it is called deep down a set of nested functions. It is very convenient, however:

//convert from __FlashStringHelper * as above, but let's assume it's a char* here
int size = strlen_P(MY_ARRAY);
if (size!=0){
 char * data = new char[size];
 if (data!=NULL){
 strcpy_P(data,MY_ARRAY);
 //do stuff with the string
 delete data;
 }
}

The second method is more annoying to actually do, and requires more control over how the data is processed, but you can also read out the data byte-by-byte as shown in my first decoding example above using pgm_read_byte_near. This lets you work with data that is too large to fit in RAM and keeps a constant stack size too. The downside is that it requires that you are actually able to stream the data byte-by-byte in all places that it's then used (or that you re-read it when it needs to be accessed out-of-order).

Answer 3

There is also an OO solution possible, using the strategy design pattern, but this comes with some (slight) memory and performance penalties. You need to use C++ for this.

• Put each array you want to differ between flash/SRAM in a separate class (one per type). You get data hiding and possible separation of concerns for free.
• Instead of putting the array in each class, put a reference/pointer or use it directly.
• You can use inheritance:
  • Create a Flash Array class with a SRAM array reader/writer
  • Create an SRAM Array class with a Flash array reader/writer
  • Inherit each of the array types you use from one of these classes where the base class takes care of reading/writing.
• Alternative solution: use the strategy pattern; involves a bit more work, but you can prevent multiple inheritance (although since you use C now, you don't use inheritance (yet)). This would involve roughly:
  • Create a Flash and SRAM Array class reader as above.
  • Instead of inheritance, create a pointer to one of these classes which suits the type of the array.

Although I can imagine this will mean quite some changes in your sketch.

Explanation using Inheritance

(requested by comment)

@Casey I assume you mean the Inheritance way. Assume in a class you have x_array which is of type FlashArray. You just call x_array.Read(...). This method is implemented in FlashArray, but also with the same parameters in RamArray. You can enforce this by inheriting both FlashArray and RamArray by a base class (e.g. ArduinoArray and create a (pure) virtual function Read with the same arguments. Anyway, the implementation of Read method in FlashArray should read from Flash and the same method in RamArray should read from RAM.

When you change the type of x_array from FlashArray to RamArray or vice versa, you don't need to change anything (except the types). To make this easier consider defining a type for x_array, so when you later need to change it, you only need to do this in one location.

• Simon Richter: The advantage of two separate classes is that you can make a per-class policy on data accesses, e.g. "for this class, I would like to copy the data into the current stack frame before working with it". Given the typically small scope of Arduino programs, the code duplication is probably still more readable than a template monster that tries to avoid it. Typical data classes I work with have a "from-flash" constructor, where they copy all the members out of a struct with no methods, and the compiler optimizes out the copies for the members that go unused.
• Simon Richter: Yes, that's the beauty of it: I can change the performance characteristics later on by changing the communication between the "in-flash" and "in-ram" classes, without changing the users of these classes.
• Casey: I think I see how this would work, but one of my goals here is to make the handling method be able to deal with either without relying on the author of the caller to do anything special. My fear is the wrong array type (flash or ram) gets send to a routine and the reader uses the wrong method and produces garbage as a result. Your OO option seems to force a bunch more on the caller side. Do I have this right?
• Michel Keijzers: @Casey I added an example in my answer.
• Edgar Bonet: Or you could have a single class and overload the read() method.

Answer 4

The classic avr-gcc way is to have dedicated functions with different names

e.g. void* memcpy(void* dest, const void* src, size_t n); // Only works if src is RAM

vs. void* memcpy_P(void *dest, PGM_VOID_P src, size_t n); // Only works if src is PROGMEM

BTW: That's mentioned in a comment to the question already by @Delta_G. More about the example memcpy_P here

• Casey: PGM_VOID_P looks very similar to what I'm after. Can I just use that to enforce that the pointer (well, array really) is in PROGMEM? Or, do I need a different version of it that is for uint8_t arrays?
• Casey: I tried replacing my "const uint8_t *" declaration with "PGM_VOID_P" but that doesn't work because they are different types. avr/pgmspace.h has a bunch of PROGMEM data types, such as prog_uint8_t, but they have all been deprecated. pgmspace.h recommends using the attribute during declaration. So, it looks like what I want to do use to be possible, but isn't any longer.