Efficient way to save accelerometer data on sd card

Question

I have analog accelerometer that transfer acceleration on 3 axis with frequency 5 kHz. I want to store that data in SD card for in txt file. My sketch is the following:

void loop()
{
 
 // Get raw accelerometer data for each axis
 int rawX = analogRead(A0);
 int rawY = analogRead(A1);
 int rawZ = analogRead(A2);
 
 // Scale accelerometer ADC readings into common units
 // Scale map depends on if using a 5V or 3.3V microcontroller
 float scaledX, scaledY, scaledZ; // Scaled values for each axis
 if (micro_is_5V) // Microcontroller runs off 5V
 {
 scaledX = mapf(rawX, 0, 1023, -scale, scale); // 3.3/5 * 1023 =~ 675
 scaledY = mapf(rawY, 0, 1023, -scale, scale);
 scaledZ = mapf(rawZ, 0, 1023, -scale, scale);
 }
 else // Microcontroller runs off 3.3V
 {
 scaledX = mapf(rawX, 0, 675, -scale, scale);
 scaledY = mapf(rawY, 0, 675, -scale, scale);
 scaledZ = mapf(rawZ, 0, 675, -scale, scale);
 }
 
 String dataString = "";
 // Concatenate string
 dataString += String(scaledX);
 dataString += "\t\t";
 dataString += String(scaledY);
 dataString += "\t\t";
 dataString += String(scaledZ);
 dataString += "\t\t";
 saveSD(); 
}
// Same functionality as Arduino's standard map function, except using floats
float mapf(float x, float in_min, float in_max, float out_min, float out_max)
{
 return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;
}
void saveSD()
{
 // create file for writing
 File dataFile = SD.open("datalog.txt", FILE_WRITE);
 // if file is available
 if (dataFile) {
 // save the data
 dataFile.println(dataString);
 // close file
 dataFile.close();
 } else {
 // if file is unavailable
 Serial.println("Error opening datalog.txt");
 }
}

So in loop I open file, write to it and after that close it. When my file will be big enough there will be bottle neck.It will take a lot of time to read and write file. How to deal with saving data to files on SD card?

UPD: I measured the time of executing code like that:

 ...
 start_open_t=micros(); // time in milli seconds
 File dataFile = SD.open("datalog.txt", FILE_WRITE);
 delta_open_t=micros() - start_open_t ;
 // if file is available for writing
 if (dataFile) {
 // save the data
 start_write_t=micros(); // time in milli seconds
 dataFile.println(dataString);
 delta_write_t=micros() - start_write_t ;
 // close the file
 start_close_t=micros(); // time in milli seconds
 dataFile.close();
 delta_close_t=micros() - start_close_t ;
 ...

And there is the result:

Open time:
7656
Write time:
2176
Close time:
8076

The open time is increasing there is no surprise (with increasing the file size).
Write to file and close time is almost constant. I thought to open file in setup() and initialize timer. Write to file in loop() and after some time interval to close the file, after that open new file and so on. But as can be seen the print operation dataFile.println(dataString); takes approximately 2 ms that is 500 Hz, and it is too slow. I need to register with 5 kHz. If I print in serial the data is printed much faster but I can not connect to usb because arduino is located to far (approx 30 meters). May be there is the way to connect to serial with wifi or bluetooth or may be there are another methods to store the data fast?

• timemage: I thought about offering this as a potential duplicate, but I didn't feel that strongly about it, and you got some good advice (not mentioned there) below that I'm going to vote up. Still, you may find it an interesting read.
• Kracozebr: Thanks for advice, I'll read
• timemage: You'd edited to include the line: "May be there is the way ... serial with wifi or bluetooth or ... another methods to store the data fast?" It's a valid question (or at least a kernel) but it is a different question that's been shoehorned in there. And that's been done after the original question had already been given a thoughtful answer. IMO, if you want to ask something like that, remove it from this question and start a new one. You'd need to be careful that you're asking something that can be factually answered and not something that is only a matter of opinion.

Answer 1

I can see it is safe to open/close the file every time, especially closing, as if the Arduino loses power, the entire file can be corrupted possibly.

However, I can find of some 'solutions':

1. Backup battery

Open the file in setup(), and only close the file when the backup battery becomes active (instead of closing the file after every write). This will however need an external battery, and some implementation to check if it is active. Also, during the external battery is active, no data should be written.

2. Using multiple files

More easier is to use multiple files, and write each time period (less than the size of the file that cause problems) in a separate file. You can either close the file after each write, or when starting a new period, meaning when the Arduino loses power, you only lose the last time period.

3. Using binary format

This can be used in combination with solution 1 and 2, and is to not write the files as text files but in binary format, reducing the size considerably.

• Kracozebr: Thanks for replying. About battery, for now I'm gonna to use crone battery as main power for arduino because is located away from electrical power. About second your suggestion I thought about it, so I need to measure time of loop and time of opening file with different file sizes and find the max size that it can open not to be too slow to catch data from acceleromerter. And your third suggestion is great I should try it!
• Michel Keijzers: If your file doesn't need to be humanly readable, that is the easiest solution. I think it can save a factor 4 , and maybe more if you use interpunction and spaces.
• Kracozebr: Thanks, I can decode the file after that. I have an update, I measured the time of open, write and close operation and it seems that it not the best way to store logs on sd. See updates
• Michel Keijzers: Use binary format, that will reduce the write time.