Files not saving even with flush

Question

I have a program that is intended to log the incoming data from a sensor. The sensor is reporting at 200 hz and needs to be logged at least that fast so there are no lapses in data. It is important that the data is preserved even in the event of a power loss. However, if I pull the power from my Pi while the process is running, when I log back in, the file has a size of 0 bytes. Based on information I've gotten from this site and the Unix SE, I've attempted a number of mitigation techniques.

I am using a Raspberry Pi Model 2 B v1.1

What I've tried:

• Opening file with append mode and close after every write
• Flush after every write
• Change /etc/fstab to add commit=1 option

I also tried calling sync() after every write but that limited my logging to ~50hz which is no acceptable for my purposes

Any idea why my log files are not persistent across power cycles?

A sample of the code (c++) in question:

char vecFileStr[] = "VECTORNAVDATA00.CSV";
ofstream vecFile;
int main(int argc, const char *argv[]) {
 // a bunch of stuff
 vs.registerAsyncPacketReceivedHandler(NULL, asciiOrBinaryAsyncMessageReceived);
 // other stuff
}
 `void asciiOrBinaryAsyncMessageReceived(void* userData, Packet& p, size_t index){
 //some processing
 vecFile.open(vecFileStr, std::ofstream::binary | std::ofstream::app)
 vecFile.write(p.datastr().c_str(), PACKETSIZE );
 vecFile.close();
 }

• OyaMist: @goldilocks that seems like an excellent answer. I would have written much the same. Why not just post the answer?
• goldilocks: @OyaMistAeroponics Point taken.
• cat: i am pretty sure that there is no guarantee from Linux or POSIX that your file should be saved in case of catastrophic failure, if the kernel wants to keep your writes buffered then it will do that

Answer 1

You can try adding sync to the mount options in /etc/fstab:

PARTUUID=11eccc69-02 / ext4 defaults,sync,noatime 0 1
 ^^^^

This should be faster than calling it from userspace all the time. Unlike commit, which reduces the amount of time something can be cached before writing, this syncs userspace writes when they happen.

However, that's contingent on you using the C library wrapper on the write() syscall, i.e., something with no userspace buffering. I'm not so sure that ofstream::write() will do that, since it is built on a high level stream object. OTOH, calling ofstream::close() immediately probably does flush any such buffer. [As it turns out -- see comments -- ofstream::write() likely doesn't buffer as the optimal methodology was to use sync with the mount options and to hold the fstream open between write calls.]

Also note that you cannot absolutely prevent filesystem corruption on power cut when writing no matter what you do, because the power to the SD controller and the card itself (which also contains a controller, NAND arrays, etc.) is being arbitrarily cut too.

• Ingo: In addition: I know that on drivers for storages you cat set write through instead of write back but don't know where to do that on a raspi.
• irowe: This worked, but only if I removed the ofstream::open and () ofstream::close() around each write. Having both this plus the sync option degraded performance too much, but just the sync mount option was good.
• goldilocks: Interesting -- I guess then the GNU C++ implementation doesn't add more buffering inside fstream.