Why do i get additional characters when reading a file through a buffer if buffer size > 15?

Question

This is a follow up question to How to speed up writing a file to a WifiClient?

I modified the old code to read from one Stream and write to another which looked like this and worked fine albite being a bit slow

while (file.available() > 0)
{
 client.print((char)file.read());
}
client.flush();

to use a buffer, like so

char buffer[300];
size_t bufferSize = sizeof(buffer);
// ...
while (file.available() > 0)
{ 
 memset(buffer, 0, bufferSize); 
 file.read(buffer, bufferSize);
 client.print(buffer);
}
client.flush();

Now, if the buffer size is> 15 (in the above case, it's 300), two additional characters Øb are introduced after the buffer size amount of characters. If the buffer size is <= 15, this does not happen.

If the file to be read contains abcdefg repeated several times, I get the following result for a buffer size of 32:

|-------- 32 characters -------|Øb|-------- 32 characters -------|Øb|-----
abcdefgabcdefgabcdefgabcdefgabcdØbefgabcdefgabcdefgabcdefgabcdefgaØbbcdefg

What's causing those additional characters to appear?

The above output is produced in the browser If I do not specify a charset. if I do, I get � instead of Øb.

Answer 1

Try recording the actual number of bytes read and only write that number of bytes:

size_t nr = file.read(buffer, bufferSize);
client.write(buffer, nr);

• cross: That worked, btw. what's the matter with the downvotes? How can I improve my questions?
• Majenko: @cross Who knows? This place can be a bit fickle sometimes...
• Andrew: Since you are doing this for speed drop the memset() it's a waste of CPU cycles in this situation
• Majenko: You are quite right there. I have removed it from my example for clarification too.

Answer 2

If the buffer is full of data then there is no null terminator on your data to output, the client.print will keep going past the end of the buffer until it hits a 0. Because of the way the compiler is placing things in memory you happen to always get 0b and nothing else (does your code happen to have some other variable with a value of 11?)

There are two solutions: Either only output the number of bytes you read (in which case the memset instruction is no longer needed) or limit the file.read to one less than your buffer size so that there is always a 0 left at the end of the buffer.

const int buffersize = 32; 
char buffer[buffersize];
while (file.available() > 0)
{ 
 int bytes = file.read(buffer, bufferSize);
 client.write(buffer,bytes);
}

or

while (file.available() > 0)
{ 
 memset(buffer, 0, bufferSize); 
 file.read(buffer, bufferSize-1);
 client.print(buffer);
}

Also it's more a personal preference than a requirement but I don't like to use sizeof(<array>), it only works within the same scope as the array definition and doesn't work if the array is passed to a function. This site is full of examples of people getting caught out by that and wondering why their array is always giving them a size of 4. So while it works fine in this instance I avoid using it if at all possible.

• cross: I am aware of the pitfalls of sizeof and had tried it with a macro definition to double check, but the result was the same. I will try your suggestions regarding the null termination. The question still stands: Why does this only happen if bufferSize > 15?
• Andrew: If the issue does turn out to be a lack of termination (the second option is the best way to prove that) then the reason is that the behavior without the termination is completely undefined. It could well be that with a smaller buffer the compiler placed things in memory in a different way such that there always happened to be a 0 in the byte after the buffer.