-
Notifications
You must be signed in to change notification settings - Fork 7.7k
-
Since this commit we have crashes using the Filesystem. Reverted the PR, Filesystem and SD Card useage is working again. Tbh, not really fully understanding what the PR does.
See comment #6456 (comment) too
@P-R-O-C-H-Y @me-no-dev
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 6 comments 34 replies
-
Hi @Jason2866,
I tested reading different chunks of data to test switching between But its good that we released the 2.0.3-RC1 and all of you are testing that. I can say that the SDWebServer is not working with switching between read/fread or only when using read(). Its works only with fread() (the previous implementation). I'm testing and sorting out why its not working at all. I will appriciate any advices or test cases/sketches to be able to test more scenarios. |
Beta Was this translation helpful? Give feedback.
All reactions
-
Thx for your fast answer! Have you seen the comment / questions from @s-hadinger in the PR?
Regarding a test sketch, it is difficult since the Filesystem and SD Card access is a part of the Project Tasmota and can not be used standalone or extracted. If you have a changed version, we will test asap.
Beta Was this translation helpful? Give feedback.
All reactions
-
Answer for @s-hadinger comment in PR. To be able to discuss everything in one place :)
I have tested this in Arduino and I don't get wrong results as you did, everything is good. Scenario 1 - read 128 + read 128
Scenario 2 - read 128 + read 256
I will give a great example - Thanks to @igrr Now imagine you call fread to read 16 bytes, then call read to read 512 bytes.
So as a result, the first call reads bytes 0-15 from the file, and the next call reads bytes 128-667 from the file, instead of 16-571. Bytes 16-255 are still stuck in FILE* internal buffer. You can only call I have tested different scenarios where its switches between The table below show how bytesinbuf and pointer behave. The read 512 is the only one called that call read directly (size > 128)
|
Beta Was this translation helpful? Give feedback.
All reactions
-
What I don't get why In scenario 2 from @s-hadinger, the second read for 256 bytes actually starts at offset 0x1000 in the file, instead of offset 0x80. I do see a crash later as a hint of corrupt memory. Thats 4kb wrong offset. In the code it should only move from 0 to 128 bytes back, not ahead.
Beta Was this translation helpful? Give feedback.
All reactions
-
I have not rational explanation why I get data from offset 0x1000. Maybe it's just a side effect of some memory corruption.
Thanks for your detailed response.
Beta Was this translation helpful? Give feedback.
All reactions
-
I have found that using LittleFS the buffer size is 4096 bytes. Thats doing the x1000 offset. Working on fix now ;)
Beta Was this translation helpful? Give feedback.
All reactions
-
Thank you for the detailed explanation. But "somewhere" is "something" wrong which generates issues.
We are not the only ones which have problems with this change. Issue #6502 identified this change as reason for the problems too.
Maybe it is a side effect with LittleFS. It is involved in both cases.
Beta Was this translation helpful? Give feedback.
All reactions
-
Yes there is definitely something wrong, I am working on that :)
For what I tested even SDWebServer is not working with the read() implementation as it is now.
I saw you PR - hot fix and seems to be reasonable now till it gets fixed.
Beta Was this translation helpful? Give feedback.
All reactions
-
@Jason2866 @s-hadinger
Can you test this changed implementation of read? No PR yet, want to make sure it works :)
Thank you for testing!
size_t VFSFileImpl::read(uint8_t* buf, size_t size)
{
if(_isDirectory || !_f || !buf || !size) {
return 0;
}
if(size > READ_SIZE_SWITCH)
{
//check some data in buffer exists –> clear buffer and move pointer to deleted data
size_t buf_size = __fbufsize(_f);
size_t bytesinbuf = __fpending(_f);
if (bytesinbuf && (bytesinbuf != buf_size)) //buffer lenght is 128 bytes
{
fpos_t pointer = 0;
fpurge(_f);
fgetpos(_f, &pointer);
lseek(fileno(_f),(-pointer+bytesinbuf),SEEK_CUR);
}
int res = ::read(fileno(_f), buf, size);
if (res < 0) {
// an error occurred
log_d("ERROR OCCURED");
return 0;
}
return res;
}
else
{
return fread(buf, 1, size, _f);
}
}
Beta Was this translation helpful? Give feedback.
All reactions
-
Great! I am gonna clean and comment the code a bit and post a PR :) but i think it still wont work for SDwebserver.. There is some different problem. Will continue on that tomorrow ;)
Beta Was this translation helpful? Give feedback.
All reactions
-
I added logs. Here is the sequence:
- open file
- seek to position
8094
- read
256
bytes
The bytes returned are actually from offset 4008
instead of 8094
(decimal offsets)
Reading the entire file at once does work ok.
Beta Was this translation helpful? Give feedback.
All reactions
-
I added logs. Here is the sequence:
- open file
- seek to position
8094
- read
256
bytesThe bytes returned are actually from offset
4008
instead of8094
(decimal offsets)Reading the entire file at once does work ok.
Will test this scenario tomorrow :)
Thanks for testing.
Beta Was this translation helpful? Give feedback.
All reactions
-
I'm sorry but there is still something wrong with this latest version. It's yet another problem but my zip file parser still fails. It's too late for me do go deeper into it.
Edit: I'm testing with M5Stack_Fire.autoconf in the file system.
Works:
f=open("M5Stack_Fire.autoconf") f.seek(8094) s=f.read(127) f.close() print(s) # shows valid Berry code from `autoexec.be
Does not work:
f=open("M5Stack_Fire.autoconf") f.seek(8094) s=f.read(129) f.close() print(s) # `s` is empty, meaning that the read failed
Beta Was this translation helpful? Give feedback.
All reactions
-
I'm sorry but there is still something wrong with this latest version. It's yet another problem but my zip file parser still fails. It's too late for me do go deeper into it.
Edit: I'm testing with M5Stack_Fire.autoconf in the file system.
Works:
f=open("M5Stack_Fire.autoconf") f.seek(8094) s=f.read(127) f.close() print(s) # shows valid Berry code from `autoexec.beDoes not work:
f=open("M5Stack_Fire.autoconf") f.seek(8094) s=f.read(129) f.close() print(s) # `s` is empty, meaning that the read failed
I see the problem, the f.seek()
function calls internal fseek()
. You can see that in changed read code there is used lseek()
, that because fseek
was not working for read()
(only works for fread
). I will post PR with all changes and add this too. So you can always that the updated PR :)
Beta Was this translation helpful? Give feedback.
All reactions
-
Created new PR #6536 - with all changes. Please if you can test this. Edited file.seek to be working.
Please let me know if there is still some problem :)
Beta Was this translation helpful? Give feedback.
All reactions
-
Actually I suspect
tell()
to be wrong now:f=open("M5Stack_Fire.autoconf") f.seek(8480) d=f.read(32) d=f.read(256) print(f.tell()) f.close() # output: 8795 # correct output should be: 8480+たす32+たす256 =わ 8768
Changed position function to call seek with no offset instead of ftell. Can you try if now the position returned works?
Beta Was this translation helpful? Give feedback.
All reactions
-
Hmmm. It looks to be working but it's super-super slow for a reason I can't explain. There is definitely some nasty side-effect.
When loading LVGL fonts, the reads are done 1 byte per one byte. I.e. a call to
f->read()
for each byte.I suspect a problem in some other APIs.
This is something to take a deeper look at. Thanks for info and testing
Beta Was this translation helpful? Give feedback.
All reactions
-
Not good:
f=open("M5Stack_Fire.autoconf") f.seek(8480) d=f.read(32) d=f.read(256) print(f.tell()) f.close() # prints 8480 -- like if the reads did not happen
Same result for other starting points:
f=open("M5Stack_Fire.autoconf") f.seek(7000) d=f.read(32) d=f.read(256) print(f.tell()) f.close() # prints 8480 as well
Beta Was this translation helpful? Give feedback.
All reactions
-
Its even worse:
f=open("M5Stack_Fire.autoconf") f.seek(32) d=f.read(32) print(f.tell()) d=f.read(224) print(f.tell()) f.close() # prints: # 4128 -- should be 64 # 4352 -- should be 288
I.e. a seek followed by a short read already pushes ahead by 0x1000.
Beta Was this translation helpful? Give feedback.
All reactions
-
Sorry that I did not replied for few days. I'm now working on 2 solutions so there won't be mix of fread/read but only 1 of it. Actually it seems we will stick with same fread implementation as it was before. There will be small change in file buffering causing SD lib to be slow.
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
New PR created #6569.
Beta Was this translation helpful? Give feedback.
All reactions
-
👀 1
-
I confirm all is working well now.
However because of declaring virtual bool setBufferSize(size_t size) = 0;
, it forces me to add this method to my existing code (which I don't need). Not a big deal but it may cause a some other compilation issues.
Why not set a do-nothing implementation by default?
Beta Was this translation helpful? Give feedback.
All reactions
-
Anyways I did the change in Tasmota to make compilation work with this PR.
Beta Was this translation helpful? Give feedback.