-
Notifications
You must be signed in to change notification settings - Fork 10
When trying to retrieve data for the previous month, e.g. "last 90 days" period of time in Grafana I receive the following error lines in ticktock db log file:
2023年06月28日 07:53:32.546 [ERROR] [qexe_task_1] Failed to mmap file /mount_point/tt/data/2023/4/1681171200.1681257600/data.0, errno = 12
2023年06月28日 07:53:32.551 [ERROR] [qexe_task_1] Caught exception while performing query.
"Last 30 days" view is working fine and all datapoints show up.
Is there any chance to recover without any backup snapshot available?
All reactions
Replies: 4 comments 4 replies
HI, errno = 12 usually indicates out of memory. Would you mind to share how big is your data and available memory in your device? Is this a 32bit or 64bit OS?
"Last 30 days" view is working fine and all datapoints show up.
Is there any chance to recover without any backup snapshot available?
Your data should be ok in data files. Please try to just zoom in to a specific time range to see if query can succeed.
All reactions
Hi, it is running Raspbian GNU/Linux 10 (buster) 32-bit OS on a Raspberry Pi 3b+, equipped with 1GB and currently 180 MB free RAM.
Please find the disk usage distribution of the data directory:
pi@raspberry:/mount_point/tt/data $ du -sk *
174 1970
11121 2023
4 append.log
1 ticktock.meta
All reactions
Your physical data size is 11MB, not too large. It may be a virtual memory limit problem since it is a 32bit OS. We recently fixed a related bug in version 0.11.8. What version of TickTockDB are you running? You can find that in log when starting.
Would you mind to:
- Try querying, e.g,. just 4/2023 instead of last 3 months, to see if success.
- run:
pmap -x <TT id> > /tmp/pmap.txt, and send the result over.
thanks
All reactions
Currently, I am using version 0.11.1. How to "upgrade" :) ?
- Actually, when looking only for month of April, I realized that I was not precise enough: I am missing data from 2023年04月10日 01:30:00 (last visible data point) onwards.
When trying to display month of May 2023 no data is visualized at all!
But I am able to
visualize from 2023年05月10日 02:00:00 until now.
--> And now I found a problem which started with 2023年05月14日 resulting in a lot of error lines looking identical except the timestamp:
2023年05月14日 03:10:02.608 [ERROR] [tcp_listener_0] fd (273) >= m_min_file (94)
Restart on 2023年05月20日 brought up some log lines like "2023-05-20 21:50:02.184 [INFO] [main] tsdb dir /mount_point/tt/data/2023/2/1675209600.1675209600.back ignored during restore_tsdb()
and a single error line which reoccurred yesterday when I reported this issue:
2023年05月20日 21:50:15.289 [ERROR] [main] Bad line in /mount_point/tt/data/ticktock.meta: RX_bytes
- Output is:
7817: /opt/ticktock/bin/tt -d -c /opt/ticktock/conf/tt.conf
All reactions
Currently, I am using version 0.11.1. How to "upgrade" :) ?
Rebuild new src codes or replace the binary in ./bin.
--> And now I found a problem which started with 2023年05月14日 resulting in a lot of error lines looking identical except the timestamp: 2023年05月14日 03:10:02.608 [ERROR] [tcp_listener_0] fd (273) >= m_min_file (94)
Restart on 2023年05月20日 brought up some log lines like "2023-05-20 21:50:02.184 [INFO] [main] tsdb dir /mount_point/tt/data/2023/2/1675209600.1675209600.back ignored during restore_tsdb() and a single error line which reoccurred
It is ok to ignore .back folders. They are original data before compaction.
yesterday when I reported this issue: 2023年05月20日 21:50:15.289 [ERROR] [main] Bad line in /mount_point/tt/data/ticktock.meta: RX_bytes
Seem corruption in meta file. This worries me.
If possible, would you mind to share your data in ./data folder with me? Let me repro your problem in my side. It would be much easier.
- Output is:
7817: /opt/ticktock/bin/tt -d -c /opt/ticktock/conf/tt.conf
oh, if tt's pid is 7817, I meant to run pmap -x 7817. And send the results to me.
All reactions
Any private and less public possibility (personal message, etc.) to share the data?
All reactions
Please send them to my email yi.lin@ticktockdb.com.
All reactions
@mr-rulez Did you send the data to my email yet?
Another thing you can try is to replace, e.g., 1529280000.1531872000 dir with 1529280000.1531872000.back. And check if the queries can succeed. Basically, an old TSDB (e.g., 1529280000.1531872000) would be compacted. The original data is backup in 1529280000.1531872000.back. We discovered a problem in compaction which may require huge virtual memory. Even though not a problem in 64bit OS, it may fail large queries in 32bit OS since the max vsize is only 4GB (in practice 2.5GB). Note that the bug we fixed in 0.11.8 only fixes incoming data, but it does not adjust old existing data.
yesterday when I reported this issue: 2023年05月20日 21:50:15.289 [ERROR] [main] Bad line in /mount_point/tt/data/ticktock.meta: RX_bytes
I am more worried about this problem.