Run unix command precisely at very short intervals WITHOUT accumulating time lag over time

Question

Question

I'd like to be able to run a UNIX command precisely every second over a long time period.

I need a solution, which does not lag behind after a certain time, because of the time the command itself needs for execution. sleep, watch, and a certain python script all failed me in this regard.

On the microcontroller's such as the http://Arduino.cc I'd do that through hardware clock interrupts. I'd like to know whether there is a similar time-precise shell script solution. All the solutions which I found within StackExchange.com, resulted in a noticeable time lag, if run over hours. See details below.

Practical purpose / application

I want to test whether my network connection is continuously up by sending timestamps via nc (netcat) every 1 second.

Sender:

precise-timestamp-generator | tee netcat-sender.txt | nc $receiver $port

Receiver:

nc -l -p $port > netcat-receiver.txt

After completion, compare the two logs:

diff netcat-sender.txt netcat-receiver.txt

The diffs would be the untransmitted timestamps. From this I would know at what time my LAN / WAN / ISP makes troubles.

---

Solution SLEEP

while [ true ]; do date "+%Y-%m-%d %H:%M:%S" ; sleep 1; done | tee timelog-sleep.txt

Gets a certain offset over time, as the command within the loop also takes a little time.

Precision

cat timelog-sleep.txt
2012年07月16日 00:45:16
[...]
2012年07月16日 10:20:36

Seconds elapsed: 34520

wc -l timelog-sleep.txt

Lines in file: 34243

Precision summarized:

• 34520-ひく34243 =わ 277 timing problems
• 34520/34243 = 1.008 = 0.8 % off

Solution REPEAT PYTHON

Found at: Repeat a Unix command every x seconds forever

repeat.py 1 "date '+%Y-%m-%d %H:%M:%S'" >> timelog-repeat-py.txt

Supposed to avoid the time offset, but fails to do so.

Precision

wc -l timelog-repeat-py.txt
2012年07月16日 13:42:44
[...]
2012年07月16日 16:45:24

Seconds elapsed: 10960

wc -l timelog-repeat-py.txt

Lines in file: 10859

Precision summarized:

• 10960-ひく10859 =わ 101 timing problems
• 10960/10859 = 1.009 = 0.9 % off

Solution WATCH

watch -n 1 "date '+%Y-%m-%d %H:%M:%S' >> ~/Desktop/timelog-watch.txt"

Precision

wc -l timelog-watch.txt
2012年07月16日 11:04:08
[...]
2012年07月16日 13:25:47

Seconds elapsed: 8499

wc -l timelog-watch.txt

Lines in file: 8366

Precision summarized:

• 8499-8366 = 133 timing problems.
• 8499/8366 = 1.016 = 1.6 % off.

• jippie: What is the required resolution, accuracy and why do you need it/what are you using it for?
• Tilo Wiklund: What happens if you nice the process that sleeps?
• mdpc: Have you though of using the REAL TIME scheduling to hopefully minimize the delay on a sleep() call.
• Hack Saw: It strikes me that your timing is always going to be affected by the current operating state of the box. If something causes the program you care about to be unloaded from the cache, you'll get a rise in time, unless you can guarantee that the typical runtime of the program is significantly less than the interval you want. I would want to be on a realtime system, in a significantly stripped down setup with no one else logged on, or in single user mode. Probably a better solution is to modify the program in question to do the loop itself, instead of invoking it from another program.
• Joel Cornett: Launch each command in it's own thread, that way any IO-related blocking does not throw off your time.

Answer 1

Have you tried watch with the parameter --precise?

watch -n 1 --precise "date '+%Y-%m-%d %H:%M:%S.%N' >> ~/Desktop/timelog-watch.txt"

From the man page:

Normally, this interval is interpreted as the amout of time between the completion of one run of command and the beginning of the next run. However, with the -p or --precise option, you can make watch attempt to run command every interval seconds. Try it with ntptime and notice how the fractional seconds stays (nearly) the same, as opposed to normal mode where they continuously increase.

The parameter might not be available on your system, though.

You should also consider what should happen when the execution of your program needs more than one second. Should the next scheduled execution be skipped, or should it be run late?

Update: I ran the script for some time, and it didn't loose a single step:

2561 lines
start: 2012年07月17日 09:46:34.938805108
end: 2012年07月17日 10:29:14.938547796

Update: The --precise flag is a Debian addition, the patch is however rather simple: http://patch-tracker.debian.org/patch/series/view/procps/1:3.2.8-9squeeze1/watch_precision_time.patch

• krlmlr: Exactly the way to go. Wish I could +10 this.
• tylerl: Which version of watch supports that option? It was on none of the machines I checked.
• daniel kullmann: Its version 0.3.0, which is the current version on Ubuntu 12.04. It comes from version 3.2.8-11ubuntu6 of the procps package.
• daniel kullmann: Hmm, the procps source package does not support --precise. This is a Debian addition (3.2.8-9, watch_precision_time.patch)
• math: Ok, but same as mdpc in comments on question stated: This might also fail when your system is under heavy load. I just tested it in conjunction with stress (putting load on disk and cores) and got this: 2012年07月24日 07:20:21.864818595 2012年07月24日 07:20:22.467458430 2012年07月24日 07:20:23.068575669 2012年07月24日 07:20:23.968415439 The real time stuff (kernel etc.) is out there for a reason!

Answer 2

The POSIX ualarm() function lets you schedule the kernel to periodically signal your process, with microsecond precision.

Whip up a simple program:

 #include<unistd.h>
 #include<signal.h>
 void tick(int sig){
 write(1, "\n", 1);
 }
 int main(){
 signal(SIGALRM, tick);
 ualarm(1000000, 1000000); //alarm in a second, and every second after that.
 for(;;)
 pause();
 }

Compile

 gcc -O2 tick.c -o tick

Then attach it to whatever you need done periodically like so:

./tick | while read x; do
 date "+%Y-%m-%d %H:%M:%S"
done | tee timelog-sleep.txt

• math: Do I need a special shell or C-std for that? I've compiled it (which gave a small warning on missing return) but no output was produced.
• Dave: @math With -std=c99, you won't get the warning about the missing return. Otherwise, you shouldn't need anything special. Did you mistype an extra zero? strace ./tick will show you what it's doing from a syscall perspective
• math: I get: gcc -O2 -std=c99 -o tick tick.c tick.c: In function ‘main’: tick.c:10:5: warning: implicit declaration of function ‘ualarm’ [-Wimplicit-function-declaration] tick.c: In function ‘tick’: tick.c:5:10: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] :: Seems that my system (Ubuntu 12.04) does not support it. However at least there is a man page that ualarm should be in unistd.h. (gcc is 4.6.3)

Answer 3

crontab has a resolution of 1 minute. If you're fine with lag time accumulating per that minute and then resetting on the next minute, this basic idea could work:

* * * * * for second in $(seq 0 59); do /path/to/script.sh & sleep 1s;done

Note that script.sh is also run in the background. This should help to minimize the lag that accumulates with each iteration of the loop.

Depending on how much lag sleep generates, there is however the chance of second 59 overlapping with second 0 of the next minute.

EDIT to toss in some results, in the same format as in the question:

$ cat timelog-cron
2012年07月16日 20:51:01
...
2012年07月16日 22:43:00

1 hour 52 minutes = 6720 seconds

$ wc -l timelog-cron
6720 timelog-cron

0 timing problems, 0% off. Any time accumulation resets every minute.

• Izkata: May I ask why this was downvoted?
• hhaamu: It's an ugly hack
• jw013: @hhaamu What is ugly about it? General purpose OS's on PC's are not designed for very precise timing-critical operations, so what more can you expect? If you want "elegant" and absolutely precise timing, you'd have to use a different CPU scheduler, or switch to a real time kernel, or use dedicated hardware, etc. This is a perfectly legitimate solution and I see no reason for any downvotes. It's certainly an improvement on the one that only had the "run in background" without the periodic re-sync'ing via cron.
• Izkata: Plus stopping it is easy. No need to risk killing it in the middle of a cycle - remove the entry from the crontab and it finishes on its own at the end of the minute.
• Dmitry Grigoryev: You're just lucky that on your system cron is precise to the second, but it's not the case in general.

Answer 4

Your problem is that you're sleeping for a fixed amount of time after you run your program without taking in to consideration the amount of time that has elapsed since the last time you slept.

You can do this is bash or any other programming language, but the key is to use the clock to determine how long to schedule the next sleep. Before you sleep, check the clock, see how much time you have left, and sleep the difference.

Because of process scheduling compromises, you are not guaranteed to wake up right on the clock tick, but you should be fairly close (within a few ms unloaded, or within a few hundred ms under load). And you won't accumulate error over time because each time you are re-synchronizing on every sleep cycle and removing any accumulated error.

If you need to hit the clock tick exactly, then what you're looking for is a real time operating system, which are designed exactly for this purpose.

• symcbean: I think it's also very probable that the programs porg has tested block while running the intended process - which logically they should do to avoid killing the machine they are running on.
• tylerl: Whether or not you block, the mechanism works just fine. If you block, you sleep the time left after blocking. If you dont block, then your timing thread or process is sleeping while the other is working. Either way, same result.
• porg: @tylerl: How would the concrete command line look like for your solution?
• porg: I guess you meant the same like @lynxlynxlynx
• tylerl: @porg you need to use date +%S.%N to get the number of seconds with sub-second precision, and usleep to sleep with sub-second precision, but after that it's just a matter of math.

Answer 5

How does this Perl script I just whipped up work?

#!/usr/bin/perl
use strict;
use warnings;
use Time::HiRes qw/time sleep/;
sub launch {
 return if fork;
 exec @_;
 die "Couldn't exec";
}
$SIG{CHLD} = 'IGNORE';
my $interval = shift;
my $start = time();
while (1) {
 launch(@ARGV);
 $start += $interval;
 sleep $start - time();
}

Use: perl timer.pl 1 date '+%Y-%m-%d %H:%M:%S'

It has been running 45 minutes without a single skip, and I suspect it will continue to do so unless a) system load becomes so high that fork() takes more than a second or b) a leap second is inserted.

It cannot guarantee, however, that the command runs at exact second intervals, as there is some overhead, but I doubt it is much worse than an interrupt-based solution.

I ran it for about an hour with date +%N (nanoseconds, GNU extension) and ran some statistics on it. The most lag it had was 1 155 microseconds. Average (arithmetic mean) 216 μs, median 219 μs, standard deviation 42 μs. It ran faster than 270 μs 95% of the time. I don't think you can beat it except by a C program.

• porg: I ran it overnight with no other active user applications with an interval of 1 sec, and it ran for 29241 secs, without a single skipped second! This will fit my purpose. Then I ran it again this morning with an interval of 0.1 sec, GNU date with +%N and after only 3 min it threw that error: Time::HiRes::sleep(-0.00615549): negative time not invented yet at ~/bin/repeat.pl line 23. Line 23 in my saved script: sleep $start - time();
• porg: If you run it with 0.01 sec or 0.001 sec intervals, it is just a matter of a few seconds or less until the program aborts with the "negative time" error. But for my purpose it fits!
• alexis: Easy to avoid the error condition, by skipping the sleep if the difference is negative...

Answer 6

I've always just given up on having something run exactly on interval. I think you'll have to write a C program, and pay very careful attention to not exceeding the portion of the 1-second interval with your own code. You'll probably have to use threading or multiple, inter-communicating processes to get this to work. Take care to avoid thread-starting or process-starting time overhead.

One reference that seems relevant dates to 1993: A Randomized Sampling Clock for CPU Utilization Estimation and Code Profiling You'll want to take a look at the appendix "Adversary Source Code" to see how they accurately measured time intervals, and "woke up" their program at just the correct time. Since the code is 19 years old, it probably won't port directly or easily, but if you read it and try to understand it, the principles involved might guide your code.

EDIT: Found another reference that might help: Effects of Clock Resolution on the Scheduling of Interactive and Soft Real-Time Processes That should help you with any theoretical background.

Answer 7

Take a look at nanosleep() (from http://linux.about.com/library/cmd/blcmdl2_nanosleep.htm). Instead of making your program sleep 1 second, make it sleep (1 - ammount it takes to run) seconds. You'll get a much better resolution.

• Dmitry Grigoryev: You can do the same with regular sleep, i.e. sleep 0.99. The problem is that the amount of time it takes to run is far from constant, even its mean value may fluctuate over time.

Answer 8

Try running your command in the background so it does not affect the loop timing so much, but even that will not be enough if you don't want any accumulation for long periods of time as there is surely a few millisecond cost associated with it.

So, this is likely better, but also likely still not good enough:

while [ true ]; do date "+%Y-%m-%d %H:%M:%S" & sleep 1; done | 
tee timelog-sleep.txt

On my computer this gave 2 errors in 20 minutes or 0,1 per minute, which is roughly a low five fold improvement over your run.

• hhaamu: The problem with sleep 1 is that it is guaranteed to sleep at least one second -- never less. Hence the error accumulates.
• Dmitry Grigoryev: Comparing timing results from two different computers is quite meaningless, unless you have run the original code on your system and obtained the same result as the OP.

Answer 9

Ugly but it works. You should probably rethink the design of your program if you need a loop like this. It basically checks if the current whole second is equal to the previous checked one and prints the number of nanoseconds since change of the second. The accuracy is influenced by the sleep .001.

while true; do T=$( date +%s ); while [[ $T -eq $( date +%s ) ]]; do sleep .001; done; date "+%N nanoseconds late"; done

Accuracy is in the milliseconds, provided that the 'payload' date "+%N nanoseconds late" does not take longer than just under a second. You can lower CPU load by increasing the sleep period or if you really don't mind just replace the sleep command by true.

002112890 nanoseconds late
001847692 nanoseconds late
002273652 nanoseconds late
001317015 nanoseconds late
001650504 nanoseconds late
002180949 nanoseconds late
002338716 nanoseconds late
002064578 nanoseconds late
002160883 nanoseconds late

This is bad practice because you basically make thee CPU poll for an event and you are wasting CPU cycles. You probably want to attach to a timer interrupt (not possible from bash) or use dedicated hardware like a microcontroller. A PC and its operating system are not designed for high timing accuracy.

Answer 10

Another method would be to use a suspend in a loop and send SIGCONT from a precise external program. Sending a signal is very lightweight and will have much less latency than executing something. You could also pre-queue a bunch of commands with the "at" command, hardly anybody uses "at" anymore I'm not sure how precise it is.

If precision is critical and you want to get serious about this, this sounds like the sort of application where typically you would use RTOS, which could be done under Linux with RT-Preempt patched kernel, that will give you the precision and some measure of interrupt control, but it may be more bother than it's worth.

https://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO

Xenomai also might be helpful, it's a full RTOS implementation and is ported for x86 and x86_64, but there's some programming involved.

http://www.xenomai.org/index.php/Main_Page

Answer 11

With ksh93 (which has a floating point $SECONDS and a builtin sleep)

typeset -F SECONDS=0
typeset -i i=0
while true; do
 cmd
 sleep "$((++i - SECONDS))"
done

The same script will work with zsh as well but will invoke your system's sleep command. zsh has a zselect builtin, but with 1/100 resolution only.

Answer 12

I'd go with a small C program:

#include <sys/time.h>
#include <unistd.h>
int main(int argc, char **argv, char **envp)
{
 struct timeval start;
 int rc = gettimeofday(&start, NULL);
 if(rc != 0)
 return 1;
 for(;;)
 {
 struct timeval now;
 rc = gettimeofday(&now, NULL);
 useconds_t delay;
 if(now.tv_usec < start.tv_usec)
 delay = start.tv_usec - now.tv_usec;
 else
 delay = 1000000 - now.tv_usec + start.tv_usec;
 usleep(delay);
 pid_t pid = fork();
 if(pid == -1)
 return 1;
 if(pid == 0)
 _exit(execve(argv[1], &argv[1], envp));
 }
}

This program expects the program to call with full path as its first argument, and passes on any remaining arguments. It will not wait for the command to finish, so it will happily start multiple instances.

Also, the coding style here is really sloppy, and a number of assumptions are made that may or may not be guaranteed by applicable standards, i.e. the quality of this code is "works for me".

This program will get somewhat longer or shorter intervals when the clock is adjusted by NTP or by manually setting it. If the program should handle this, POSIX provides timer_create(CLOCK_MONOTONIC, ...) which is unaffected by this.

Answer 13

You should be keeping track of the current time and be comparing it to the start time. So you sleep a calculated ammount of time each iteration, not a fixed amount. In this way you wont accumulate timing errors and shift away from where you should be because you reset your timings each loop to absolute time from the start.

Also some sleep functions return early if there is an interupt, so in this case you will have to call your sleep method again until the full amount of time has passed.

Answer 14

Here's one that's a bash script and highly accurate. It uses usleep for microsecond precision.

http://wiki.junkemailfilter.com/index.php/How_to_run_a_Linux_script_every_few_seconds_under_cron

There's 3 scrips there. Look at the one at the bottom. Will do up to about 2400 executions per minute with reasonable accuracy. And it's really simple.

Answer 15

This one can run at least 100 times a second with very accurate resolution.

The existence of the directory of the number of loops per minute creates the schedule. This version supports microsecond resolution assuming your computer can handle it. The number of executions per minute does not have to to evenly divisible by 60 nor is it limited to 60 I have tested it to 6000 and it works.

This version can be installed in the /etc/init.d directory and run as a service.

#! /bin/sh
# chkconfig: 2345 91 61
# description: This program is used to run all programs in a directory in parallel every X times per minute. \
# Think of this program as cron with microseconds resolution.
# Microsecond Cron
# Usage: cron-ms start
# Copyright 2014 by Marc Perkel
# docs at http://wiki.junkemailfilter.com/index.php/How_to_run_a_Linux_script_every_few_seconds_under_cron"
# Free to use with attribution
# The scheduling is done by creating directories with the number of"
# executions per minute as part of the directory name."
# Examples:
# /etc/cron-ms/7 # Executes everything in that directory 7 times a minute
# /etc/cron-ms/30 # Executes everything in that directory 30 times a minute
# /etc/cron-ms/600 # Executes everything in that directory 10 times a second
# /etc/cron-ms/2400 # Executes everything in that directory 40 times a second
basedir=/etc/cron-ms
case "1ドル" in
 start|restart|reload)
 0ドル stop
 mkdir -p /var/run/cron-ms
 for dir in $basedir/* ; do
 0ドル ${dir##*/} &
 done
 exit
 ;;
 stop)
 rm -Rf /var/run/cron-ms
 exit
 ;;
esac
# Loops per minute is passed on the command line
loops=1ドル
interval=$((60000000/$loops))
# Just a heartbeat signal that can be used with monit to verify it's alive
touch /var/run/cron-ms
# After a restart the PIDs will be different allowing old processes to terminate
touch /var/run/cron-ms/$$
# Sleeps until a specific part of a minute with microsecond resolution. 60000000 is full minute
usleep $(( $interval - 10#$(date +%S%N) / 1000 % $interval ))
# Deleting the PID files exit the program
if [ ! -f /var/run/cron-ms/$$ ]
then
 exit
fi
# Run all the programs in the directory in parallel
for program in $basedir/$loops/* ; do
 if [ -x $program ] 
 then
 $program &> /dev/null &
 fi
done
exec 0ドル $loops