How to get all processes, currently running in kernel mode?

Question

How can I list processes that are busy in a system call? Is there a way to find it out with the command top? I couldn't find any appropriate option in the man pages.

Answer 1

The state of a process indicates what it's doing. The main process states are:

• S: sleeping, i.e. in a system call and blocked waiting for something. The process can wake up if it has something to do.
• D: busy, waiting for hardware – uninterruptible sleep. The process can't wake up, it has to wait for a specific event to happen. (Sometimes this can be cancelled, but not always.)
• R: running, i.e. executing code. This is usually application code (so running in user mode), but some computations performed inside the kernel also leave the process state on R.

Computations performed inside the kernel can be in state R or D. I think it depends on whether those computations are using resources (e.g. whether the code has a lock held). I don't think there's a portable direct way to distinguish between R-in-user-mode and R-in-kernel mode, but maybe there's a way in Linux's /proc somewhere.

The top version in Linux procps lists the process state. It doesn't seem to have an option to list only processes in state D, but you can use the i key or the -i command line option to hide all processes that remained idle since the last screen update, which will typically leave less than a screenful.

If you just want to list the PIDs, you can filter the output of ps.

ps -o state=,pid= | sed -n 's/^D //p'

On a system that isn't doing heavy I/O, it's to be expected that most of the time, this will list zero processes.

• Alex: Excuse me. Is the answer true? If we launch $ strace dd if=/dev/urandom of=/dev/null bs=30M count=1000 we will see that all the time read() and write() syscall happens. $ top will show that the process has 'R' status all the time. $nmon will show that 99-100% of cpu load is 'kernel mode load'. So i suppose your answer is not correct.
• Alex: Also (as proof ), if you launch at the second terminal $ awk '{print 14,ドル 15ドル}' /proc/$$/stat you will something like: 0 3915. The first number shows 'Amount of time that this process has been scheduled in user mode, measured in clock ticks' , the second one shows 'Amount of time that this process has been scheduled in kernel mode, measured in clock ticks'. The first number will not be increasing but the second number will be increasing constantly that proves that the process will be working in kernel mode and the state of the process will be 'R' but not 'D' as you stated in the answer.
• Gilles 'SO- stop being evil': @Alex Indeed, you're right, Linux seems to keep the process in state R when it's doing computations inside kernel code, as long as it isn't waiting for hardware. I don't know what the rules are for that (for example, would the process state remain R or D if the crypto of urandom was performed by a hardware accelerator?).

Answer 2

It is not 100% beatifull answer but it can give you a flavour. If you have some process that constantly in 'R' state - you can start to monitor two fields from procfs:

$ awk '{print 14,ドル 15ドル}' /proc/$$/stat 

you will see something like: 0 3915

The first number shows 'Amount of time that this process has been scheduled in user mode, measured in clock ticks' , the second one shows 'Amount of time that this process has been scheduled in kernel mode, measured in clock ticks'. (please have a look at man proc for the details).

However the point is if 3915 is growing fast and 0 is not growing it means the process is running under kernel mode right now. The more fast 3915 is growing the more we can be sure that the process is running under kerhel mode.

an example:

$ sudo dd if=/dev/nvme0n1p2 of=/dev/null bs=30M count=1000
 $ top
 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
16691 root 20 0 45352 32712 2112 R 50,5 0,2 0:06.00 dd
 
$ awk '{print 14,ドル 15ドル}' /proc/16691/stat 
0 467
$ awk '{print 14,ドル 15ドル}' /proc/16691/stat 
0 512
$ awk '{print 14,ドル 15ドル}' /proc/16691/stat 
0 557
$ awk '{print 14,ドル 15ドル}' /proc/16691/stat 
0 594
$ awk '{print 14,ドル 15ドル}' /proc/16691/stat 
0 630
$ awk '{print 14,ドル 15ドル}' /proc/16691/stat 
0 666
$ awk '{print 14,ドル 15ドル}' /proc/16691/stat 
0 699

So we can say - yes, the process is running under kernel mode.

As for 'D' state:

As for 'D' state (correct me if i'm wrong) - it means the process in a "sleep" state. It is 'uninterruptible sleep' state anyway it is a sleep state , it means the code of the process (user space) or invoked kernel code via syscall IS NOT scheduled on cpu until some necessary data\structure is available. So i suppose the processes in 'D' state should be excluded from the review. Why? Because they just not executed on cpu at all. However there is a subtle moment. The process can switch between 'D' and 'R' states quickly so we can think the process is in 'D' state however it goes from time to time to 'R' state.

Let me give explanation in details: Very often people say that if a process in 'D' state it means it waits some I\O. It is not necessarily so.

Simple program in C.

$ cat 30.c
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <unistd.h>
#include <sys/wait.h>
int main() {
 pid_t pid = vfork();
 if (pid == 0) {
 sleep(180);
 return 0;
 }
 printf("parent: I am exiting\n");
 return(0);
}

Compile.

$ gcc -o 30.exe 30.c

Launch.

$ ./30.exe

It uses vfork that creates a child process. The parent process will be blocked until child exits. Also the state of parent process will be 'D'.

$ ps aux | grep 30.exe
vasya 6495 0.0 0.0 10700 964 pts/66 D+ 03:30 0:00 ./30.exe
vasya 6496 0.0 0.0 10700 964 pts/66 S+ 03:30 0:00 ./30.exe

So the parent doesnt do any i\o operations but have 'D' status.

Next - Lets have a look if process with 'D' uses cpu. So lets check if it really sleepls.

$ while true; do cat /proc/6495/stat | awk '{print 3,ドル 14,ドル 15ドル}'; done
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0
D 0 0

As we can see it is.

Next. Very often we can see a process is in 'D' state however "top" shows it consumes CPU. How is it possible? The quick answer - the process switches between 'D' and 'R' states. It can happen very quickly. Remember "top" reads all the information from procfs. By default "top" refreshes all the data every 3 seconds so if the process switched very often to 'D' state and not so often to 'R' state it will seem to us as the process lives all the time in 'D' state. However it is false assumption.

The next important point is that the state of the process is a property of instant moment. That is when we are talking about the process has "D" state we mean it is in this state at this particular time. However when we are talking about CPU consumption - it is not about property of instant moment of time. It is average value FOR SOME PERIOD OF TIME. Pls have a look at the picture:

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13416 vasya 20 0 24024 5308 2132 D 62,9 0,0 0:05.02 dd 
 

We see that the state = 'D' %CPU = 62,9

It means that at the moment (right now) the state = 'D' It means right now process does not consume cpu cycles , however some time before the process was NOT in 'D' state and it was consuming cpu cycles. So for the simplicity "top" calculates the average for the last three seconds. It can be like so:

Now - 0%
1 sec ago - 62,9%
2 sec ago - 62,9%
3 sec ago - 62,9%
the average = (62,9%+62,9%+62,9%+0%)/(1+1+1) = 62,9%

So thats why "top" shows 62.9% of cpu usage in spite of the state of the process is 'D'.

To proof that 'dd' switches between 'D' and 'R' states:

$ while true; do cat /proc/13416/stat | awk '{print 3,ドル 14,ドル 15ドル}'; done
R 0 745
R 0 745
D 0 746
D 0 746
D 0 746
D 0 746
D 0 746
D 0 746
R 0 746
R 0 746
R 0 746
R 0 747
R 0 747
R 0 747
R 0 748
D 0 748
D 0 748
D 0 748
D 0 748
R 0 748
R 0 748
R 0 749
D 0 749
R 0 749
R 0 749
D 0 750
D 0 750
D 0 750
D 0 750
D 0 750
D 0 750
R 0 750
R 0 751
R 0 751
R 0 752
R 0 752
R 0 752
D 0 752
D 0 752
D 0 752
D 0 752
D 0 752
R 0 753
R 0 753
D 0 753
D 0 753
D 0 753
D 0 753
R 0 754
R 0 754
R 0 755
R 0 755
D 0 756
D 0 756

About the meaning of 14ドル and %15 fields: As for "man proc":

14ドル = Amount of time that this process has been scheduled in user mode, measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).
15ドル = Amount of time that this process has been scheduled in kernel mode, measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).

As you can see "dd" switches between 'R' and 'D' states. Thats why average cpu consumption is not 0%.

Also you can see that while the process really in 'D' state it does not consume cpu cycles neither for user mode nor for kernel mode.

As the final suggestion: if you have a process and want to know if it is running right now in user or kernel space --> start monitoring

$ cat /proc/13416/stat | awk '{print 3,ドル 14,ドル 15ドル}';

If 14ドル is changing - it means the process in user space, if %15 is changing - it means the process is in kernel space

Hope it helps

Answer 3

According to the source code, it is not possible to do that with top.

Nevertheless, top has a -p parameter which allows you to tell it to watch only certain PIDs. Unfortunately this one is also limited to 20 parameters and the kernel could easily have more than 20 processes running.

I created a two line script to test that out:

kprocs=$(ls -l /proc/*/exe | grep -v " -> " | cut -d "/" -f 3)
top -c -p $(echo ${kprocs} | sed 's/ /,/g')

In case you where wandering I am detecting the kernel processes by looking at which links are not pointing to a binary on the disk.

• Gilles 'SO- stop being evil': What you're doing here is looking for kernel threads, which is not what I understand by "processes currently running in kernel mode".
• jam: as Gilles mentioned I don't want the system threads, but the processes which are busy in system calls