Tracking down where disk space has gone on Linux?

Question

When administering Linux systems I often find myself struggling to track down the culprit after a partition goes full. I normally use du / | sort -nr but on a large filesystem this takes a long time before any results are returned.

Also, this is usually successful in highlighting the worst offender but I've often found myself resorting to du without the sort in more subtle cases and then had to trawl through the output.

I'd prefer a command line solution which relies on standard Linux commands since I have to administer quite a few systems and installing new software is a hassle (especially when out of disk space!)

Answer 1

Try ncdu, an excellent command-line disk usage analyser:

The output of the "ncdu" program. The output is listing files in /usr/portage/distfiles in descending size order, with the file size and name. The user is in the middle of deleting the 239.1 MiB "firefox-57.0.source.tar.xz" file, as it is superceded by the 239.4 MiB "firefox-57.0.1.source.tar.xz" later version.

• jds: Typically, I hate being asked to install something to solve a simple issue, but this is just great.
• Orion Edwards: sudo apt install ncdu on ubuntu gets it easily. It's great
• Luke Cousins: You quite probably know which filesystem is short of space. In which case you can use ncdu -x to only count files and directories on the same filesystem as the directory being scanned.
• B. Shea: best answer. also: sudo ncdu -rx / should give a clean read on biggest dirs/files ONLY on root area drive. (-r = read-only, -x = stay on same filesystem (meaning: do not traverse other filesystem mounts) )
• Chris: I have so little space that I can't install ncdu

Answer 2

Don't go straight to du /. Use df to find the partition that's hurting you, and then try du commands.

One I like to try is

# U.S.
du -h <dir> | grep '[0-9\.]\+G'
# Others
du -h <dir> | grep '[0-9,円]\+G'

because it prints sizes in "human readable form". Unless you've got really small partitions, grepping for directories in the gigabytes is a pretty good filter for what you want. This will take you some time, but unless you have quotas set up, I think that's just the way it's going to be.

As @jchavannes points out in the comments, the expression can get more precise if you're finding too many false positives. I incorporated the suggestion, which does make it better, but there are still false positives, so there are just tradeoffs (simpler expr, worse results; more complex and longer expr, better results). If you have too many little directories showing up in your output, adjust your regex accordingly. For example,

grep '^\s*[0-9\.]\+G'

is even more accurate (no < 1GB directories will be listed).

If you do have quotas, you can use

quota -v

to find users that are hogging the disk.

• zzapper: This is very quick, simple and practical
• jchavannes: grep '[0-9]G' contained a lot of false positives and also omitted any decimals. This worked better for me: sudo du -h / | grep -P '^[0-9\.]+G'
• Ben Collins: @jchavannes -P is unnecessary for this expression because there's nothing specific to Perl there. Also, -P isn't portable to systems that don't have the GNU implementation.
• Vitruvie: In case you have really big directories, you'll want [GT] instead of just G
• augurar: I like to use du -h | sort -hr | head

Answer 3

For a first look, use the "summary" view of du:

du -s /*

The effect is to print the size of each of its arguments, i.e. every root folder in the case above.

Furthermore, both GNU du and BSD du can be depth-restricted (but POSIX du cannot!):

• GNU (Linux, ...):

  du --max-depth 3
  

• BSD (macOS, ...):

  du -d 3
  

This will limit the output display to depth 3. The calculated and displayed size is still the total of the full depth, of course. But despite this, restricting the display depth drastically speeds up the calculation.

Another helpful option is -h (words on both GNU and BSD but, once again, not on POSIX-only du) for "human-readable" output (i.e. using KiB, MiB etc.).

• ReactiveRaven: if du complains about -d try --max-depth 5 in stead.
• Thales Ceolin: Great anwser. Seems correct for me. I suggest du -hcd 1 /directory. -h for human readable, c for total and d for depth.
• jonathanccalixto: I'm use du -hd 1 <folder to inspect> | sort -hr | head
• srghma: du --max-depth 5 -h /* 2>&1 | grep '[0-9\.]\+G' | sort -hr | head to filter Permission denied

Answer 4

You can also run the following command using du:

~# du -Pshx /* 2>/dev/null

• The -s option summarizes and displays total for each argument.
• -h prints Mio, Gio, etc.
• -x = stay in one filesystem (very useful).
• -P = don't follow symlinks (which could cause files to be counted twice for instance).

Be careful with -x, which will not show the /root directory if it is on a different filesystem. In that case, you have to run du -Pshx /root 2>/dev/null to show it (once, I struggled a lot not pointing out that my /root directory had gone full).

• Mykhaylo Adamovych: du -Pshx .* * 2>/dev/null + hidden/system directories
• Atralb: /root/shows without issues. Why would it not be shown ?

Answer 5

Finding the biggest files on the filesystem is always going to take a long time. By definition you have to traverse the whole filesystem looking for big files. The only solution is probably to run a cron job on all your systems to have the file ready ahead of time.

One other thing, the x option of du is useful to keep du from following mount points into other filesystems. I.e:

du -x [path]

The full command I usually run is:

sudo du -xm / | sort -rn > usage.txt

The -m means return results in megabytes, and sort -rn will sort the results largest number first. You can then open usage.txt in an editor, and the biggest folders (starting with /) will be at the top.

• SamB: Thanks for pointing out the -x flag!
• B. Shea: "finding biggest takes long time.." -> Well it depends, but tend to disagree: doesn't take that long with utilities like ncdu - at least quicker than du or find (depending on depth and arguments)..
• Bruno: since I prefer not to be root, I had to adapt where the file is written : sudo du -xm / | sort -rn > ~/usage.txt

Answer 6

I use this for the top 25 worst offenders below the current directory

# -S to not include subdir size, sorted and limited to top 25
du -S . | sort -nr | head -25

• thegreendroid: This command did the trick to find a hidden folder that seemed to be increasing in size over time. Thanks!
• User: Is this in bytes?
• serg10: By default, on my system, 'du -S' gives a nice human readable output. You get a plain number of bytes for small files, then a number with a 'KB' or 'MB' suffix for bigger files.
• Clare Macrae: @Siddhartha If you add -h, it will likely change the effect of the sort -nr command - meaning the sort will no longer work, and then the head command will also no longer work
• oarfish: On Ubuntu, I need to use -h to du for human readable numbers, as well as sort -h for human-numeric sort. The list is sorted in reverse, so either use tail or change order.

Answer 7

I always use du -sm * | sort -n, which gives you a sorted list of how much the subdirectories of the current working directory use up, in mebibytes.

You can also try Konqueror, which has a "size view" mode, which is similar to what WinDirStat does on Windows: it gives you a viual representation of which files/directories use up most of your space.

Update: on more recent versions, you can also use du -sh * | sort -h which will show human-readable filesizes and sort by those. (numbers will be suffixed with K, M, G, ...)

For people looking for an alternative to KDE3's Konqueror file size view may take a look at filelight, though it's not quite as nice.

• flussence: That's only Konqueror 3.x though - the file size view still hasn't been ported to KDE4.
• pahariayogi: 'du -sh * | sort -h ' works perfectly on my Linux (Centos distro) box. Thanks!

Answer 8

At a previous company we used to have a cron job that was run overnight and identified any files over a certain size, e.g.

find / -size +10000k

You may want to be more selective about the directories that you are searching, and watch out for any remotely mounted drives which might go offline.

• rjmunro: You can use the -x option of find to make sure you don't find files on other devices than the start point of your find command. This fixes the remotely mounted drives issue.

Answer 9

I use

du -ch --max-depth=2 .

and I change the max-depth to suit my needs. The "c" option prints totals for the folders and the "h" option prints the sizes in K, M, or G as appropriate. As others have said, it still scans all the directories, but it limits the output in a way that I find easier to find the large directories.

Answer 10

One option would be to run your du/sort command as a cron job, and output to a file, so it's already there when you need it.

Answer 11

For the commandline I think the du/sort method is the best. If you're not on a server you should take a look at Baobab - Disk usage analyzer. This program also takes some time to run, but you can easily find the sub directory deep, deep down where all the old Linux ISOs are.

• Colonel Sponsz: It can also scan remote folders via SSH, FTP, SMB and WebDAV.
• timelmer: This is great. Some things just work better with a GUI to visualize them, and this is one of them! I need an X-server on my server anyways for CrashPlan, so it works on that too.

Answer 12

I'm going to second xdiskusage. But I'm going to add in the note that it is actually a du frontend and can read the du output from a file. So you can run du -ax /home > ~/home-du on your server, scp the file back, and then analyze it graphically. Or pipe it through ssh.

Answer 13

Maybe worth to note that mc (Midnight Commander, a classic text-mode file manager) by default show only the size of the directory inodes (usually 4096) but with CtrlSpace or with menu Tools you can see the space occupied by the selected directory in a human readable format (e.g., some like 103151M).

For instance, the picture below show the full size of the vanilla TeX Live distributions of 2018 and 2017, while the versions of 2015 and 2016 show only the size of the inode (but they have really near of 5 Gb each).

That is, CtrlSpace must be done one for one, only for the actual directory level, but it is so fast and handy when you are navigating with mc that maybe you will not need ncdu (that indeed, only for this purpose is better). Otherwise, you can also run ncdu from mc. without exit from mc or launch another terminal.

mwe

• StackzOfZtuff: Related question: serverfault.com/questions/246840/…

Answer 14

Try feeding the output of du into a simple awk script that checks to see if the size of the directory is larger than some threshold, if so it prints it. You don't have to wait for the entire tree to be traversed before you start getting info (vs. many of the other answers).

For example, the following displays any directories that consume more than about 500 MB.

du -kx / | awk '{ if (1ドル > 500000) { print 0ドル} }'

To make the above a little more reusable, you can define a function in your .bashrc, ( or you could make it into a standalone script).

dubig() {
 [ -z "1ドル" ] && echo "usage: dubig sizethreshMB [dir]" && return
 du -kx 2ドル | awk '{ if (1ドル > '1ドル'*1024) { print 0ドル} }'
}

So dubig 200 ~/ looks under the home directory (without following symlinks off device) for directories that use more than 200 MB.

• ndemou: It's a pity that a dozen of grep hacks are more upvoted. Oh and du -k will make it absolutely certain that du is using KB units
• Mark Borgerding: Good idea about the -k. Edited.
• dave_thompson_085: Even simpler and more robust: du -kx 2ドル | awk '1ドル>'$((1ドル*1024)) (if you specify only a condition aka pattern to awk the default action is print 0ドル)
• ndemou: Good point @date_thompson_085. That's true for all versions of awk I know of (net/free-BSD & GNU). @mark-borgerding so this means that you can greatly simplify your first example to just du -kx / | awk '1ドル > 500000'
• ndemou: @mark-borgerding: If you have just a few kBytes left somewhere you can also keep the whole output of du like this du -kx / | tee /tmp/du.log | awk '1ドル > 500000'. This is very helpful because if your first filtering turns out to be fruitless you can try other values like this awk '1ドル > 200000' /tmp/du.log or inspect the complete output like this sort -nr /tmp/du.log|less without re-scanning the whole filesystem

Answer 15

From the terminal, you can get a visual representation of disk usage with dutree

It is very fast and light because it is implemented in Rust

dutree

$ dutree -h
Usage: dutree [options] <path> [<path>..]
Options:
 -d, --depth [DEPTH] show directories up to depth N (def 1)
 -a, --aggr [N[KMG]] aggregate smaller than N B/KiB/MiB/GiB (def 1M)
 -s, --summary equivalent to -da, or -d1 -a1M
 -u, --usage report real disk usage instead of file size
 -b, --bytes print sizes in bytes
 -f, --files-only skip directories for a fast local overview
 -x, --exclude NAME exclude matching files or directories
 -H, --no-hidden exclude hidden files
 -A, --ascii ASCII characters only, no colors
 -h, --help show help
 -v, --version print version number

See all the usage details in the website

• bfontaine: You should mention that you’re the author of the tool.
• yurenchen: seems your site is down? github.com/nachoparker/dutree

Answer 16

I prefer to use the following to get an overview and drill down from there...

cd /folder_to_check
du -shx */

This will display results with human readable output such as GB, MB. It will also prevent traversing through remote filesystems. The -s option only shows summary of each folder found so you can drill down further if interested in more details of a folder. Keep in mind that this solution will only show folders so you will want to omit the / after the asterisk if you want files too.

Answer 17

Not mentioned here but you should also check lsof in case of deleted/hanging files. I had a 5.9GB deleted tmp file from a run away cronjob.

https://serverfault.com/questions/207100/how-can-i-find-phantom-storage-usage Helped me out in find the process owner of said file ( cron ) and then I was able to goto /proc/{cron id}/fd/{file handle #} less the file in question to get the start of the run away, resolve that, and then echo ""> file to clear up space and let cron gracefully close itself up.

Answer 18

I like the good old xdiskusage as a graphical alternative to du(1).

• ndemou: Note this part of the question: "I'd prefer a command line solution which relies on standard Linux commands since..."

Answer 19

You can use standard tools like find and sort to analyze your disk space usage.

List directories sorted by their size:

find / -mount -type d -exec du -s "{}" \; | sort -n

List files sorted by their size:

find / -mount -printf "%k\t%p\n" | sort -n

• vimal krishna: I find this to be the best answer, to detect the large sized in sorted order

Answer 20

I have used this command to find files bigger than 100Mb:

find / -size +100M -exec ls -l {} \;

Answer 21

Still here? Or perhaps this answer has been upvoted...

While there are various graphical tools described in other answers, they don't do much to address the underlying issue of identifying how you may be able to free up space.

I am currently researching the same issue and came across agedu - which reports on access times as well as size. I've not had a chance to play with it yet - it's written by Simon Tatham (you may have heard of PuTTy) so is probably sensible/reliable.

However, like all the tools listed here, it collects data on demand. Even the most efficint coding on the fastests hardware will take time to walk a milt-terrabyte filesystem.

• user339730: If you can't use a GUI (like you're on a remote server), ncdu -e works nicely. Once the display opens up, use m then M to display and sort by mtime, while the (admittedly small) percentage graph is still there to get you an idea of the size.
• symcbean: "If you can't use a GUI (like you're on a remote server)," - why does a remote server prevent you from using a gui?
• Dennis: ncdu -e is wrong becasue it requires an argument

Answer 22

For command line du (and it's options) seems to be the best way. DiskHog looks like it uses du/df info from a cron job too so Peter's suggestion is probably the best combination of simple and effective.

(FileLight and KDirStat are ideal for GUI.)

• oligofren: There are tons of user friendly options for the terminal that are way more effective than du scripting. The author of ncdu lists a long list (ncdu, gdu, dua-cli, ...) at the bottom of the ncdu homepage dev.yorhel.nl/ncdu

Answer 23

At first I check the size of directories, like so:

du -sh /var/cache/*/

Answer 24

Here is a tiny app that uses deep sampling to find tumors in any disk or directory. It walks the directory tree twice, once to measure it, and the second time to print out the paths to 20 "random" bytes under the directory.

void walk(string sDir, int iPass, int64& n, int64& n1, int64 step){
 foreach(string sSubDir in sDir){
 walk(sDir + "/" + sSubDir, iPass, n, n1, step);
 }
 foreach(string sFile in sDir){
 string sPath = sDir + "/" + sFile;
 int64 len = File.Size(sPath);
 if (iPass == 2){
 while(n1 <= n+len){
 print sPath;
 n1 += step;
 }
 }
 n += len;
 }
}
void dscan(){
 int64 n = 0, n1 = 0, step = 0;
 // pass 1, measure
 walk(".", 1, n, n1);
 print n;
 // pass 2, print
 step = n/20; n1 = step/2; n = 0;
 walk(".", 2, n, n1);
 print n;
}

The output looks like this for my Program Files directory:

 7,908,634,694
.\ArcSoft\PhotoStudio 2000\Samples3円.jpg
.\Common Files\Java\Update\Base Images\j2re1.4.2-b28\core1.zip
.\Common Files\Wise Installation Wizard\WISDED53B0BB67C4244AE6AD6FD3C28D1EF_7_0_2_7.MSI
.\Insightful\splus62\java\jre\lib\jaws.jar
.\Intel\Compiler\Fortran9円.1\em64t\bin\tselect.exe
.\Intel\Download\IntelFortranProCompiler91\Compiler\Itanium\Data1.cab
.\Intel\MKL8円.0.1\em64t\bin\mkl_lapack32.dll
.\Java\jre1.6.0\bin\client\classes.jsa
.\Microsoft SQL Server90円\Setup Bootstrap\sqlsval.dll
.\Microsoft Visual Studio\DF98\DOC\TAPI.CHM
.\Microsoft Visual Studio .NET 2003\CompactFrameworkSDK\v1.0.5000\Windows CE\sqlce20sql2ksp1.exe
.\Microsoft Visual Studio .NET 2003\SDK\v1.1\Tool Developers Guide\docs\Partition II Metadata.doc
.\Microsoft Visual Studio .NET 2003\Visual Studio .NET Enterprise Architect 2003 - English\Logs\VSMsiLog0A34.txt
.\Microsoft Visual Studio 8\Microsoft Visual Studio 2005 Professional Edition - ENU\Logs\VSMsiLog1A9E.txt
.\Microsoft Visual Studio 8\SmartDevices\SDK\CompactFramework2円.0\v2.0\WindowsCE\wce500\mipsiv\NETCFv2.wce5.mipsiv.cab
.\Microsoft Visual Studio 8\VC\ce\atlmfc\lib\armv4i\UafxcW.lib
.\Microsoft Visual Studio 8\VC\ce\Dll\mipsii\mfc80ud.pdb
.\Movie Maker\MUI0409円\moviemk.chm
.\TheCompany\TheProduct\docs\TheProduct User's Guide.pdf
.\VNI\CTT6.0\help\StatV1.pdf
7,908,634,694

It tells me that the directory is 7.9gb, of which

• ~15% goes to the Intel Fortran compiler
• ~15% goes to VS .NET 2003
• ~20% goes to VS 8

It is simple enough to ask if any of these can be unloaded.

It also tells about file types that are distributed across the file system, but taken together represent an opportunity for space saving:

• ~15% roughly goes to .cab and .MSI files
• ~10% roughly goes to logging text files

It shows plenty of other things in there also, that I could probably do without, like "SmartDevices" and "ce" support (~15%).

It does take linear time, but it doesn't have to be done often.

Examples of things it has found:

• backup copies of DLLs in many saved code repositories, that don't really need to be saved
• a backup copy of someone's hard drive on the server, under an obscure directory
• voluminous temporary internet files
• ancient doc and help files long past being needed

Answer 25

If you know that the large files have been added in the last few days (say, 3), then you can use a find command in conjunction with "ls -ltra" to discover those recently added files:

find /some/dir -type f -mtime -3 -exec ls -lart {} \;

This will give you just the files ("-type f"), not directories; just the files with modification time over the last 3 days ("-mtime -3") and execute "ls -lart" against each found file ("-exec" part).

Answer 26

To understand disproportionate disk space usage it's often useful to start at the root directory and walk up through some of its largest children.

We can do this by

• saving the output of du into a file
• grepping through the result iteratively

That is:

# sum up the size of all files and directories under the root filesystem
du -a -h -x / > disk_usage.txt
# display the size of root items
grep $'\t/[^/]*$' disk_usage.txt

now let's say /usr appear too large

# display the size of /usr items
grep $'\t/usr/[^/]*$' disk_usage.txt

now if /usr/local is suspiciously large

# display the size /usr/local items
grep $'\t/usr/local/[^/]*$' disk_usage.txt

and so on...

Answer 27

I had a similar issue, but the answers on this page weren't enough. I found the following command to be the most useful for the listing:

du -a / | sort -n -r | head -n 20

Which would show me the 20 biggest offenders. However even though I ran this, it did not show me the real issue, because I had already deleted the file. The catch was that there was a process still running that was referencing the deleted log file... so I had to kill that process first then the disk space showed up as free.

• ndemou: Good point but this should be a comment and not an answer by itself - this question suffers from too many answers

Answer 28

Another one is duc, sort of a collection of command line tools which are indeed scalable, fast and versatile. It also features some GUI/TUI options.

Answer 29

I've had success tracking down the worst offender(s) piping the du output in human readable form to egrep and matching to a regular expression.

For example:

du -h | egrep "[0-9]+G.*|[5-9][0-9][0-9]M.*"

which should give you back everything 500 megs or higher.

• ndemou: Don't use grep for arithmetic operations -- use awk instead: du -k | awk '1ドル > 500000'. It is much easier to understand, edit and get correct on the first try.

Answer 30

If you want speed, you can enable quotas on the filesystems you want to monitor (you need not set quotas for any user), and use a script that uses the quota command to list the disk space being used by each user. For instance:

quota -v $user | grep $filesystem | awk '{ print 2ドル }'

would give you the disk usage in blocks for the particular user on the particular filesystem. You should be able to check usages in a matter of seconds this way.

To enable quotas you will need to add usrquota to the filesystem options in your /etc/fstab file and then probably reboot so that quotacheck can be run on a idle filesystem before quotaon is called.