Why does a disk clone using dd take up more space on the target than the source image?

Question

I'm doing some Android development on Ubuntu 11.10. I'm building a set of SD card and eMMC card Android images using a Bash script.

Due to Android's idiosyncrasies, I have several file system images that need to be copied to the target card. There are several lines in my script to copy over the images that look like this:

image_dir=/some/directory
node=/dev/some/block/device
dd if=${image_dir}/data.img of=${node}; sync

Here's where things get weird. After partitioning the card and copying overdata.img using this dd command, there is a vast inconsistency between the size of the data.img on my dev machine and the amount of space taken up on the target SD card partition.

For instance, the data.img file on my disk takes up 128 megabytes of space, roughly, but takes up almost 2.5 gigabytes on the partition of the SD card. For obvious reasons, this is a serious problem.

I've tried modifying the number of blocks dd reads and writes, but that didn't seem to change the amount of space required. I've done a fair bit of googling, but I can't seem to find any references to problems like this.

What can I do to fix this issue so the image doesn't take up so much space? Could there be something wrong with the data.img file? Am I using dd wrong?

I've been tearing my hair out over this for a week, help me StackExchange, you're my only hope.

EDIT: Question clarified above: It's not that the file system takes up a lot of space on the partition, it's that the partition is filled almost completely with data after running dd, despite the file system image being substantially smaller than said partition.

• psusi: 11.10 has been end of life for almost a year; you need to upgrade immediately. Also you did not specify how you are measuring any of this. ls or du?
• Andrew: Due to the version of Android's build tools we are using, we have to use 11.10. I'm using ls -l to get the size of the image and I'm using gparted to inspect the partitions and file systems after partitioning/flashing.
• psusi: The size of the partition is the size of the partition... using dd to copy a filesystem image to it doesn't change that. And the android build tools run just fine on 12.04 or 13.10, both of which would not be exposing you to unpatched security vulnerabilities.
• Andrew: @psusi Ah, I misspoke. It isn't Android's tools that require 11.10, it's that we've also done some work using a Linaro toolchain and LTIB that only run on 11.10. Believe me, if I could upgrade, I would.

Answer 1

The first thing that jumps to mind is "files with holes". If a program opens a file, and uses the lseek() system call to set the file offset to greater than one file-system-block, then writes some bytes, the file system code only allocates a block for the data that got written. The first file-system-block does not get allocated. If another program opens the file and reads some bytes in that file-system-block, it gets zero value bytes.

A program can skip a block anywhere in the file, not just at the beginning. Thus, "files with holes".

Having a backup blow-up in size is not uncommon for linux/unix, but the culprit is usually something like a database file. I know a lot of Android apps use Sqlite3, perhaps that's the cause.

• Andrew: Hmm, riddle me this, then: if data.img is an image of a file system containing sparse files (like SQlite3 databases), what about copying it with dd makes those holes take up different amounts of space on the different block devices? Why does it take up so much space after the copy? If those holes are present in the .img file, why don't they take up as much space before the block level copy?
• cjm: @Andrew, it's not that the imaged filesystem contains sparse files, but that data.img itself is a sparse file. You don't mention where it came from.
• Andrew: @cjm My apologies, it's generated using the AOSP (Android Open Source Project) make files. Specifically, a bin called make_ext4fs in the system directory. Seems like I'll have to do some digging into that util to see if can be configured.
• user732: @Andrew - you make a good point. Sorry, I seem to have mis-read your question. I had to re-read it three times to understand your comment. Sorry for the potential mis-directrion.
• Gilles 'SO- stop being evil': @Andrew dd doesn't do a block-level copy. It does a byte-level copy. (There is no standard tool that does a block-level copy.) Use cp --sparse=always to make a sparse copy.