Arithmetic on values with memory size units

Question

Let’s say I have a bunch of numbers representing quantities of memory, written in the form 86k or 320m or 1.7g for instance. How can I compute their sum in command line, and get back a human-readable result?

Being able to compute subtractions would be nice too. The perfect tool would handle several sets of notations (such as 1g / 1G / 1GB / 1Go / 1GiB / 1.7Gio) and their meaning (binary or decimal multipliers).

I am looking for a pure calculator. These numbers are not necessarily the size of some files on my disk, so tools such as find, stat or du are not an option.

This is obviously easy to implement (with some hurdles regarding precision), but I would be damned if this didn’t exist already!

Answer 1

A little self promotion: we wrote a library called libbytesize to do these calculations in C and Python and it also has a commandline tool called bscalc

$ bscalc "5 * (100 GiB + 80 MiB) + 2 * (300 GiB + 15 GiB + 800 MiB)"
1215425413120 B
1186938880.00 KiB
 1159120.00 MiB
 1131.95 GiB
 1.11 TiB

The library is packaged in most distributions, unfortunately the tool isn't. It's in Fedora in libbytesize-tools and SuSE in bscalc package, but not in Debian/Ubuntu.

• Maëlan: In Archlinux the tool is included in the libbytesize package, so this is my favorite answer. In my opinion, it just lacks the ability to select the best prefix automatically w.r.t. the order of magnitude of the result (like numfmt does), and perhaps an option to switch the meaning of ambiguous units such as K to base 1000.
• Vojtech Trefny: So you want to print only 1.11 TiB in the example? I could add an option for that. I'm not sure about changing the meaning of k vs K that might be confusing, but I'll think about that. Feel free to report bugs and/or RFEs on GitHub if you find any issues.

Answer 2

There is Bcal.

$ bcal -m "(5kib+2mib)/2"
1051136 B
$ bcal -m "(5kb+2mb)/2"
1002500 B

The -m flag is for brief output. Removing it outputs verbosely with base 2 (KiB, MiB, GiB, TiB) and base 10 (kB, MB, GB, TB) results.

It does not understand 86k or 320m or 1.7g, after all those are not proper byte units. In that case, you could use Sed to add the b after each letter and then pipe it to bcal:

$ cat file
1.7g+320m+86k
$ sed 's/[gmk]/&b/g' file | bcal -m
bcal> 1.7gb+320mb+86kb
2020086000 B

You can also use it in interactive mode.

• Maëlan: After playing with it, I find that it does the trick but lacks flexibility. On input: indeed k, m and g are not proper units, but they are in wide use. [My ideal tool would accept any syntax in use but treat ambiguities conservatively, i.e. if units such as k or kB are used it would force the user to say on the command line whether they mean 1000 or 1024. (Another ambiguity is whether b means bit or byte.)] On output: it’s all or nothing, even though I can pipe bcal -m into |numfmt --to=iec-i |tr -d ' '... I prefer bscalc` on that regard.

Answer 3

In zsh, you could define a math function like:

() {
 typeset -gA bsuffix
 local n=1 ni=1 s
 for s (k m g t p e) {
 (( n *= 1000 )); (( ni *= 1024 ))
 (( bsuffix[$s] = bsuffix[${s}ib] = bsuffix[${s}io] = ni ))
 (( bsuffix[${s}b] = bsuffix[${s}o] = n ))
 }
}
b() {
 set -o localoptions -o extendedglob
 local s=${(M)1%(#i)(${(j:|:k)~bsuffix})}
 (( ${1%$s} * ${bsuffix[$s:l]-1} ))
}
functions -Ms b

Then you'd be able to use b(1G), b(1mB) in any zsh arithmetic expression, like in (( .... )), $(( ... )), $array[...], etc, or in zcalc:

$ <<< $((b(86k) + b(320mb) + b(1.7gio)))
2145449164.8
$ autoload zcalc
$ zcalc
1> b(86k) + b(320mb) + b(1.7gio)
2.14545e+09
2> :sci 15
2145449164.8
$ echo $(( b(infeo) ))
Inf 😎

(note that we make no difference between b and B (or o / O), the match it case insensitive. It's not interpreted as bit vs byte).

Another approach could be to have the b() function take the whole expression as argument, and replace all the suffixes with * $bsuffix[<suffix>]

b() {
 set -o localoptions -o extendedglob
 local s=${(M)1%(#i)(${(j:|:k)~bsuffix})}
 (( ${1//(#bi)([0-9.][[:blank:]]#)(${(j:|:k)~bsuffix})/$match[1] * $bsuffix[$match[2]:l] } ))
}

And then:

$ echo $(( b(1m + 1Mb) ))
2048576

There's the problem of e/E (exa) though which puts a spanner in the works in that 1e-3GB would not be interpreted as 0.001 * 1000000000 but as 1 * 1152921504606846976 - 3 * 1000000000.

In any shell with support for floating point arithmetic (ksh93, zsh, yash), you could always define:

 K=1024 M=$((K * K)) G=$((M * K)) T=$((G * K)) P=$((T * K)) E=$((P * K))
KiB=$K MiB=$M GiB=$G TiB=$T PiB=$P EiB=$E
 KB=1000 MB=$((KB*KB)) GB=$((MB*KB)) TB=$((GB*KB)) PB=$((TB*KB)) EB=$((PB*KB))

Or to golf it:

K=1024 EiB=$((E=K*(P=PiB=K*(T=TiB=K*(G=GiB=K*(M=MiB=K*K))))))
KB=1000 EB=$((EB=KB*(PB=KB*(TB=KB*(GB=KB*(MB=KB*KB))))))

And write $(( 1.1*GB + 5*K ))

to add the suffixes on output, you could use GNU numfmt:

$ human() numfmt --field=- --to=iec --suffix=iB
$ echo $(( b(1m + 1Mb) )) | human
2.0MiB

• Maëlan: I don’t use zsh, but this is a nice trick! If i read correctly the :l in ${bsuffix[$s:l]-1} (and the %(#i) on the line before, I guess?), it even supports case variations. What does the -1 do, however?
• Stéphane Chazelas: @Maëlan, ${var-default} is a standard Bourne shell (from the 70s!) operator, to expand to default if $var is unset, here applied to the associative array expansion with 1 as the default. $var:l (or ${(L)var}) converts to lowercase, (#i) is an extendedglob operator to enable case-insensitive match (similar to perl's (?i) regexp operator. See how zsh used the same trick using # (equivalent to regex *) after ( which otherwise would have been invalid, like ? is invalid otherwise after ( in regexps to introduce a new operator without breaking backward compatibility).

Answer 4

powershell (it's one of the shells for *nix for those of you who are wondering) has built-in "SI" suffixes for numeric literals so you can use them directly

PS /home> $f = 5 * (100GB + 80MB) + 2 * (300GB + 15GB + 800MB)
PS /home> $f
1215425413120
PS /home> "In MB: " + $f/1MB
In MB: 1159120
PS /home> "In GB: "; $f/1GB
In GB:
1131.953125
PS /home> "In TB: {0}" -f ($f/1TB)
In TB: 1.10542297363281

Those are binary prefixes and not decimal prefixes so if you want decimal units just multiply by 1eX to get the desired value

PS /home> 1KB; 1MB; 1GB; 1TB; 1PB
1024
1048576
1073741824
1099511627776
1125899906842624
PS /home> $K = 1e3; $M = 1e6; $G = 1e9
PS /home> 1*$G
1000000000
PS /home> 5 * (100*$G + 80*$M) + 2 * (300*$G + 15*$G + 800*$M)
1132000000000

You can store those constants in the profile (similar to .bashrc or .bash_profile in bash) to reuse them every time you open PowerShell

PowerShell is actually much more powerful than that. It runs on the .NET framework so it can do anything .NET can do: bigint math, decimal math, bitwise operations, trigonometry, date-time calculations...

---

Since the OP wants a pure calculator, I'll show some more examples on how PowerShell can be used for that purpose. The math functions above are mainly from the .NET Math class and Numerics namespace. You'll put class .NET types inside [], like [math] or [system.math] for the Math class (PowerShell is case-insensitive). Here are some other things that may be useful to programmers:

• Bitwise operations (bitwise operators begin with -b except shift operators)

  [uint64]::MaxValue/3 + (-bnot 20) + (1L -shl 22) + (0x23 -band 0x34)
  

• Big integer math: [bigint]::Pow([uint64]::MaxValue, 20)

• Arbitrary integer and floating-point math expressions

  1.56 + 0.23/[math]::Pow([math]::Sqrt([math]::Log(20) + [math]::Sin([math]::PI/3)), 4)
  

• Math on decimal type (128-bit): 1.23d * 3.45d / 28

• Math on 128-bit integer types

• Calculate file or object sizes: Use number suffixes 12.5GB + 5.8MB + 1392KB for binary units and 12.5e9 + 5.8e6 + 1392e3 for decimal units (G = 1e9, M = 1e6, K = 1e3)

• Convert to/from base64: [Convert]::ToBase64String and [Convert]::FromBase64String

• Date/time manipulation. For example convert from raw Epoch values to datetime and vice versa

   [datetime]::FromFileTime(0x01d15614cbaee92c)
   [datetime]::ParseExact("08-12-2012","dd-MM-yyyy", `
   [Globalization.CultureInfo]::InvariantCulture) 
  

• String formatting and base conversion. Anything that String.Format in .NET supports will work. For more information read about the formatting operator. You can also do advanced string and regex manipulation. Some examples:

   'somestring'.Substring(4) * 3 -replace 'ings', 'eet'
   '{0:X}' -f (0x12 + 34)
   [convert]::ToString(0x12 + 34, 16)
   'This is an emoji' + [char]::ConvertFromUtf32(0x1F60A)
  

• Direct XML, CSV, JSON... manipulation

• Call functions in *.SO (*.DLL in Windows) files directly

• GUI programming. Here's a small sample clipboard history app

  PowerShell sample

For more information read

• Doing Math with System.Math
• Using PowerShell to Work with Numbers
• Playing with JSON and PowerShell
• Time conversions in PowerShell (and .NET in general)
• Understanding Numbers in PowerShell

or follow Dr Scripto's blog

Answer 5

How can I compute their sum in command line, and get back a human-readable result?

Ksh supports floating-point math, so you can define the required constants in .kshrc and use them to do math

Then to get the human-readable result just use the built-in printf with %#d or %#i

printf [ -v varname ] format [ arg ... ]

...

• # The # flag, when used with the %d format without an output base, displays the output in powers of 1000 indicated by one of the following suffixes: k M G T P E, and when used with the %i format displays the output in powers of 1024 indicated by one of the following suffixes: Ki Mi Gi Ti Pi Ei.

https://www.manpagez.com/man/1/ksh/

For example

$ K=1024 M=$((K * K)) G=$((M * K)) T=$((G * K)) P=$((T * K)) E=$((P * K))
$ KB=1000 MB=$((KB*KB)) GB=$((MB*KB)) TB=$((GB*KB)) PB=$((TB*KB)) EB=$((PB*KB))
$ printf "%#i\n" $((12e+12 + 1.2e+9 + 12e+6))
11Ti
$ printf "%#d\n" $(((58*$K + 2*$M)/2))
1.1M
$ printf "%#d\n" $((5 * (100*$G + 80*$M) + 2 * (300*$G + 15*$G + 800*$M)))
1.2T
$ printf "%#d\n" $((1.7*$GB + 320*$MB + 86*$KB))
2.0G

Answer 6

This is horrible, but in some cases one can get by with something like

cat my_columns.txt | cut -d " " -f 5-6 | sed "s/MB/*0.001/" | sed "s/GB//" | paste -sd+ - | bc