Fetching individual bytes from a binary file into a variable with bash

Question

I need to make a fairly simple bash script to pull bytes one at a time from a binary file, send it out a serial port, and then wait for a byte to come back before I send the next one. This is effectively for an EEPROM programmer, where an off the shelf solution won't work.

I'm mostly stuck at pulling the bytes out of the file into a variable, beyond that I know I can echo the values down the serial port and read them back with dd. (Is there a better way to do that also?)

Thanks for the help!

• mikeserv: will the return byte definitely always arrive singly? As in, will you receive only one byte per byte sent?
• LatinSuD: You can work with hex values, like using the command xxd -ps and xxd -r -ps. It would be safer.
• Gilles 'SO- stop being evil': IIRC bash can't cope with null bytes, so it's unsuitable to deal with binary data. Most shells aren't designed to deal with binary data (zsh is an exception). Is this an embedded system? If so how restricted is your tool set?
• Minifig666: Yeah, the device on the far end should relay the same byte that is sent out, after it has gone through a bit of processing on the device end.
• Minifig666: This script is just going on a lubuntu laptop, so I have no real restrictions on the tools I can use. It would be nice to send straight binary values down the serial port, but I could potentially send two ASCII hex characters per byte, and then edit the code on the device end to cope.

Answer 1

Use dd or xxd (part of Vim), for example to read one byte (-l) at offset 100 (-s) from binary file try:

xxd -p -l1 -s 100 file.bin

to use hex offsets, in Bash you can use this syntax $((16#64)), e.g.:

echo $((16#$(xxd -p -l1 -s $((16#FC)) file.bin)))

which reads byte at offset FC and print it in decimal format.

Alternatively use dd, like:

dd if=file.bin seek=$((16#FC)) bs=1 count=5 status=none

which will dump raw data of 5 bytes at hex offset FC.

Then you can assign it into variable, however it won't work when your data has NULL bytes, therefore either you can skip them (xxd -a) or as workaround you can store them in plain hexadecimal format.

For example:

1. Read 10 bytes at offset 10 into variable which contain bytes in hex format:

   hex=$(xxd -p -l 10 -s 10 file.bin)
   

2. Then write them into file or device:

   xxd -r -p > out.bin <<<$hex
   

---

Here are few useful functions in Bash:

set -e
# Read single decimal value at given offset from the file.
read_value() {
 file="1ドル"
 offset="2ドル"
 [ -n "$file" ]
 [ -n "$offset" ]
 echo $((16#$(xxd -p -l1 -s $offset "$file")))
}
# Read bytes in hex format at given offset from the file.
read_data() {
 file="1ドル"
 offset="2ドル"
 count="3ドル:-1"
 xxd -p -l $count -s $offset "$file"
}

Sample usage:

read_value file.bin $((16#FC)) # Read 0xFC offset from file.bin.

• uli42: I think your dd example is wrong, it should use skip= instead of seek= because seek= is of the outputfile whiel skip is for the input file.

Answer 2

I’m not sure I understand the big picture of what you’re asking for, but I have a couple of approaches that you might want to consider:

Use dd to read your input file

exec 3< input_file
while true
do
 dd bs=1 count=1 <&3 > this_byte
 if ! [ -s this_byte ] # If size is zero, we’re done.
 then
 break
 fi
 code using the this_byte file
 ...
done

Use od to digest your input file

od -bv input_file | while read a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 a13 a14 a15 a16
do # Skip $a0; it’s the address.
 for byte_val in "$a1" "$a2" "$a3" "$a4" "$a5" "$a6" "$a7" "$a8" "$a9" "$a10" \
 "$a11" "$a12" "$a13" "$a14" "$a15" "$a16"
 do
 if [ "$byte_val" = "" ]
 then
 break
 fi
 code using the $byte_val value
 for example: echo -e "0円$byte_val\c"
 ...
 done
done

Answer 3

This is probably easier to do with a tiny C program, or an (almost) one-liner in a scripting language like Python or Perl.

The Unix shell is an interpreter for user commands, that (for convenience in writing repetitive commands) has rudimentary facilities for programming. It often gets (mis)used because a version of the Bourne shell is guaranteed to be available on any Unix system, not because it is a particularly good (or even half-way decent) programming language.

Answer 4

You can use perl instead:

$ printf '%s%s%s' 'ăâé' | LC_ALL=C.UTF-8 perl -Mopen=locale -ne '
 BEGIN { $/ = 1円 }
 printf "%s\n", $_;
'
ă
â
é

• mikeserv: You're just printing the same byte back? As I understand it the asker wants to know how to read a byte from one input, print it to output, and then to wait for another byte from a different input before continuing the loop. Unless that's already what you do and I don't get it - which is entirely possible.