Why Bash does not scan all of its own history file

Question

I use a common history for all sessions. It happens to me frequently, that a part of the history is lost, however. Therefore, I just swap different "common session" scripts in a hope that one will fix it, no way so far. It works mostly, but not always.

I just decided to look more closely into this, when Ctrl+R valgrind showed nothing. Yes, it is in the history file - opening .bash_history in an editor and searching shows multiple commands that begin with valgrind. Still, history |grep valgrind shows just itself, and in fact, that very command is put into the said .bash_history, at the very end of it, long after the other commands containing valgrind. And it is not a typo - the history file contains many iptables commands, these are invisible by history and by Ctrl+R. The more recent history in the same file is visible and accessing it works fine.

I would doubt, that Bash history is broken so much. Any explanation?

EDIT: There is a similar question, with no real answer: Is there a limit on how many entries the 'Control R' bash shortcut searches?

• Jimbo: Does this help? superuser.com/questions/211966/…
• scriptfoo: No. As I said, I already use a common Bash history.
• Jimbo: So you did, sorry.
• curusarn: Can you check your ~/.bashrc to see what is HISTSIZE and HISTFILESIZE set to?

Answer 1

bash does not load the whole ~/.bash_history at the start of the session. It only loads a number of commands specified by shell variable HISTSIZE.

There is a difference between the history that is kept in memory and history that is saved in ~/.bash_history file. To make bash load more from ~/.bash_history you need to set HISTSIZE to a higher number.

GNU bash documentation:

The value of the HISTSIZE shell variable is used as the number of commands 
to save in a history list.

The "history list" in the documentation refers to the history kept in memory.

• scriptfoo: The commands which can not be recalled are already saved in the history file.
• curusarn: The size of the history loaded in memory and size of the history that is kept in the HISTFILE are two different things. If your HISTFILESIZE is larger than HISTSIZE then the number of commands loaded into memory can be smaller than the total number commands in your HISTFILE.
• curusarn: This SO question explains that in detail: stackoverflow.com/questions/19454837/…
• curusarn: I extended the answer to explain the difference between the history file and the history in memory.
• scriptfoo: My .bashrc contains HISTFILESIZE=$HISTSIZE, at least now... I do not know if it was the case when I wrote this question. I will set both to 10M and see if relatively recent commands are still lost sometimes.