logserver

pager designed for rapid navigation with multiple keyword searching on giant log files

Concept

The idea of logserver started when doing Android operating system development. The system logs are heavily multiplexed, and often one wants to filter by a log tag and then search within those tags. Or search for all occurances of some keyword, like crash, and then have a holistic view of the results. The key is to be able to go from the summary of matches back to the full log at that position, without having to repeated search when there can be many results.

As it began to be used for more types of files, additions features were added. New features got added to support analysis of web traffic transcripts, e.g., from HAR files. This include decoding of base64 and hex as well as line breaking intelligently for particularly types of lines, such as URL query parameters and pretty printing JSON.

The last major change was supporting cross-references to ease code navigation. This is not fully implemented as it is part of another project called GEB that builds cross references on code bases and thus supports quickly jumping being function definitions and their invocations, among other things. When GEB is finished then it will be put up on codeberg too and integrated into logserver. For now, you can use the output of grep -rn into logserver, search within those results, and use f to go to that file and line number. It also supports reading ctags files with similar behaviour.

requirements

• libz-dev
• libncurses-dev
• catch2 (for unit tests)
• cmake
• g++

building

project uses cmake:

cmake -B build
make -C build

Dependencies include catch2 and txt2man (brew install catch2 and brew install txt2man on OSX)

You can pass the following options to CMake using -D<OPTION>=<VALUE>: The default value is marked with *.

• CMAKE_BUILD_TYPE -- set the build configuration (Debug or Release*)
• BUILD_TESTS -- enable building tests (ON* or OFF)
• BUILD_CPACK -- enable building cpack packaging (ON* or OFF)
• BUILD_MANPAGES -- enable building man pages (ON* or OFF) Example:

cmake -B build -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTS=OFF -DBUILD_MANPAGES=OFF

packaging

cmake -B build
make -C build
cpack --config build/CPackConfig.cmake

installing

There are several ways to install logserver:

• Run sudo make -C build install after building
• Copy build/logserver into a directory in your $PATH
• Install the .deb package generated by cpack (see packaging)

using

command-stdout | logserver
logserver filename

keybindings

Navigation

• arrows: Move around in the file. Long lines do not wrap so left and right moves accordingly.
• home: Move to the top of the file.
• end: Move to one-past the end of the file, which will display streaming data
• shift+home: move left to the start of the line
• shift+end: move right to the end of the line
• page-up: move to the top line visible
• page-down: move to the bottom line visible
• colon: accepts a number afterwards, and moves to that line number

Searching

• slash: Accepts a keyword afterwards, and adds it as a search term.
• backslash: Accepts a keyword afterwards, and adds it as a reverse search.
• caret: Accepts a keyword afterwards, and adds it as a starts with search.
• dollar sign: Accepts a keyword afterwards, and adds it as an ends with search.
• tab: Alternates among search modes.

  •  ALL mode shows all lines and highlights matching keywords.
    

  •  OR mode (disjunctive) shows lines that match any keyword.
    

  •  AND mode (conjunctive) shows lines that match all keywords.
    

• shift+left: move left on the current line to the next matching keyword
• shift+right: move right on the current line to the next matching keyword
• shift+up:
  • In ALL mode, moves up to the next line that matches any keyword.
  • In OR mode, moves up to the next line that matches a keyword that is not matched on the current line. Useful for staying in OR mode but skipping large amounts of the same match.
• shift+down: Same as shift+up but searches downwards.
• backspace: Removes the most-recently added search term.
• plus: Add one more line of context around matching lines.
• minus: Remove one line of context around matching lines.

Line Operations

• octothorpe: Accepts a string afterwards, and adds the current line and surrounding view along with that comment to a file in current directory called storytime.txt.
• letter e: Accepts a string starting with the current line. Changes to that line are reflected in the display (but not the original file).
• letter b: Breaks a long line up and inserts the new lines. This uses a number of heuristics in an attempt to be elegant. It uses spaces and punctuation to give a ragged-right in text. It uses ampersands and equals to infer HTTP query strings and breaks on the ampersand. It uses quotes and braces to infer JSON for pretty printing, and it uses periodic backslash-n to infer escape newlines and breaks on those.
• letter B: Accepts a single character next, and performs the break functionality described for the letter b using that specific character, i.e., replaces that character with newlines.
• letter i: Intelligence for lines. Replaces UNIX timestamps with human time, and looks for sequences of base64 or base16 encoded text based on printable characters after decoding. Repeated pressing of 'i' softens the heuristics of how much text needs to be printable.
• letter d: If hit twice in a row, deletes the current line from the display (not from the original file).
• asterisk: Pins the current line. When searching for keywords pinned lines will appear in OR and AND mode despite not matching.
• letter s: Accepts a string afterwards, and writes the current line to the file specified by that string.
• letter f: Follows a link on the current line. If logserver is given the output of grep -rn, then each line will link to that file and line number and 'f' will follow it. If logserver is reading a ctags file, then each line will be a link to that target.
• letter m: Merge the next line to the end of the current line.
• letter r: Revert the line if it has been changed.

Global Operations and Stacked View

• letter q: Quits logserver.
• letter n: Toggle line numbers on and off.
• letter c: Toggle colouring on and off.
• letter S: Accepts a string afterwards, and writes the entire log file to the file specified by that string. This is useful when data is bring streamed into logserver by program output.
• letter h: Launch the help screen, pushes on the stack.
• letter C: Clears the entire contents of the log. This is useful when streaming in data, e.g., a device log, and you want the logs relevant to a particular event that is about to be triggered.
• exclamation: inserts a new pinned dash line at the current position. Appends it if the current line is one-past the end. Useful for separating segments of the log, such as the debug log corresponding to right before pressing a button and right after the program crashes, so just the area between those events is conspicuous.
• percent:
  • In OR and AND mode, applies the current filter and creates a new view with just the matching lines and puts that on the stack (i.e., some percent of the logs).
  • In ALL mode, goes up and down from the current position searching for a pinned like, such as one created by an exclamation, and pushes a new view on the stack bounded by those (or the top and bottom if none are found).
• less-than: Moves left on the stack of views.
• greater-than: Moves right on the stack of views.
• escape: Pops the top most (right most) view on the stack. Does nothing if there is only one view.
• pipe: Accepts a string afterwards and runs that command, passing the current view as stdin, and pushing a new view on the stack with the stdout of the command as its contents. For security purposes the set of commands that can be run is limited to the following: cat, sort, uniq, ls, grep, cut, tr, sed, awk, fgrep, which, whoami, base64, echo, file, wc, xsel, mplayer.
• letter T: Consider the next key pressed as the tab character, so T-comma can be for CSVs
• letter t: Toggle tab mode, where tab character and column widths are used to align tabular data
• numbers 0-9: In tab mode, toggle suppression of column number. Columns are indexed starting at 1, with number ten as 0

Features to add

• make the list of commands that can run be configurable
• with context lines, and mode should search within context window