Link Tools

Contents

1. Introduction
2. Download
3. Configuration
4. Usage
5. General Notes
6. Known Bugs
7. History

Introduction

Link Tools (linktools.sh) is a menu-driven Bash shell script for cleaning, verifying and replacing broken URLs in various file types including plain text, HTML, XML, or a database file used by blogging software such as ClassicPress, WordPress, etc.. The script is designed to work with the HTTP protocol only and includes many configuration options for fine-tuning it. The script tries to be server-friendly by delaying server queries when a domain is queried consecutively.

In cases where a single domain is being queried consecutively, or recently in the case of the Link Checker tool, a delay will occur between queries in order to avoid potential problems. The URLs in the source file are mixed before they are processed in order to help avoid such delays. Absent any delays, the script will work as fast as a web server responds (and your internet connection allows), often processing several URLs per second.

Dependencies for the script are listed in the "SCRIPT INFORMATION" section.

It is highly recommended to install ShellCheck (a Bash script linter tool) if you make any changes to the options contained within the script. Most repositories should carry the shellcheck package, however there is also a web version available at https://www.shellcheck.net/.

Link Tools is comprised of 4 tools, each accessible from the Main Menu and each of which should generally be run in order beginning with the first desired task.

Notes common to all tools

Each tool creates a text file in the 'output' directory containing a list of URLs which may require manual review. In the case of the Link Checker tool, additional information is contained for each URL including HTTP status codes and potential replacement URLs. This file is used by the Link Replacer tool and thus should not be edited.

Each tool also creates a log file in the 'logs' directory which can generally be ignored. Each log file contains list of URLs which were successfully processed and is used to resume a previous session which was terminated before all URLs in the source file were processed. These files should not be edited.

Link Cleaner

Link Cleaner cleans links in the source file by converting a subset of unsafe characters to their encoded equivalents as well as removing specified URL parameters, such as Google's 'utm' tracking parameters.

Link Checker

Link Checker checks for broken URLs in the source file. This tool uses curl-impersonate (https://github.com/lwthiker/curl-impersonate) and cURL to query web servers and saves the results for broken links to 'logs/checker-review-urls.txt'. When a broken URL is found, other websites can be automatically queried in order to locate a working one, such as web archive services. The checker-review-urls.txt file is used by the Link Replacer tool when replacing broken links.

Unfortunately cURL may fail to provide an expected result for a variety of reasons, such as idiotic web developers requiring JavaScript, specific request headers, cookies, etc.. For this reason curl-impersonate is used for the first request. This modified build of cURL aims to mimic a web browser exactly. In the event a web server attempts to prevent a request, and depending on the reason why, the request may be automatically sent again using different curl-impersonate and/or cURL options.

Link Checker relies on the HTTP status code returned from the server. In some cases, and especially for some video hosting platforms, the server may return a 200 status code even when the resource is not available.

Features include:

• broken URLs can be sent to an archive website in an attempt to find replacements
• optionally print a list of all unique URLs found in the source file
• skip specified domains
• swap one domain/TLD with another (useful for replacing short links, seized domains, etc.)

Link Replacer

Link Replacer uses the checker-review-urls.txt file provided by Link Checker in order to assist in the replacement of broken URLs in the source file with working ones. If replacements are performed, the copy of the source file in the 'output' directory is modified and, in theory, should be ready for reuse.

Link Downloader

Link Downloader is capable of downloading various file types using wget, or videos from video hosting platforms such as YouTube and all of the other websites supported by yt-dlp.

Link Replacer / Link Cleaner

Both the Link Replacer and Link Cleaner tasks share the following options:

• convert characters to ASCII
• strip URL parameters
• strip URL anchors
• encoded a subset of unsafe characters
• original and cleaned URLs can be opened in a web browser to verify correctness
• display context for the URL in order to aid decision making

Download

ZIP archive: https://codeberg.org/12bytes/broken-link-checker/archive/main.zip

TAR.GZ archive: https://codeberg.org/12bytes/broken-link-checker/archive/main.tar.gz

To be notified of updates to the Link Tools repository you may subscribe to the RSS feed: https://codeberg.org/12bytes/broken-link-checker.rss

Configuration

Configuration options for the script are contained within it in an INI-like format. You should review and edit the options contained in the following sections:

• GLOBAL OPTIONS
• LINK CLEANER OPTIONS
• LINK CHECKER OPTIONS
• LINK REPLACER OPTIONS

Be careful to not make any syntax errors when editing the options. It is highly recommended to check for common errors using ShellCheck prior to running the script: $ shellcheck -o all linktools.sh

If no problems are found ShellCheck will not print a message. If there are issues, it will provide helpful information and web resources for correcting them.

Some of the option values are Perl Compatible Regular Expressions (PCRE2) or POSIX regular expressions and it is important to not make any mistakes if changing them (ShellCheck will likely not reveal errors in regular expressions). Unless you are very comfortable with regular expressions, be sure to test any changes before running the script.

For PCRE syntax and testing see:

• Syntax: https://www.pcre.org/original/doc/html/pcrepattern.html
• Testing: https://regex101.com/ (be sure to set the expression language to PCRE2)

For POSIX syntax and testing see:

• https://archive.today/o1qxh
• https://www.rexv.org/ (be sure to set the expression language to POSIX)

Note that a single quote in an expression which is encapsulated in single quotes must itself be escaped: '\''

Each option consists of a key-value pair, the key being the part before the '=' character and the value being the part after it. There must not be any spaces before or after the '=' character. There are 3 types of keys: arrays, integers and strings, each of which can be identified by its lower-case first letter, either 'a' 'i' or 's'. Array values are typically space-separated strings contained in parenthesis. Integer values must not be quoted (floating point numbers (e.g. 1.2) are not allowed). String values must be contained within single quotes. Following are examples of correctly formatted options:

aMyArray=( 'value 1' 'value 2' '...' )
iMyInteger=1
sMyString='a value'

The sPCREUrlMatch option for the Link Checker tool is particularly important since this expression is used to match the URLs in the source file. The value for this option is a Perl Compatible Regular Expression. The default value may not be appropriate for the file you want to work with in which case it may not match any URLs, or not all URLs, or only a parts of URLs. In addition to the default sPCREUrlMatch option, there are other example options which may be used, or you may of course create your own.

Usage

For help using the script run: $ ./linktools.sh -h

To run the script, supply it with a file to work on: $ ./linktools.sh -f '<file>'. The script will copy the source file to the /output directory and work on the copy.

If you want to check an entire blog/CMS website for broken links, such as a WordPress website, you should export its content from the administration back-end if possible rather than work on a database file which could have unintended consequences.

When running the Link Checker or Link Replacer tools, you may terminate processing at any time and later resume where you left off. The Link Cleaner tool will start processing the URLs all over again if it is terminated early, however this tool runs very quickly.

General Notes

• When querying an archive website in order to find a cached copy of the URL, the protocol (http/https) is stripped from the original URL. This seems to return results that may not otherwise be returned.

Known Bugs

With the exception of the following known issues, please report any issues you may find.

• The Link Checker tool may occasionally produce the following error which can be safely ignored: warning: command substitution: ignored null byte in input. The "error" is related to compression when the server compresses the web page before sending it.

History

Link Tools was born out of frustration with the WordPress plugin, Broken Link Checker. For a very long time the WordPress plugin was/is buggy and has not received the attention it deserved given it was the only fairly comprehensive link checker plugin for WordPress. After the developer(s) failed to fix some non-trivial issues, they then apparently sold the plugin to an unethical developer who added 3rd party crap to it, and so i decided to create my own solution. Link Tools also provides more granular control over how the script works and it has some features the WordPress plugin doesn't.