Simple Faircamp Feed Aggregator

This script requests rss feeds from a list of Faircamp sites, then reads them and generates a small html file containing an aggregated list of releases ordered by date.

The html file can be uploaded to a Faircamp site as an add-on.

If you add the html file to the main directory of your site it should render correctly and pick up the site's theme.

A few theming options are also available in the config file (config.cfg).

What this script can do

This script will let you add a page of links to releases on other Faircamp sites chosen by you. As it generates a single, small html file, it is easy to automate, so the page can automatically be kept up to date with the latest releases from those sites. (One way to automate it is described below.) New sites can be added at any time. The aim is to foster and increase interconnection of the Faircamp community, in a way that is controlled and curated by each site owner. The page does not have to be hosted on a Faircamp site, but it needs some Faircamp assets to render correctly: styles.css, favicon.svg, favicon_light.png, logo.svg, font and background.jpg (if used).

What this script CAN'T do

This script cannot generate the 'Faircamp sites home page'! The method used is simple, generates a static html file and would not scale up to include 1000s of sites.

Usage

TLDR (Details below!)

Download the files, install feedparser and requests with pip. Then add the sites you want to include to sitelist.txt. There are a few configuration options in config.cfg.

Refresh feeds and generate the page:

python aggregator.py refresh

Just generate the page:

python aggregator.py

Installation

Download the zip, or clone this repository

git clone https://codeberg.org/yonder/simple-faircamp-feed-aggregator.git

It's recommended to use a python virtual environment.

The script requires the feedparser and requests packages, which can be installed with pip.

pip install feedparser requests

Configuration

config.cfg contains some configuration options you can edit:

[TEXT]
# The title and main heading of the aggregator page.
Page Heading = More From The Community.
# The main text of the aggregator page. This can span multiple lines
# in the config file, but extra lines must be indented as below.
Page Text = Check out more releases from the Faircamp community!
 This list is updated daily.
 
# Occasionally, no artist name is provided for a release. In this case,
# this is the text that will appear in place of their name.
Unnamed Artists Text = Mystery Artist: click to discover!
# Very long words in titles can affect the styling of the whole feed.
# Setting this to 'yes' will add a hidden word break to long words in titles
Break Long Title Words = yes
[RELEASES]
# The maximum number of releases to show
Maximum Releases To Display = 150
# The colour of the date displayed for each release.
# HTML colour names also work e.g. Date Color = red
Date Color = #778899
# Colours of the substitute 'covers' used for releases.
# They appear as thin lines above the title, as a linear gradient from 
# left to right. These can be set to 'random'.
Artist Cover Color 1 = #FFFFFF
Artist Cover Color 2 = #778899
[FILES]
# If Prune Database is set to 'yes' and you remove a site from your
# site list, all the releases from that site will be deleted from 
# the database when the script is run.
Prune Database = no
# The file containing your list of Faircamp sites to check.
Sitelist File = sitelist.txt
# The name of the generated html file.
Save As = aggregator.html
[SITE]
# If you want to add aggregator.html to the main directory of
# your Faircamp site, setting this to 'yes' will help to integrate it
# by providing a configured nav bar at the top of the page.
Faircamp Site = yes
# If you want to remove the nav bar, set this to 'no'.
Nav Bar = yes
# The name of your Faircamp site (this will appear at
# the top level of the nav bar, so the page integrates with your site).
Site Name = My Faircamp Site
[SCRIPT]
# Whether to report if a feed was fetched successfully. If set to no,
# the script will still report if there were any problems fetching a feed
Report Fetched = yes

Site list

Add the sites you want to include to the site list file (the default is 'sitelist.txt', but this can be changed in the config file). You should include the full address to the main page of the site, including https:// or http:// as appropriate. If no protocol is specified, https:// will be tried.

Running the script

Running the script with the 'refresh' argument will request rss feeds files from the sites in the site list, then generate the page.

python aggregator.py refresh

You can also provide one or more website addresses after 'refresh'. If the feeds are requested successfully, it will update the database from those sites only and add them to the site list file if they're not already there. A new page will be generated, but database pruning will be ignored.
The site addresses must include the protocol (http:// or https://)
For example:

python aggregator.py refresh https://faircamp.example.com/ http://example.com/faircamp

If you run the script with no arguments, it won't request rss feeds, but will re-generate a page from entries in the database.

python aggregator.py

Refreshing the feed

Running the refresh command again will request rss feeds again, add any new releases to the database and generate a new page.

python aggregator.py refresh

Pruning the database

If you remove a site from your site list, you may also want to remove its entries from the database. If Prune Database is set to 'yes' in config.cfg, those entries will be removed when you run the script.

releases.db

The script creates and maintains a local sqlite database called releases.db which is used to order releases by date and track release dates. The database is created in the same directory as the script.

Viewing the generated page

If you open the page by itself, it won't render correctly. It needs the styles.css, font, favicons and logo from a Faircamp site in the same directory to render correctly. If you have a .faircamp_build directory from using faircamp, you could copy it and open it from there. It should then render correctly, and pick up your site's theme. Alternatively, copy styles.css, favicon.svg, favicon_light.png, logo.svg, the font (and background.jpg if used) to the same folder as the page.

Adding the page to your site

The simplest way to add the page is to upload it to the main directory of your Faircamp site. It will then render correctly and pick up your site's theme. You can then easily link to it from your main page.
If you want to add it to a different part of your site, it needs some Faircamp assets to render correctly (styles.css, favicon.svg, favicon_light.png, logo.svg, font and background.jpg if used). These could be placed in the same directory as aggregator.html. Alternatively, you could edit aggregator.html to point to the assets in the main directory.

Automating the script

If you download/upload the script to your Faircamp server, you could use a cron job to automate it. Here's a cron guide: https://www.digitalocean.com/community/tutorials/how-to-use-cron-to-automate-tasks-ubuntu-1804
The easiest way is to write a bash script that runs aggregator.py then copies the html file to your website.

The bash script might look like something like this if localuser has the script files in a folder called 'aggregator' in their home directory :

python /home/localuser/aggregator/aggregator.py refresh
cp /home/localuser/aggregator/aggregator.html /path/to/mywebsite/

If you're using a Python virtual environment you can activate it and deactivate it in the script too.

source /home/localuser/aggregator/env/bin/activate
python /home/localuser/aggregator/aggregator.py refresh
deactivate
cp /home/localuser/aggregator/aggregator.html /path/to/mywebsite/

A cron job to run the script daily at 3 p.m. would look like this (with the bash script placed in the same directory as the python script):

0 15 * * * bash /home/localuser/aggregator/myscript

The page would then update daily and any new releases would automatically appear on your site.

Allowing or disallowing the bot

The script has the user agent Faircamp-Feed-Bot
Site maintainers can allow or disallow the bot access to their site using robots.txt

Credits

The html templates used by this script were derived from Faircamp by Simon Repp. https://codeberg.org/simonrepp/faircamp Many thanks!

License

AGPL-3.0-or-later