mastolink2rss

A small Django app that allows folks to login with their Mastodon account - to then extract all links that are posted to the home timeline each day, and converting the most frequent links into an RSS feed for catching up with at any point.

If you don't want to host your own version you can see an example deployment at https://masto2rss.gedanken.uber.space/.

A lot of the code was inspired or even copied from Adam Hill's fediview, which takes a similar approach of a small Django app to find the most "trending" items in your timeline, not just links!

What's the algorithm for the aggregation?

It's really basic: For all the links seen during the last calendar day, sum up how often each of them was seen and rank them based on that. Then put the top X of them into an RSS feed.

Duplicate links are not counted and defined as the same link being posted more than once by the same user, regardless of when the link is encountered again. This avoids the same links appearing again and again, as some posters like to boost and reshare links to e.g. their own writings.

Deploying

This app is intentionally kept simple and small. To run it on your end you only need to use pipenv to install the basic dependencies, migrate the DB and you're basically good to go

pipenv install 
pipenv run ./manage.py migrate

The front page can also show some additional FAQ style items, which can be created either from the command line or just via the admin interface, for which you need an admin account (pipenv run ./manage.py createsuperuser).

Setting up the CRON jobs

Having your webserver running is only the first step, to actually aggregate data you will have to setup two different crontabs for background tasks:

1. regularly fetching users' timelines.

pipenv run ./manage.py get_all_links

This will fetch the most recent posts for each user and search for links in them. As the Mastodon API can only fetch the last 800 posts in a user's home timeline, run this appropriately often enough. First tests seem to indicate that every 6 hours will be fine for most folks.

2. aggregate the links for each user once a day

pipenv run ./manage.py aggregate_all_links

This will go through all the links posted during the previous calendar day, and then extract the most frequently posted ones. Given that it's based on the calendar day, this can ideally be run just after midnight.

Example CRON setup

This is how Bastian runs it on his Uberspace deployment.

1 2,8,14,20 * * * cd ~/mastolink2rss;pipenv run ./manage.py get_all_links > $HOME/logs/masto_rss_fetch.log 2>&1
15 2 * * * cd ~/mastolink2rss;pipenv run ./manage.py aggregate_all_links > $HOME/logs/masto_rss_aggregate.log 2>&1
15 3 * * * cd ~/mastolink2rss;pipenv run ./manage.py delete_stale_links > $HOME/logs/masto_rss_delete_stale.log 2>&1

Tests

There's very few tests written yet for this, but they are already setup to just run via pytest

Contributing

Please feel free to open issues and PRs for things that could be improved. This is really just a little hobby project to scratch my own itch for wanting this feature (or wanting this feature again, as I made a similar thing about 10 years ago for Twitter)!