Watch 1

Star 0

HTTPS

Download ZIP Download TAR.GZ Download BUNDLE Open with VS Code Open with VSCodium Open with Intellij IDEA

Mastodon to Markdown Converter

A collection of Python scripts to convert Mastodon toots to Markdown format with full conversation threads, media attachments, and proper metadata.

Overview

This toolset consists of:

• post_to_markdown.py - Core script to fetch and convert individual toots
• process_bookmarks.py - Batch process bookmarks from Mastodon export
• process_likes.py - Batch process likes from Mastodon export
• process_outbox.py - Batch process your posts from Mastodon export
• process_all.py - Master script to process all three JSON files at once
• fediverse_adapters.py - Support for non-Mastodon fediverse platforms

Features

• Full conversation threads - Shows complete reply chains, not just individual toots
• Media downloads - Saves avatars and attachments locally
• Shared resources - Efficient storage with deduplicated avatars and media
• Rich metadata - YAML frontmatter with title, author, date, tags, etc.
• Hashtag extraction - Automatically collects hashtags from entire threads
• Proper formatting - Converts HTML to Markdown with proper formatting
• MkDocs compatibility - Video paths optimized for MkDocs rendering
• Fediverse support - Works with Misskey/Sharkey/Firefish, Lemmy, BookWyrm, Pleroma, and other ActivityPub platforms
• Library integration - Uses Mastodon.py and pythorhead for robust API interactions
• Error handling - Network errors don't create empty/error files

Installation

Required Dependencies

Python 3.7+ and the following libraries for optimal functionality:

# Install recommended libraries for better performance and reliability
pip install Mastodon.py pythorhead
# Or install individually:
pip install Mastodon.py # For Mastodon/Pleroma/Akkoma support
pip install pythorhead # For Lemmy support

Note: The scripts will work without these libraries using direct API calls and fallbacks, but the libraries provide:

• Better error handling and rate limiting
• More reliable API interactions
• Automatic retries and pagination
• Reduced code complexity

Optional Setup

# Make scripts executable (optional)
chmod +x post_to_markdown.py process_*.py

Usage

Individual Toot Conversion

Convert a single toot to Markdown:

python post_to_markdown.py <post_url> <output_folder>

Example:

# Mastodon
python post_to_markdown.py https://mastodon.social/@username/123456 output/posts
# Lemmy
python post_to_markdown.py https://discuss.tchncs.de/post/30348291 output/posts
# BookWyrm
python post_to_markdown.py https://bookwyrm.social/user/username/quotation/12345 output/posts
# Pleroma
python post_to_markdown.py https://fe.disroot.org/objects/abc-def-uuid output/posts

Options:

• -o, --output - Specify custom output filename (default: auto-generated)

Batch Processing from Mastodon Export

Getting Your Official Mastodon Archive

These scripts work with the official Mastodon archive export from your instance:

1. Log into your Mastodon instance (e.g., mastodon.social)
2. Go to Settings → Import and export → Request your archive
3. Wait for an email notification (usually within a few hours)
4. Download the .zip file from the email link
5. Extract the archive to a folder (e.g., official_export/)

The extracted folder contains these JSON files that the scripts process:

• bookmarks.json
• likes.json
• outbox.json

Process Bookmarks

python process_bookmarks.py <bookmarks.json> <output_base_path>

Example:

python process_bookmarks.py official_export/bookmarks.json username@instance/markdown

This will create markdown files in username@instance/markdown/bookmarks/

Process Likes

python process_likes.py <likes.json> <output_base_path>

Example:

python process_likes.py official_export/likes.json username@instance/markdown

This will create markdown files in username@instance/markdown/likes/

Process Your Posts

python process_outbox.py <outbox.json> <output_base_path>

Example:

python process_outbox.py official_export/outbox.json username@instance/markdown

This will create markdown files in username@instance/markdown/posts/

Process Everything at Once

For convenience, you can process all three files with a single command:

python process_all.py <export_folder> <output_base_path>

Example:

python process_all.py official_export username@instance/markdown

This will process bookmarks.json, likes.json, and outbox.json sequentially.

Options:

• --skip-bookmarks - Skip processing bookmarks.json
• --skip-likes - Skip processing likes.json
• --skip-outbox - Skip processing outbox.json

Example (skip likes):

python process_all.py official_export username@instance/markdown --skip-likes

Output Structure

The scripts create a clean directory structure with shared resources:

username@instance/markdown/
├── posts/
│ ├── 2025年01月06日-toot-123456.md
│ └── 2025年01月07日-toot-789012.md
├── bookmarks/
│ └── 2025年01月08日-toot-345678.md
├── likes/
│ └── 2025年01月09日-toot-901234.md
├── avatars/ ← Shared across all folders
│ ├── user1-avatar.png
│ └── user2-avatar.webp
└── media/ ← Shared across all folders
 ├── image1.jpg
 └── video1.mp4

All markdown files reference media using relative paths (../avatars/, ../media/), so the structure remains portable.

Markdown Format

Each toot is saved as a markdown file with YAML frontmatter:

---
title: Mon 06 Jan 2025 14:30
author: username@mastodon.social
instance: mastodon.social
post_url: https://mastodon.social/@username/123456
date: 2025年01月06日 14:30:15.123000-07:00
tags:
 - linux
 - opensource
---
![User Name](../avatars/hash.png){.avatar} [**User Name** (@username@mastodon.social)](https://mastodon.social/@username)
This is the toot content with proper markdown formatting.
![Image description](../media/image-hash.jpg)
<video width='1068' height='728' controls>
<!-- Local path: ../media/video-hash.mp4 -->
<source src='../../media/video-hash.mp4' type='video/mp4'>
</video>

Note on media paths:

• Images: Use ../media/ (works for both local viewing and MkDocs)
• Videos: Use ../../media/ in the src attribute for MkDocs compatibility, with a comment showing the local path (../media/)

Frontmatter Fields

• title - Human-readable date/time (formatted for your timezone)
• author - Full handle of the post author (username@instance)
• instance - Mastodon instance domain
• post_url - Original URL of the post
• date - ISO 8601 timestamp with timezone
• tags - Array of hashtags from the entire conversation thread (optional)

Conversation Threads

When a toot is part of a reply chain, the script automatically:

1. Walks up to find the root of the conversation
2. Fetches all parent toots
3. Downloads avatars for all participants
4. Displays the complete conversation in chronological order

Example with thread:

![Original Poster](../avatars/user1.png){.avatar} [**Original Poster** (@user1@instance)](...)
This is the original toot
---
![Replier](../avatars/user2.png){.avatar} [**Replier** (@user2@instance)](...)
This is a reply to the original toot
---
![You](../avatars/you.png){.avatar} [**You** (@you@instance)](...)
This is your reply to the reply

Error Handling

The batch processing scripts:

• Continue processing even if individual toots fail
• Print progress for each toot
• Display a summary at the end with success/error counts
• Have a 60-second timeout per toot to prevent hanging

For inaccessible toots (deleted, private, etc.), an error document is created with details about the retrieval attempt.

Performance Notes

• Media deduplication - Files are only downloaded once based on URL hash
• Shared storage - Avatar and media folders are shared across posts/bookmarks/likes
• Progress tracking - Real-time progress display during batch processing
• Resume capability - Scripts automatically skip already-processed toots
• Concurrent-safe - Uses file existence checks to avoid re-downloading
• Error logging - Errors written to timestamped log files for easy review

Rate Limiting & Authentication

Rate Limiting Detection

The script automatically detects rate limiting (HTTP 429 "Too Many Requests") and:

• Retries with exponential backoff (5s, 10s, 20s)
• Makes up to 3 retry attempts
• Provides clear feedback when rate limited

Using Authentication

To avoid rate limits and access private toots, set an authentication token for your instance:

export MASTODON_AUTH_TOKEN="your_access_token_here"
export MASTODON_AUTH_INSTANCE="mastodon.social"
python post_to_markdown.py <url> <output>

Getting your token:

1. Log into your Mastodon instance
2. Go to Settings → Development → New Application
3. Name it (e.g., "Toot Archiver")
4. Required scopes: read:statuses, read:accounts
5. Click Submit and copy your access token
6. Set it in your environment: export MASTODON_AUTH_TOKEN="your_token"
7. Set the instance: export MASTODON_AUTH_INSTANCE="your.instance.com"

Important: The token will ONLY be used when fetching toots from the instance specified in MASTODON_AUTH_INSTANCE. This prevents accidentally sending your token to other instances.

The token will be automatically used by all processing scripts when accessing the matching instance.

Security Note: Keep your token private! Don't commit it to version control.

Resume Processing

The batch processing scripts automatically skip toots that have already been converted to markdown:

# First run - processes all toots
python process_all.py official_export output
# Interrupted? Just run again - skips already-processed toots
python process_all.py official_export output

How it works:

• Before fetching each post, checks if a markdown file already exists
• Looks for files matching the pattern *--{ID}.md in the output folder
• Skips fetching if file exists (saves API requests and time)
• Media files also deduplicated (only downloaded once)
• Safe to run multiple times without wasting resources

Benefits:

• Interrupt processing anytime (Ctrl+C) and resume later
• Avoid consuming rate limits re-fetching already-processed toots
• Add new bookmarks/likes to export and re-run to get only new items
• Failed toots can be retried without re-processing successful ones

Error Logging

Batch processing scripts automatically log errors to timestamped files:

python process_bookmarks.py bookmarks.json output
# Creates: output/bookmarks/errors_20251104_194723.log (if errors occur)

Error log location:

• Same folder as the output markdown files
• Named with timestamp: errors_YYYYMMDD_HHMMSS.log
• Only created if errors occur during processing

Error log contains:

• Full error messages (not truncated)
• URLs that failed to process
• Exit codes and tracebacks
• Easy to review after long batch runs

Example error log entry:

[16/325] FAILED: https://fosstodon.org/users/test/statuses/12345
Exit code: 1
Error output:
 Error: Could not parse toot URL: https://fosstodon.org/users/test/statuses/12345

Fediverse Platform Support

The scripts support multiple fediverse platforms beyond Mastodon:

Supported platforms:

• Mastodon - Full support including threading (via Mastodon.py library)
• Pleroma/Akkoma - Full support, Mastodon API compatible (via Mastodon.py library)
• Misskey/Sharkey/Firefish - Full support via native Misskey API with timestamps and usernames
• Lemmy - Reddit-like discussion posts (via pythorhead library)
• BookWyrm - Book quotations, reviews, and comments (via ActivityPub)

How it works:

• Automatically detects platform from URL pattern
• Uses platform-specific libraries where available (Mastodon.py, pythorhead)
• Falls back to native APIs (Misskey API) or ActivityPub protocol
• Falls back to HTML parsing as last resort
• Normalizes to common format for markdown conversion
• Same output structure across all platforms

Platform-specific features:

• Mastodon/Pleroma: Full threading, proper rate limiting, avatar downloads
• Misskey/Sharkey/Firefish: Full post data including timestamps and complete usernames (e.g., user@instance.zone)
• Lemmy: Post content and metadata via pythorhead library
• BookWyrm: Basic post content via ActivityPub

Limitations for some platforms:

• Threading not yet supported for Misskey and Lemmy (shows single post only)
• Some platform-specific features may not be preserved

Example URLs:

# Mastodon
https://mastodon.social/@Gargron/1
# Misskey/Sharkey/Firefish
https://blahaj.zone/notes/a9e5t96um44p01xe
# Lemmy post
https://discuss.tchncs.de/post/30348291
# BookWyrm quotation
https://bookwyrm.social/user/absurd-sage/quotation/8476755
# Pleroma post
https://fe.disroot.org/objects/6ee93ffd-7b19-4c3e-bcbe-0e263e841087

These URLs work in your bookmarks/likes exports and can be processed with the batch scripts just like Mastodon toots!

Tips

1. Timezone - The script uses America/Denver timezone. Edit format_date_title() in post_to_markdown.py to change this.

2. Large exports - For large archives, process in batches or use timeout command to limit execution time.

3. Rate limiting - With authentication, you can process many toots quickly. Without it, expect rate limits after ~300 requests. The script handles this automatically with retries.

Non-Mastodon URLs - Bookmarks/likes may contain URLs from supported fediverse platforms (Lemmy, BookWyrm, Pleroma) which will be converted automatically. Unsupported platforms will be skipped with a clear error message.

5. Multiple instances - Your bookmarks/likes may include toots from many different instances. Set MASTODON_AUTH_INSTANCE to your home instance to authenticate requests to it. Other instances will be accessed without authentication.

6. Resume processing - You can safely interrupt and restart processing. Already-converted toots are automatically skipped, saving API requests and time.

7. MkDocs users - Video paths in the generated markdown use ../../media/ for proper rendering in MkDocs. A comment with the local path (../media/) is included for reference.

8. Fediverse platforms - Lemmy, BookWyrm, and Pleroma posts are automatically detected and converted. No special handling needed in your bookmarks/likes exports.

Troubleshooting

Script can't find post_to_markdown.py:

• Ensure all scripts are in the same directory
• Use absolute paths if needed

Rate limiting (429 errors):

• The script will automatically retry with backoff
• If you hit rate limits frequently, set MASTODON_AUTH_TOKEN and MASTODON_AUTH_INSTANCE
• Consider adding delays between batches for unauthenticated requests

Token not being used:

• Verify both MASTODON_AUTH_TOKEN and MASTODON_AUTH_INSTANCE are set
• Check that MASTODON_AUTH_INSTANCE matches the instance domain (e.g., "mastodon.social")
• Script will warn if token is set but instance is not

Timeout errors:

• Some toots may take longer to fetch (large threads, slow servers)
• The timeout is currently set to 60 seconds per toot

"Could not parse toot URL" error:

• The URL format may not be recognized
• Currently supports standard Mastodon URL patterns

Avatar/media not downloading:

• Check network connectivity
• Some instances may block automated downloads
• Files may be too large or have connection issues

Many errors during processing:

• Check the error log file in the output folder (e.g., bookmarks/errors_YYYYMMDD_HHMMSS.log)
• Error log contains full details for each failed toot
• Common causes: deleted toots, private content, non-Mastodon URLs

License

These scripts are provided as-is for personal use. No warranty or support is provided.

Contributing

Feel free to modify these scripts for your needs. Common modifications:

• Change timezone in format_date_title()
• Adjust timeout values in processing scripts
• Modify markdown formatting in create_markdown()
• Add rate limiting in batch processors