1
0
Fork
You've already forked WARN
0
Look at WARN Notices https://sfba.social/@warn_act_ca
  • Python 67.8%
  • HCL 31.5%
  • Makefile 0.7%
2024年08月03日 18:08:34 -07:00
src More things that pylint nudged me to do 2024年08月03日 18:08:34 -07:00
terraform Bumped the Lambda base image from Python 3.8 to 3.9 2024年05月29日 21:30:14 -07:00
.gitignore Add .DS_Store to the ignore list 2023年05月07日 18:19:14 -07:00
.pylint Suppress the pylint complaints about docstrings 2023年03月27日 13:58:19 -07:00
.python-version Note the Python version that's in use in the work environment 2023年03月27日 13:58:19 -07:00
LICENSE Adding GPLv3 License file to the repo 2024年03月21日 21:10:21 -07:00
Makefile Update the pylint paths, and add terraform targets 2023年05月07日 22:00:27 -07:00
README.md Update the README.md for the new layout and purpose of the repo 2023年05月07日 22:00:52 -07:00

Repository Layout

  • src/
    • posts/ Python code for the post Lambda function
    • report/ Python code and requirements for the report Lambda function
  • terraform/ Terraform that creates infrastructure needed to run this 'serverless' setup
    • An S3 bucket for permanent storage
    • Two SQS queues (the main one, and the dead-letter queue that goes with it)
    • Two Systems Manager Parameter Store entries
    • Two Lambda functions
    • Two CloudWatch log group configurations
    • Various roles/policies that are attached to the Lambda functions to access the resources above
    • A CloudWatch alarm that checks the dead-letter queue
    • An SNS topic to which the alarm sends its alerts
    • An EventBridge "cron" rule for the report Lambda function

process_report.py

This script fetches the California WARN Act listings

https://edd.ca.gov/en/jobs_and_training/Layoff_Services_WARN https://edd.ca.gov/siteassets/files/jobs_and_training/warn/warn_report.xlsx

process_posts

This is a relatively generic script that is set up as an SQS queue listener and, when there is something there will attempt to post to the Mastodon server of choice, and post a new message to the queue for the next message.

Logic behind the setup

There are a few reasons behind this setup:

  • I'm putting a pause between the posts in a thread (doing one post per notice, and there are have generally been multiple notices on the days that California updates their spreadsheet), to be kind to the server and the folks following the account.
  • Lambdas are billed on how long they run, so time is money.
  • Lambdas also have limit to how long they can run, so if I put 10-20 seconds each post and there is a crazy number of notices that day, it might run out of time.
  • I considered Step Functions but the free limit for state changes per month seems relatively low, and the number of free SQS requests is much higher.

Example:

  • On May 4 2023 there were 25 WARN notices.
  • The report Lambda ran for 8 seconds
  • The first 'post' Lambda ran for 2 seconds, the next 23 runs were under a second, and the last one 1.2 seconds.
  • All together that that was 34-35 seconds of Lambda runtime, versus 250+ seconds if it had been attempted in a single Lambda run

Links to WARN Notice websites of other states