Is there a way to have files updated between Netlify deploys?

Question

I have a website on Netlify built with 11ty with certain data for it stored in Airtable. Using Airtable, my client can modify certain information on this website (photos and some other info), and then, while still in Airtable, push a button which sends a webhook for Netlify to rebuild the website, using up-to-date information from Airtable.

For images, I have a Gulp task that, at each build :

• Looks at all the records in the "Images" Airtable table (their API gives me filenames and URLs) and lists them in a variable in JSON format
• Then it compares this list with the list it created during the last build, stored in a file (portfoliocache.json)
• If these lists are the same, that means the images in Airtable were not changed since the last build, so do nothing, save bandwidth.
• If these lists are different, however, it deletes all files in my "images" folder, downloads them from Airtable and then optimises them, putting the new images is my "images" folder. portfoliocache.json gets overwritten with the new list. This takes some time and consumes some Netlify build minutes, so I only do this if the Airtable images have been changed.

This works great in my local Node.js environment but as Netlify uses a Git repo as its source, it takes the same portfolioCache.json and the same "images" folder every time, so all my images get re-downloaded at each Netlify build.

Is there a way to have my portfoliocache and my "images" folder updated and saved during a Netlify build so that the next build would have an up-do-date image list? Maybe there's a way to update files in my Git repository during build?

Not sure if showing my code would be relevant here but I'll be happy to, if needed.

• Luke Storry: You probably want to set up a GitHub Action that does the portfolio caching for you before netlify uses them to compare

Answer 1

Don't store the files and cache in the repository

I don't think committing your cache file and the images to the repository is a good idea. First, it doesn't really belong there, as caches should not be part of a repository and having automated commits in your repository will clutter your commit history. From a practical perspective, this might even cause an endless loop — the netlify build changes the repository which triggers a rebuild and so on.

Besides that, your approach can't work like it does on your local environment. Netlify uses a clean virtual environment for each build, so the images folder from a previous build will never exist during the next build.

Use a build plugin

Your build step appears to be working, the only issue being that processing all images every time takes too long for regular netlify builds, consuming many build minutes. In this case, why not move the caching to a netlify build plugin? Take a look at the documentation for creating your own plugin, it includes a cache utility. You could use that to store the image information in this cache and persist it in between builds. Or even cache the processed images there, only pulling images that don't exist in the cache yet.

Separate the image processing from the build step

Another solution would be to separate the image processing and the build step entirely, allowing you to store the processed images in the repository. For example, this should be possible using Github Actions or maybe the Zapier integration for Airtable + Github.