A service (also known as a "daemon") is a process that performs tasks in the background and responds to system events.
Services can be written using any language. We use Python in these examples as it is one of the most versatile languages out there.
For more information, be sure to read our blogpost on the subject.
To run any of the examples, you will need Python 3. We also recommend using a virtual environment.
python3 -m venv venv
First, create a script that scrapes a website. Make sure the script also handles OS signals so that it exits gracefully.
linux_scrape.py:
import json import re import signal from pathlib import Path import requests from bs4 import BeautifulSoup class SignalHandler: shutdown_requested = False def __init__(self): signal.signal(signal.SIGINT, self.request_shutdown) signal.signal(signal.SIGTERM, self.request_shutdown) def request_shutdown(self, *args): print('Request to shutdown received, stopping') self.shutdown_requested = True def can_run(self): return not self.shutdown_requested signal_handler = SignalHandler() urls = [ 'https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html', 'https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html', 'https://books.toscrape.com/catalogue/sharp-objects_997/index.html', ] index = 0 while signal_handler.can_run(): url = urls[index % len(urls)] index += 1 print('Scraping url', url) response = requests.get(url) soup = BeautifulSoup(response.content, 'html.parser') book_name = soup.select_one('.product_main').h1.text rows = soup.select('.table.table-striped tr') product_info = {row.th.text: row.td.text for row in rows} data_folder = Path('./data') data_folder.mkdir(parents=True, exist_ok=True) json_file_name = re.sub('[\': ]', '-', book_name) json_file_path = data_folder / f'{json_file_name}.json' with open(json_file_path, 'w') as book_file: json.dump(product_info, book_file)
Then, create a systemd configuration file.
/etc/systemd/system/book-scraper.service:
[Unit]
Description=A script for scraping the book information
After=syslog.target network.target
[Service]
WorkingDirectory=/home/oxylabs/python-script-service/src/systemd
ExecStart=/home/oxylabs/python-script-service/venv/bin/python3 main.py
Restart=always
RestartSec=120
[Install]
WantedBy=multi-user.target
Make sure to adjust the paths based on your actual script location.
A fully working example can be found here.
To create a Windows service, you will need to implement methods such as SvcDoRun and SvcStop and handle events sent by the operating system.
windows_scrape.py:
import sys import servicemanager import win32event import win32service import win32serviceutil import json import re from pathlib import Path import requests from bs4 import BeautifulSoup class BookScraperService(win32serviceutil.ServiceFramework): _svc_name_ = 'BookScraperService' _svc_display_name_ = 'BookScraperService' _svc_description_ = 'Constantly updates the info about books' def __init__(self, args): win32serviceutil.ServiceFramework.__init__(self, args) self.event = win32event.CreateEvent(None, 0, 0, None) def GetAcceptedControls(self): result = win32serviceutil.ServiceFramework.GetAcceptedControls(self) result |= win32service.SERVICE_ACCEPT_PRESHUTDOWN return result def SvcDoRun(self): urls = [ 'https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html', 'https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html', 'https://books.toscrape.com/catalogue/sharp-objects_997/index.html', ] index = 0 while True: result = win32event.WaitForSingleObject(self.event, 5000) if result == win32event.WAIT_OBJECT_0: break url = urls[index % len(urls)] index += 1 print('Scraping url', url) response = requests.get(url) soup = BeautifulSoup(response.content, 'html.parser') book_name = soup.select_one('.product_main').h1.text rows = soup.select('.table.table-striped tr') product_info = {row.th.text: row.td.text for row in rows} data_folder = Path('C:\\Users\\User\\Scraper\\dist\\scrape\\data') data_folder.mkdir(parents=True, exist_ok=True) json_file_name = re.sub('[\': ]', '-', book_name) json_file_path = data_folder / f'{json_file_name}.json' with open(json_file_path, 'w') as book_file: json.dump(product_info, book_file) def SvcStop(self): self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) win32event.SetEvent(self.event) if __name__ == '__main__': if len(sys.argv) == 1: servicemanager.Initialize() servicemanager.PrepareToHostSingle(BookScraperService) servicemanager.StartServiceCtrlDispatcher() else: win32serviceutil.HandleCommandLine(BookScraperService)
Next, install dependencies and run a post-install script.
PS C:\> cd C:\Users\User\Scraper
PS C:\Users\User\Scraper> .\venv\Scripts\pip install pypiwin32
PS C:\Users\User\Scraper> .\venv\Scripts\pywin32_postinstall.py -install
Bundle your script into an executable.
PS C:\Users\User\Scraper> venv\Scripts\pyinstaller --hiddenimport win32timezone -F scrape.py
And finally, install your newly-created service.
PS C:\Users\User\Scraper> .\dist\scrape.exe install
Installing service BookScraper
Changing service configuration
Service updated
PS C:\Users\User\Scraper> .\dist\scrape.exe start
Starting service BookScraper
PS C:\Users\User\Scripts>
A fully working example can be found here.
Instead of dealing with the Windows service layer directly, you can use the NSSM (Non-Sucking Service Manager).
Install NSSM by visiting the official website. Extract it to a folder of your choice and add the folder to your PATH environment variable for convenience.
Once you have NSSM installed, simplify your script by getting rid of all Windows-specific methods and definitions.
simple_scrape.py:
import json import re from pathlib import Path import requests from bs4 import BeautifulSoup urls = ['https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html', 'https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html', 'https://books.toscrape.com/catalogue/sharp-objects_997/index.html', ] index = 0 while True: url = urls[index % len(urls)] index += 1 print('Scraping url', url) response = requests.get(url) soup = BeautifulSoup(response.content, 'html.parser') book_name = soup.select_one('.product_main').h1.text rows = soup.select('.table.table-striped tr') product_info = {row.th.text: row.td.text for row in rows} data_folder = Path('C:\\Users\\User\\Scraper\\data') data_folder.mkdir(parents=True, exist_ok=True) json_file_name = re.sub('[\': ]', '-', book_name) json_file_path = data_folder / f'{json_file_name}.json' with open(json_file_path, 'w') as book_file: json.dump(product_info, book_file)
Bundle your script into an executable.
PS C:\Users\User\Scraper> venv\Scripts\pyinstaller -F simple_scrape.py
And finally, install the script using NSSM.
PS C:\> nssm.exe install SimpleScrape C:\Users\User\Scraper\dist\simple_scrape.exe
PS C:\Users\User\Scraper> .\venv\Scripts\pip install pypiwin32
A fully working script can be found here.