Hodor PyPI
A simple html scraper with xpath or css.
pip install hodorlive
WARNING: This package by default doesn't verify ssl connections. Please check the arguments to enable them.
from hodor import Hodor from dateutil.parser import parse def date_convert(data): return parse(data) url = 'http://www.nasdaq.com/markets/stocks/symbol-change-history.aspx' CONFIG = { 'old_symbol': { 'css': '#SymbolChangeList_table tr td:nth-child(1)', 'many': True }, 'new_symbol': { 'css': '#SymbolChangeList_table tr td:nth-child(2)', 'many': True }, 'effective_date': { 'css': '#SymbolChangeList_table tr td:nth-child(3)', 'many': True, 'transform': date_convert }, '_groups': { 'data': '__all__', 'ticker_changes': ['old_symbol', 'new_symbol'] }, '_paginate_by': { 'xpath': '//*[@id="two_column_main_content_lb_NextPage"]/@href', 'many': False } } h = Hodor(url=url, config=CONFIG, pagination_max_limit=5) h.data
{'data': [{'effective_date': datetime.datetime(2016, 11, 1, 0, 0),
'new_symbol': 'ARNC',
'old_symbol': 'AA'},
{'effective_date': datetime.datetime(2016, 11, 1, 0, 0),
'new_symbol': 'ARNC$',
'old_symbol': 'AA$'},
{'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
'new_symbol': 'MALN8',
'old_symbol': 'AHUSDN2018'},
{'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
'new_symbol': 'MALN9',
'old_symbol': 'AHUSDN2019'},
{'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
'new_symbol': 'MALQ6',
'old_symbol': 'AHUSDQ2016'},
{'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
'new_symbol': 'MALQ7',
'old_symbol': 'AHUSDQ2017'},
{'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
'new_symbol': 'MALQ8',
'old_symbol': 'AHUSDQ2018'}]}ua(User-Agent)proxies(check requesocks)authcrawl_delay(crawl delay in seconds across pagination - default: 3 seconds)pagination_max_limit(max number of pages to crawl - default: 100)ssl_verify(default: False)robots(if set respects robots.txt - default: True)reppy_capacity(robots cache LRU capacity - default: 100)trim_values(if set trims output for leading and trailing whitespace - default: True)
- By default any key in the config is a rule to parse.
- Each rule can be either a
xpathor acss - Each rule can extract
manyvalues by default unless explicity set toFalse - Each rule can allow to
transformthe result with a function if provided
- Each rule can be either a
- Extra parameters include grouping (
_groups) and pagination (_paginate_by) which is also of the rule format.
- Install uv.
- Review the uvx execution model for running tools without global installs.
- Hatch documentation: https://hatch.pypa.io/latest/.
Run the release helper to build and publish wheels and source archives via Hatch:
./upload.sh
The script shells out to uvx hatch build followed by uvx hatch publish so that Hatch is executed in an ephemeral environment.
Configure credentials in ~/.pypirc as described in the PyPI configuration specification.
Example configuration:
[distutils] index-servers = pypi testpypi [pypi] repository = https://upload.pypi.org/legacy/ username = __token__ password = <pypi-token> [testpypi] repository = https://test.pypi.org/legacy/ username = __token__ password = <testpypi-token>
Replace token placeholders with secrets from the team password manager and avoid committing the file to version control.