Scripts for maintaining a local Elasticsearch instance
- Shell 100%
| custom-mappings | feat: Add custom mappings support, and add one for virology. | |
| migration | feat: Add another migration script to remove IDs. | |
| .gitignore | chore: Fix gitignore. | |
| COPYING | feat: Initial scripts that I built while hacking on this. Needs work. | |
| create-index | feat: Silence curl, but show errors if they occur. | |
| delete-index | feat: Silence curl, but show errors if they occur. | |
| find-doi | feat: Silence curl, but show errors if they occur. | |
| mapping.json | feat: Set maximum reasonable value for ignore_above. | |
| README.md | docs: Add docs for ID removal script. | |
| upload-document | feat: Silence curl, but show errors if they occur. | |
Elasticsearch Scripts
Here we keep scripts useful for spinning up an Elasticsearch instance on top of our JSON files that can help us search over and query them.
- Scripts:
upload-document [index] [file]: Upload a single JSON document to ES.
- Utility files:
mapping.json: A mapping and settings file that can be used to build a new ES index that is ready to receive JSON files in our standard schema.
- Migration scripts: (likely one-time use)
remove-mongo-id [file]: MongoDB JSON dumps export internal IDs that clash with ES. This script removes them.patch-mongo-dates [file]: MongoDB JSON dumps produce exports where all date-format variables are exported in a sub-variable called$date. This script fixes these so that they can be parsed by ES.
All these scripts require jq available.
Environment
To set the path to the server, set the environment variable ES_SERVER_URL to
the server root (including path and authentication, if required).
To set these using mise, create a .mise.toml file containing:
[env]
ES_SERVER_URL = 'localhost:9200'
License
All scripts released under the GNU GPL v3. See the file COPYING.