Help:Toolforge/OpenSearch

OpenSearch is a full-text search system built on Apache Lucene. It can be used to index and search data stored as JSON documents. OpenSearch is a fork of Elasticsearch, which was used previously but is no longer distributed under a free license.

OpenSearch is the technology used to power Wikimedia's CirrusSearch system.

An OpenSearch v2 cluster for all tools is available at http://opensearch.svc.tools.eqiad1.wikimedia.cloud, on the non-standard port 80 (see also Help:Toolforge/Envvars#Globally_set_environment_variables).

This OpenSearch cluster is a shared resource. All documents indexed in it can be read by anonymous users from within Toolforge. Write access is needed to create new indexes, and a password is needed to store or update documents. However, see addition notes on the talk page.

The OpenSearch servers allow anyone to read any of the indexes that it contains. This access is limited to other hosts in the Toolforge project (e.g. Kubernetes containers, and the bastion servers).

The OpenSearch service is available on port 80 at http://opensearch.svc.tools.eqiad1.wikimedia.cloud

Note: The default OpenSearch port (9200) is not used.

Our deployment is not currently using OpenSearch's multi-tenant access control functionality, due to historic reasons as Elasticsearch's free software releases did not support it. This may change in the future.

PUT, POST, or DELETE requests sent to the OpenSearch servers require HTTP Basic Authentication using a username and password specific to each tool.

Requests for write access can be made by filing this Phabricator task.

When credentials have been created they will be made available as envvars. Your tool can access them as TOOL_OPENSEARCH_USER and TOOL_OPENSEARCH_PASSWORD. (Tools that were granted access before the OpenSearch migration will use environment variables named after ELASTICSEARCH instead.)

An older procedure placed the credentials in /data/project/$TOOL/.elasticsearch.ini.

Access requests are currently processed manually and may take a few days to be fulfilled.

If you get an error message:

elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch

you have probably installed a client library which is incompatible with the version running on the server. Running the obvious pip install elasticsearchwill get you the wrong version, as will pip install opensearch. What you want is pip install opensearch-py. There's more about this on stackoverflow.

• Portal:Toolforge/Admin#Granting a tool write access to OpenSearch - administrator documentation for managing OpenSearch access

Support and administration of the WMCS resources is provided by Wikimedia Foundation staff and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

• Toolforge
• How-to-guide