In order to contribute, please follow the Contributing Guidelines.
The Hyperdrive client is managing the Hyperdrive Controller endpoint for CloudServer. It exposes a simple API for a group of HDController.
It does not change the data, it is merely a proxy to HDController.
Basically a subset of HTTP 1.1 using POST, GET and DELETE verbs to create, to get and to delete an object.
On POST, the HDController will create a key that can be used in GET / DELETE to access the data stored.
npm install --save scality/hdclient
# Check for dependencies vulnerabilities
npm auditTo use the linter and run the tests:
# All tests npm test tests/ # Other options and Mocha help npm test -- -h # Code coverage npm run coverage
# Generating JSDoc, to start browsing open docs/jsdoc/index.html
npm run jsdocNode-clinic is installed by default as dev dependency. It can be used to diagnose general performance, I/O specific, event-loop issues, etc. It can also be used to generate flame graphs. Below are some usage examples usage. Note that data acquisition and visualization can be sepearated, they are not in the examples.
# Node clinic diagnosis tool - help npm run clinic # Clinic doctor help npm run clinic doctor # Diagnosing on the same machine - use --on-port to kickstart load generator NODE_ENV=production npm run clinic doctor -- \ --on-port='for i in {..10}; do curl -XPUT --data-binary @/etc/hosts "http://localhost:6767/bucket/testobj$i" ; done' \ -- node scripts/server.js 6767 scripts/example_hdclient_proxy.conf.json NODE_ENV=production npm run clinic doctor -- \ -- node scripts/server.js 6767 scripts/example_hdclient_proxy.conf.json & pid=$! # Start and wait for load generator to finish from somewhere else... kill -SIGINT $pid # Or keep process in foreground and Ctrl-C when done wait $pid # Flame graph help npm run clinic flame # Flame graph - load-generator on different machine, to be started whenever the server is up NODE_ENV=production npm run clinic flame -- node scripts/server.js <port> <config file> ... Ctrl-C Analysing data ...
Because deploying the full S3 server might be too much of a hassle for your specific need, a HTTP server using HdClient is provided. The mapping between object key and internal keys (the ones actually stored on the hyperdrive) is stored in-memory only. The internal keys are not accessible on the outside, mirrorring behavior of Zenko-like deployment.
# Start Hyperdrive 'proxy' # example conf assumes 1 hdcontroller listening on localhost:18888 NODE_ENV=production node scripts/server.js 8888 scripts/example_hdclient_proxy.conf.json & # Have fun curl -XPUT --data-binary @/etc/hosts -v http://localhost:8888/mybucket/testobj * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 8888 (#0) > PUT /mybucket/testobj HTTP/1.1 > Host: localhost:8888 > User-Agent: curl/7.47.0 > Accept: */* > Content-Length: 267 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 267 out of 267 bytes < HTTP/1.1 200 OK < Date: 2018年6月27日 10:44:32 GMT < Connection: keep-alive < Transfer-Encoding: chunked < * Connection #0 to host localhost left intact curl -v http://localhost:8888/mybucket/testobj * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 8888 (#0) > GET /mybucket/testobj HTTP/1.1 > Host: localhost:8888 > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 OK < Content-Length: 267 < Date: 2018年6月27日 10:44:39 GMT < Connection: keep-alive < * Connection #0 to host localhost left intact <payload...> # <url>/<bucket>/<object>/<version> curl -XDELETE -v http://localhost:8888/mybucket/testobj/64 * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 8888 (#0) > DELETE /mybucket/testobj HTTP/1.1 > Host: localhost:8888 > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 OK < Date: 2018年6月27日 10:44:58 GMT < Connection: keep-alive < Transfer-Encoding: chunked < * Connection #0 to host localhost left intact
How to run integrated hyperdrive client inside S3 or Zenko deployment? This section is only a work in progress since actual S3 integration code is not yet merged.
# Checkout S3 repository and checkout proper hdclient integration branch git clone https://github.com/scality/CloudServer.git cd CloudServer/ git checkout feature/RING-28500-add-hyperdrive-client-data-backend-real # Modify package.json to use the version of hdclient you want in case latest development/1.0 is not good # To use a local repository # sed s%scality/hdclient%file:<path to hdclient repository% package.json # To use a tag or commit # sed -i s%scality/hdclient%scality/hdclient#<tag/commit>% package.json # Add new locationConstraints # Region us-east-1 is mandatory, since the default config still references it cat <<EOF > hdclient_locationConfig.json { "us-east-1": { "type": "file", "objectId": "iod1", "legacyAwsBehavior": true, "details": {} }, "hyperdrive-cluster-1": { "type": "scality", "objectId": "oid2", "legacyAwsBehavior": false, "details": { "connector": { "hdclient" : { "bootstraps": "localhost:18888", "path": "/store/", } } } } } EOF # Pattern match restEndpoints - haven't found a better way yet... # Edit config.json restEndpoints section to use hyperdrive-cluster-1 # e.g. to map localhost onto hdclient: sed -i %"localhost": "us-east-1"%"localhost": "hyperdrive-cluster-1" # e.g. to map 127.0.0.1 onto hdclient: sed -i %"127.0.0.1": "us-east-1"%"127.0.0.1": "hyperdrive-cluster-1" # Install dependencies npm install # Start a Kafka instance on passed kafkaBrokers parameters (in the example 127.0.0.1:6666) # Start CloudServer (memory backend ie metadata in-memory) # More informations inside S3 repository documentation NODE_ENV=production S3DATA=multiple S3_LOCATION_FILE=hdclient_locationConfig.json npm run mem_backend
In a separate tab, have fun with AWS CLI
# Running S3 server uses default accessKey and secretKey export AWS_ACCESS_KEY_ID=accessKey1 export AWS_SECRET_ACCESS_KEY=verySecretKey1 # Create a bucket aws --endpoint-url=http://localhost:8000 s3 mb s3://brandnewbucket # List buckets aws --endpoint-url=http://localhost:8000 s3 ls # Put data aws --endpoint-url=http://localhost:8000 s3 cp /etc/hosts s3://brandnewbucket/shiny_new_object # List bucket content aws --endpoint-url=http://localhost:8000 s3 ls s3://brandnewbucket # Get data aws --endpoint-url=http://localhost:8000 s3 cp s3://brandnewbucket/shiny_new_object /tmp/retrieved # Delete data aws --endpoint-url=http://localhost:8000 s3 rm s3://brandnewbucket/shiny_new_object