SessionStorage

This page is currently a draft.
Material may not yet be complete, information may presently be omitted, and certain parts of the content may be subject to radical, rapid alteration. More information pertaining to this may be available on the talk page.

Session storage provides application servers with shared access to user session data, making this service critical to site functionality for logged in users. It utilizes Kask, a RESTful opaque key/value storage abstraction on top of Apache Cassandra, deployed in an active/active configuration in both the eqiad and codfw data-centers.

Production Environment

Service endpoints

Cassandra

Given the sensitivity of the information stored, a dedicated Cassandra cluster is used.

Monitoring and debugging

Sessionstore has a Grafana dashboard.

Failures in sessionstore operation are very likely to have an impact upon user sessions in a variety of ways. Reports of CSRF token failures are a frequent giveaway. Spiking errors of save failures on the Edit Count dashboard and spiking MediaWiki login errors are also a common sign of issues with sessions.

Common Tasks

Deployment

k8s

Follow the Kubernetes documentation for the sessionstore service.

The namespace and Kubernetes service is called "sessionstore" while the Helm chart used is named "kask".

Deployment-prep

Unlike production, Docker is used to deploy the session storage service in the deployment-prep environment.

In a web browser:

Visit https://horizon.wikimedia.org
Select the deployment-prep project
Select Project → Compute → Instances (in the sidebar)
Locate and select the deployment-sessionstore06 instance
Select the Puppet Configuration tab
Scroll to the bottom of the page, and click edit for Hiera Config (see below)
Click "Apply Changes"
Run sudo run-puppet-agent from deployment-sessionstore06

Sample Hiera Config

profile::cassandra::instances:
deployment-sessionstore06.deployment-prep.eqiad1.wikimedia.cloud:{}
profile::docker::engine::declare_service:true
profile::docker::engine::settings:{}
profile::docker::engine::version:1.12.6-0~debian-jessie
profile::docker::runner::service_defs:
mediawiki-services-kask:
config:
base_uri:/sessions/v1
cassandra:
authentication:
password:cassandra
username:cassandra
hosts:
-deployment-sessionstore06.deployment-prep.eqiad1.wikimedia.cloud
keyspace:sessions
table:values
default_ttl:86400
listen_address:0.0.0.0
listen_port:8080
openapi_spec:/go/src/gerrit.wikimedia.org/r/mediawiki/services/kask/openapi.yaml
service_name:sessions
namespace:wikimedia
override_cmd:./kask --config /etc/mediawiki-services-kask/config.yaml
port:8080
version:v1.0.10

The contents of mediawiki-services-config.config are written verbatim to the container as /etc/mediawiki-services-kask/config.yaml (the Kask configuration file)

Use mediawiki-services-kask.version to specify the Kask Docker image tag. Use mediawiki-services-kask.port to assign the port Kask will listen on.

Pool and Depool

Pool and depool using the sre.discovery.service-route cookbook, for example:

eevans@cumin1003:~$ sudocookbooksre.discovery.service-route-r'Down for maintenance'\
depoolcodfwsessionstore
...
eevans@cumin1003:~$

Performance testing

The Kask image ships with a simple —purpose built— load generator/benchmark utility that can be used to evaluate performance. This is particularly useful for validating a new release in staging, prior to deploying in production. It's best run from a deployment host:

$ VERSION=1.0.18# Or the current Kask version
$ dockerrun-it--rm--entrypoint="./bench"--networkhostdocker-registry.wikimedia.org/repos/mediawiki/services/kask:v$VERSION-urlhttps://staging.svc.eqiad.wmnet:8081/sessions/v1-post-ratio0.3-c20-d5m
Starting load test...
URL: https://staging.svc.eqiad.wmnet:8081/sessions/v1
Concurrency: 20
Duration: 5m0s
POST Ratio: 0.30
Payload Size: 1024 bytes
Timeout: 30s
=== Load Test Results ===
Total Duration: 5m0.01486963s
Total Requests: 184278
Successful Requests: 184278
Failed Requests: 0
POST Requests: 55392
GET Requests: 128886
Requests/sec: 614.23
Bytes Transferred: 131979264
Transfer Rate: 429.60 KB/sec
Average Latency: 32.545064ms
Min Latency: 31.457123ms
Max Latency: 423.059581ms
Status Code Distribution:
 201: 55392 requests
 200: 128886 requests
$

Runbook

SessionStorage/Runbook

Production Environment

Service endpoints

Discovery

eqiad

codfw

staging

Prometheus metrics

Readiness