Data Platform/Systems/DataHub/Administration
Datahub Service Details
Deployment
As per Kubernetes/Deployments, we deploy these services from the deployment server, using helmfile.
datahub-next
kube-envdatahub-next-deploydse-k8s-eqiad
cd/srv/deployment-charts/helmfile.d/dse-k8s-services/datahub-next
helmfile-edse-k8s-eqiad-iapply
datahub
kube-envdatahub-deploydse-k8s-eqiad
cd/srv/deployment-charts/helmfile.d/dse-k8s-services/datahub
helmfile-edse-k8s-eqiad-iapply
Restore Indices
The MariaDB database is the cacnonical data store for DataHub and the OpenSearch indices may be recreated from the database. On occasions, it may be necessary to recreate these indices, either due to an upgrade or a problem.
There is a suspended CronJob that can be used for this purpose. We create a new CronJob based on this template.
kube-envdatahub-deploydse-k8s-eqiad
kubectlcreatejob--from=cronjob/datahub-production-restore-indices-job-templatedatahub-restore-indices-job
The logs of the job can be viewed with:
kubectlgetpods kubectllogs-fdatahub-restore-indices-job-<hash>-cdatahub-upgrade-job
The job should complete within about 20 minutes.
Alerting
The app isn't running
If you're getting paged because the app isn't running, investigate if something in the `datahub-gms-production` logs could explain the crash. In case of a recurring crash, the pod would be in CrashloopBackoff state in Kubernetes. To check whether this is the case, ssh to the deployment server and run the following commands
kube-envdatahub-deploydse-k8s-eqiad kubectlgetpods kubectllogs-fdatahub-gms-production-<hash>-cdatahub-gms-production
If no pod at all is displayed, re-deploy the app by following the deployment instructions above.