Healthchecks #7686

Here's one example -- in this case the server is not known. This particular issue is less relevant with a server launcher - but other errors like permissions would be the same.

Note that the real info we need is in relatedHTTPCode.

➜ ~ http --verify=no --pretty=format GET "https://44623abc-eu-gb.lb.appdomain.cloud:9443/open-metadata/admin-services/users/admin/servers/cocoMDS99/instance/status"
HTTP/1.1 200
Connection: keep-alive
Content-Type: application/json
Date: 2023年5月18日 17:08:15 GMT
Keep-Alive: timeout=60
Transfer-Encoding: chunked
{
 "actionDescription": "getActiveServerStatus",
 "class": "OMAGServerStatusResponse",
 "exceptionClassName": "org.odpi.openmetadata.frameworks.connectors.ffdc.InvalidParameterException",
 "exceptionErrorMessage": "OMAG-MULTI-TENANT-404-001 The OMAG Server cocoMDS99 is not available to service a request from user admin",
 "exceptionErrorMessageId": "OMAG-MULTI-TENANT-404-001",
 "exceptionErrorMessageParameters": [
 "cocoMDS99",
 "admin"
 ],
 "exceptionProperties": {
 "parameterName": "serverName",
 "serverName": "cocoMDS99"
 },
 "exceptionSystemAction": "The system is unable to process the request because the server is not running on the called platform.",
 "exceptionUserAction": "Verify that the correct server is being called on the correct platform and that this server is running. Retry the request when the server is available.",
 "relatedHTTPCode": 404
}

From a script we can get this out with

➜ ~ cat > /tmp/t
{
 "actionDescription": "getActiveServerStatus",
 "class": "OMAGServerStatusResponse",
 "exceptionClassName": "org.odpi.openmetadata.frameworks.connectors.ffdc.InvalidParameterException",
 "exceptionErrorMessage": "OMAG-MULTI-TENANT-404-001 The OMAG Server cocoMDS99 is not available to service a request from user admin",
 "exceptionErrorMessageId": "OMAG-MULTI-TENANT-404-001",
 "exceptionErrorMessageParameters": [
 "cocoMDS99",
 "admin"
 ],
 "exceptionProperties": {
 "parameterName": "serverName",
 "serverName": "cocoMDS99"
 },
 "exceptionSystemAction": "The system is unable to process the request because the server is not running on the called platform.",
 "exceptionUserAction": "Verify that the correct server is being called on the correct platform and that this server is running. Retry the request when the server is available.",
 "relatedHTTPCode": 404
}
➜ ~ cat /tmp/t | jq .relatedHTTPCode
404

which may be sufficient (though jq is likely not on our base image, nor python.. so may need to add - or do in script

Adding 'jq' (I use this - or python - mostly) is probably the most pragmatic & understandable. An awk version could work for this case, but less readable/ likely to be more buggy

If we assume that a good status is both only true, and sufficient, when the relatedHTTPCode is 200, then a very simple check can be done by doing an exec with the above curl command, and piping to a simple regex -- in the test case this might becat /tmp/t | grep '.*\"relatedHTTPCode\": 200'

This does not solve a more general case, has or even begin to address aggregated results, but will work today with the existing platform in determining a server is at least running. Generally an external check is also much better than relying on exec too. However, I will add this note to the docs

HOWEVER ...

A server starting up will also return 200 -- yet it is not ready

{
"class": "OMAGServerStatusResponse",
"relatedHTTPCode": 200,
"serverStatus": {
 "serverName": "cocoMDS2",
 "serverType": "Metadata Server",
 "serverActiveStatus": "STARTING",
 "services": [
 {
 "serviceName": "Open Metadata Repository Services (OMRS)",
 "serviceStatus": "STARTING"
 }
 ]
}

Therefore the relatedHTTPCode is insufficient.

Using jq seems inevitable for any check to be possible without adding new rest api calls. A better solution is that we need clean status check calls.

As discussed with Nigel, I think a new healthcheck API would make sense which returns the 200 or not as per Kubenetes expectations. I envisage for each servertype it checks the status of its capabilities - initially this could be using existing status calls. If these are not adequate will could add more logic in this server based healthcheck API shim layer. For example it might want to issue a call to a downstream OMAS to see if it is working. The glossary author UI issues get all glossaries with a page size of 1 to see if the backend if active and the UI is usable.; we could do something similar in this layer if we need to.

I'll take a look at prototyping this

I've merged the first version of some prototype code into the egeria-cloudnative repository

This is very rudimentary but offers a basic API to see if a server is running.

Next steps may include

• Test this chassis in a container, with real k8s healthchecks
• Consider what resources should provide health check endpoints
• Decide on what the URLs should be
• Look at status in more detail & propose a more complete algorithm
• Decide on HTTP Status codes & messages to return

As well as some tech omissions like

• Remove hardcoded userid
• remove commented code
• consolidate gradle file
• Sort out certificates

However I thought it worth checking in as-is since

• This sets up the repo with a gradle build
• This demonstrates one way to build a customized chassis external to egeria (at least 3 different orgs wanted to do something like this)
• It provides a basic endpoint