How should I handle database failures in a web application? [duplicate]

Question

This question already has an answer here:

How to handle exception in REST API (1 answer)

Closed 10 years ago.

I'm developing a simple RESTful API using Go's Goji framework (although this question is language-agnostic), in which parameters from the URL are queried against a PostgreSQL database. Here's how it works:

First we define a struct that holds a reference to our ORM

This struct contains an Init method that allows us to initialize a connection. This struct contains other methods to handle HTTP requests and exists primarily as a means to associate the database with the request handler.

type Context struct {
 db *gorm.DB
}
func (ctx Context) Init(user string, name string) (Context, error) {
 db, err := gorm.Open(
 "postgres",
 fmt.Sprintf("port=5432 sslmode=require host=domain.xxx"),
 )
 ctx.db = &db
 return ctx, err
}
func (ctx Context) handleGetRequest(c web.C, w http.ResponseWriter, r *http.Request) {
 // ... business logic
}

This structure is then initialized exatly once when the web application starts up

With this architecture, any database failure implies that the web app must be restarted.

func main() { 
 // "Context" is a means of including a DB object with route handlers
 ctx, err := context.Context{}.Init(dbuser, dbname)
 if err != nil {
 panic(err)
 }
 defer ctx.Close()
 // do other stuff, e.g. set up routing and whatnot ...
}

My questions are as follows:

1. From a software architecture point of view, what is a reasonable approach for gracefully handling database downtime and automatically reconnecting when the DB becomes available?
2. How can I work the recommendations in this answer into my design?

tl;dr: what's the industry standard for handing DB failures within web applications, from a software architecture perspective?

• Louis Thibault: @DanPichelman, thanks for your suggested answer. Unfortunately your link doesn't really address my question, which pertains to how I should structure my code to perform all the actions suggested in the answer you indicated (hence the design and architecture tag). I will edit my question to make this clearer!
• user50236: Looks like you are not respecting the MVC design. You need to detach the view from the controller. If the controller can't get to the model, then feed bogus data to the view until the connection restablish.
• Louis Thibault: @husker, silly question but isn't MVC a bit excessive for such a simple application? I typically associate MVC with big, hulking applications (but I'm very new to the web game...)
• user50236: @blz it might be, but in this case you need to restart the UI when a problem happens in the database. That leads to my question, is it acceptable? If yes, then you're right, you need to reconect from the view. If not then you might need to split view and controller, then let the controller deal with the reconnect, and let the view show a loading clock until the connection is back.

Answer 1

Requiring a reboot of the application when the database goes down is definitely a bad design, because it increases maintainance effort which will negatively affect your uptime.

A good design would try to reconnect to the database every few seconds so business can continue as soon as the database is available again.

In the meantime, returning a "500 Internal Server Error" is reasonable, although it would be better when those features of the application which don't strictly require the database would still work to avoid hurting the users faith in the reliability of your service (under the premise that your application has such features).

• Louis Thibault: I was hoping for concrete suggestions for how to achieve the behavior you are suggesting. Are there any common design patterns that can be applied to my case, for example?
• initzero: @blz Consider using Ping() from the database/sql package: golang.org/pkg/database/sql/#DB.Ping. For instance, you could start a goroutine in your Init() method to periodically ping the db and reform the connection when it comes back.