Postgresql error: failed to re-find parent key in index

Question

I have a Django app with a postgresql 9.3 DB (hosted on Ubuntu VM), and very recently the VM ran out of disk space (resulting in no space left on device errors showing up in my logs).

I cleared the space on the device and everything came back online.

But now, for a minority of my requests, I've started receiving a django.db.utils:DatabaseError alert, with the description failed to re-find parent key in index "links_grouptraffic_time" for split pages 24582/24583. links_grouptraffic is one of my data models, and time is an attribute within that data model.

Can someone explain the background of what this means, and a remedy for this problem? Thanks in advance!

---

• ypercubeᵀᴹ: Seems like a corrupted index. Is there anything related in the Postgres logfile?
• ypercubeᵀᴹ: Also, which exact version do you run?

Answer 1

Your database is likely corrupted, especially the indexes. You can rebuild all indexes and check all tables on a database by:

REINDEX SYSTEM; -- run once
REINDEX DATABASE <your dbname>; -- run for each database
VACUUM (FULL VERBOSE ANALYZE); -- also run for each database after reindexing

• Hassan Baig: Fair enough. Although, shouldn't I just run reindex index links_grouptraffic_time, since that's the index the error relates to?
• Ezequiel Tolnay: That is the error that you "stumbled upon" so far. If you used your database and ran out of space, you'll likely have similar errors in any indexes of tables updated while out of space.
• Gudmundur Orn: I had the same error on postgres 9.6 and used these steps: reindex system <dbname>; reindex database <dbname>; vacuum (full) <dbname>
• Sanjay: In My case I am not able to start database. So Can I reindex?
• Ezequiel Tolnay: @Sanjay it depends on the kind and extent of damage to your database. In order to recover information from a corrupted database, you will require some basic understanding of what are the files in a postgresql system's PGDATA folder (e.g. /var/lib/postgresql/xx/main). Your database is in base. You may be able to recover your database by saving a copy of your PGDATA folder, then reinitialise your repository (i.e. initdb), start it, create a new empty database in (e.g. CREATE DATABASE recovered), stop the service, replace the contents of that new database with your old database, then start.