How to increase index row maximum size in Postgresql?

Question

I want to perform KNN on 512-dimensional vector using the solution suggested here.

1. I couldn't create table with 512-d cube array. Solved by changing the CUBE_MAX_DIM in the source code.
2. Couldn't create index on table with dimensions> 200. Set block_size=32. Now works with dimensionality<=510.
3. Got new error after changing block_size and trying to create index on table with 512-d vectors ERROR: index row requires 8208 bytes, maximum size is 8191

Is it possible to increase this limit?

Answer 1

That would require significant changes, and I doubt it can be done easily.

See these excerpts from src/include/access/itup.h:

/*
 * Index tuple header structure
 *
 * All index tuples start with IndexTupleData. If the HasNulls bit is set,
 * this is followed by an IndexAttributeBitMapData. The index attribute
 * values follow, beginning at a MAXALIGN boundary.
 *
 * Note that the space allocated for the bitmap does not vary with the number
 * of attributes; that is because we don't have room to store the number of
 * attributes in the header. Given the MAXALIGN constraint there's no space
 * savings to be had anyway, for usual values of INDEX_MAX_KEYS.
 */
typedef struct IndexTupleData
{
 ItemPointerData t_tid; /* reference TID to heap tuple */
 /* ---------------
 * t_info is laid out in the following fashion:
 *
 * 15th (high) bit: has nulls
 * 14th bit: has var-width attributes
 * 13th bit: AM-defined meaning
 * 12-0 bit: size of tuple
 * ---------------
 */
 unsigned short t_info; /* various info about tuple */
} IndexTupleData; /* MORE DATA FOLLOWS AT END OF STRUCT */
[...]
/*
 * t_info manipulation macros
 */
#define INDEX_SIZE_MASK 0x1FFF
#define INDEX_AM_RESERVED_BIT 0x2000 /* reserved for index-AM specific
 * usage */
#define INDEX_VAR_MASK 0x4000
#define INDEX_NULL_MASK 0x8000

The limit you are hitting is INDEX_SIZE_MASK, and to increase it, you'd have to change the tuple header so that t_info has more than two bytes.

Perhaps it is as simple as that, but it might have repercussions in other parts of the code.

• Alibi Yeslambek: I am using Postgres v10.5. 13th bit of t_info is unused in this version. I just changed INDEX_SIZE_MASK to 0x3FFF so 13th bit can be used for the size and it worked. Thank you!
• Laurenz Albe: Fine, but that way you are barring your way to a PostgreSQL upgrade. Are you sure?