Mix full-text and non-text queries

This page describes how to perform a search that mixes full-text and non-text data.

Perform a mixed full-text and non-text search

Search indexes support full-text, exact match, numeric columns, and JSON/JSONB columns. You can combine text and non-text conditions in the WHERE clause similarly to multi-column search queries. The query optimizer tries to optimize non-text predicates with a search index. If that's not possible, Spanner evaluates the condition for every row that matches the search index. Referenced columns not stored in the search index are fetched from the base table.

Consider the following example:

CREATETABLEAlbums(
AlbumIdSTRING(MAX)NOTNULL,
TitleSTRING(MAX),
RatingFLOAT64,
GenresARRAY<STRING(MAX)>,
LikesINT64,
CoverBYTES(MAX),
Title_TokensTOKENLISTAS(TOKENIZE_FULLTEXT(Title))HIDDEN,
Rating_TokensTOKENLISTAS(TOKENIZE_NUMBER(Rating))HIDDEN,
Genres_TokensTOKENLISTAS(TOKEN(Genres))HIDDEN
)PRIMARYKEY(AlbumId);
CREATESEARCHINDEXAlbumsIndex
ONAlbums(Title_Tokens,Rating_Tokens,Genres_Tokens)
STORING(Likes);

CREATETABLEalbums(
albumidcharactervaryingNOTNULL,
titlecharactervarying,
ratingbigint,
genrescharactervaryingNOTNULL,
likesbigint,
coverbytea,
title_tokensspanner.tokenlistAS(spanner.tokenize_fulltext(title))VIRTUALHIDDEN,
rating_tokensspanner.tokenlistAS(spanner.tokenize_number(rating))VIRTUALHIDDEN,
genres_tokensspanner.tokenlistAS(spanner.token(genres))VIRTUALHIDDEN,
PRIMARYKEY(albumid));
CREATESEARCHINDEXalbumsindex
ONalbums(title_tokens,rating_tokens,genres_tokens)
INCLUDE(likes);

The behavior of queries on this table include the following:

• Rating and Genres are included in the search index. Spanner accelerates conditions using search index posting lists. ARRAY_INCLUDES_ANY, ARRAY_INCLUDES_ALL are GoogleSQL functions and are not supported for PostgreSQL dialect.

  SELECTAlbum
  FROMAlbums
  WHERERating > 4
  ANDARRAY_INCLUDES_ANY(Genres,['jazz'])
  

• The query can combine conjunctions, disjunctions, and negations in any way, including mixing full-text and non-text predicates. This query is fully accelerated by the search index.

  SELECTAlbum
  FROMAlbums
  WHERE(SEARCH(Title_Tokens,'car')
  ORRating > 4)
  ANDNOTARRAY_INCLUDES_ANY(Genres,['jazz'])
  

• Likes is stored in the index, but the schema doesn't request Spanner to build a token index for its possible values. Therefore, the full-text predicate on Title and non-text predicate on Rating is accelerated, but the predicate on Likes isn't. In Spanner, the query fetches all documents with the term "car" in the Title and a rating more than 4, then it filters documents that don't have at least 1000 likes. This query uses a lot of resources if almost all albums have the term "car" in their title and almost all of them have a rating of 5, but few albums have 1000 likes. In such cases, indexing Likes similarly to Rating saves resources.

  GoogleSQL

  SELECTAlbum
  FROMAlbums
  WHERESEARCH(Title_Tokens,'car')
  ANDRating > 4
  ANDLikes>=1000
  

  PostgreSQL

  SELECTalbum
  FROMalbums
  WHEREspanner.search(title_tokens,'car')
  ANDrating > 4
  ANDlikes>=1000
  

• Cover isn't stored in the index. The following query does a back join between AlbumsIndex and Albums to fetch Cover for all matching albums.

  GoogleSQL

  SELECTAlbumId,Cover
  FROMAlbums
  WHERESEARCH(Title_Tokens,'car')
  ANDRating > 4
  

  PostgreSQL

  SELECTalbumid,cover
  FROMalbums
  WHEREspanner.search(title_tokens,'car')
  ANDrating > 4
  

What's next

• Learn about full-text search queries.
• Learn how to rank search results.
• Learn how to perform a substring search.
• Learn how to paginate search results.
• Learn how to search multiple columns.