The tsearch2 Reference

Brandon Craig Rhodes
30 June 2003

This ancient document is now of only historical interest. The prototype "tsearch2" module that it discusses is now distributed with PostgreSQL itself, and you can read about its modern form in the chapter of the PostgreSQL manual on "Full Text Search". I enjoyed working on the project; my (small) contribution was, after the full-text search operators were already working, to help document them and to suggest simpler and more uniform names for many of the built-in types and functions.

This Reference documents the user types and functions of the tsearch2 module for PostgreSQL. An introduction to the module is provided by the tsearch2 Guide, a companion document to this one. You can retrieve a beta copy of the tsearch2 module from the GiST for PostgreSQL page — look under the section entitled Development History for the current version.

Vectors and Queries

to_tsvector( [configuration,] document TEXT) RETURNS tsvector

Parses a document into tokens, reduces the tokens to lexemes, and returns a tsvector which lists the lexemes together with their positions in the document. For the best description of this process, see the section on Parsing and Stemming in the accompanying tsearch2 Guide.

strip(vector tsvector) RETURNS tsvector

Return a vector which lists the same lexemes as the given vector, but which lacks any information about where in the document each lexeme appeared. While the returned vector is thus useless for relevance ranking, it will usually be much smaller.

setweight(vector tsvector, letter) RETURNS tsvector

This function returns a copy of the input vector in which every location has been labelled with either the letter 'A', 'B', or 'C', or the default label 'D' (which is the default with which new vectors are created, and as such is usually not displayed). These labels are retained when vectors are concatenated, allowing words from different parts of a document to be weighted differently by ranking functions.

vector1 || vector2

concat(vector1 tsvector, vector2 tsvector) RETURNS tsvector

Returns a vector which combines the lexemes and position information in the two vectors given as arguments. Position weight labels (described in the previous paragraph) are retained intact during the concatenation. This has at least two uses. First, if some sections of your document need be parsed with different configurations than others, you can parse them separately and concatenate the resulting vectors into one. Second, you can weight words from some sections of you document more heavily than those from others by: parsing the sections into separate vectors; assigning the vectors different position labels with the setweight() function; concatenating them into a single vector; and then providing a weights argument to the rank() function that assigns different weights to positions with different labels.

tsvector_size(vector tsvector) RETURNS INT4

Returns the number of lexemes stored in the vector.

text::tsvector RETURNS tsvector

Directly casting text to a tsvector allows you to directly inject lexemes into a vector, with whatever positions and position weights you choose to specify. The text should be formatted like the vector would be printed by the output of a SELECT. See the Casting section in the Guide for details.

Query Operations

to_tsquery( [configuration,] querytext text) RETURNS tsvector

Parses a query, which should be single words separated by the boolean operators “&” and, “|” or, and “!” not, which can be grouped using parenthesis. Each word is reduced to a lexeme using the current or specified configuration.

querytree(query tsquery) RETURNS text

This might return a textual representation of the given query.

text::tsquery RETURNS tsquery

Directly casting text to a tsquery allows you to directly inject lexemes into a query, with whatever positions and position weight flags you choose to specify. The text should be formatted like the query would be printed by the output of a SELECT. See the Casting section in the Guide for details.

Configurations

• default — Indexes words and numbers, using the en_stem English Snowball stemmer for Latin-alphabet words and the simple dictionary for all others.
• default_russian — Indexes words and numbers, using the en_stem English Snowball stemmer for Latin-alphabet words and the ru_stem Russian Snowball dictionary for all others.
• simple — Processes both words and numbers with the simple dictionary, which neither discards any stop words nor alters them.

set_curcfg( id INT | ts_name TEXT ) RETURNS VOID

Set the current configuration used by to_tsvector and to_tsquery.

show_curcfg() RETURNS INT4

Returns the integer id of the current configuration.

Each configuration is defined by a record in the pg_ts_cfg table:

create table pg_ts_cfg (
	id		int not  null primary key,
	ts_name		text not null,
	prs_name	text not null,
	locale		text
);

create table pg_ts_cfgmap (
	ts_name		text not null,
	lex_alias	text not null,
	dict_name	text[],
	primary key (ts_name,lex_alias)
);

Parsers

create table pg_ts_parser (
	prs_id		int not null primary key,
	prs_name	text not null,
	prs_start	oid not null,
	prs_getlexem	oid not null,
	prs_end		oid not null,
	prs_headline	oid not null,
	prs_lextype	oid not null,
	prs_comment	text
);

The tsearch2 module comes with one parser named default which is suitable for parsing most plain text and HTML documents.

Each parser argument below must designate a parser with either an integer prs_id or a textual prs_name; the current parser is used when this argument is omitted.

CREATE FUNCTION set_curprs(parser) RETURNS VOID

Selects a current parser which will be used when any of the following functions are called without a parser as an argument.

CREATE FUNCTION lexem_type( [ parser ] ) RETURNS SETOF lexemtype

Returns a table which defines and describes each kind of token the parser may produce as output. For each token type the table gives the lexid which the parser will label each token of that type, the alias which names the token type, and a short description descr for the user to read.

CREATE FUNCTION parse( [ parser, ] document TEXT ) RETURNS SETOF lexemtype

Parses the given document and returns a series of records, one for each token produced by parsing. Each token includes a lexid giving its type and a lexem which gives its content.

Dictionaries

• simple simply folds uppercase letters to lowercase before returning the word.
• en_stem runs an English Snowball stemmer on each word that attempts to reduce the various forms of a verb or noun to a single recognizable form.
• ru_stem runs a Russian Snowball stemmer on each word.

CREATE TABLE pg_ts_dict (
	dict_id		int not null primary key,
	dict_name	text not null,
	dict_init	oid,
	dict_initoption	text,
	dict_lemmatize	oid not null,
	dict_comment	text
);

The argument named dictionary in each of the following functions should be either an integer dict_id or a textual dict_name identifying which dictionary should be used for the operation; if omitted then the current dictionary is used.

CREATE FUNCTION set_curdict(dictionary) RETURNS VOID

Selects a current dictionary for use by functions that do not select a dictionary explicitly.

CREATE FUNCTION lexize( [ dictionary, ] word text) RETURNS TEXT[]

Reduces a single word to a lexeme. Note that lexemes are arrays of zero or more strings, since in some languages there might be several base words from which an inflected form could arise.

Ranking

Both of these ranking functions take an integer normalization option that specifies whether a document's length should impact its rank. This is often desirable, since a hundred-word document with five instances of a search word is probably more relevant than a thousand-word document with five instances. The option can have the values:

• 0 (the default) ignores document length.
• 1 divides the rank by the logarithm of the length.
• 2 divides the rank by the length itself.

CREATE FUNCTION rank(
[ weights float4[], ] vector tsvector, query tsquery, [ normalization int4 ]
) RETURNS float4

This is the ranking function from the old version of OpenFTS, and offers the ability to weight word instances more heavily depending on how you have classified them. The weights specify how heavily to weight each category of word:

{D-weight, A-weight, B-weight, C-weight}

If no weights are provided, then these defaults are used:

{0.1, 0.2, 0.4, 1.0}

Often weights are used to mark words from special areas of the document, like the title or an initial abstract, and make them more or less important than words in the document body.

CREATE FUNCTION rank_cd(
[ K int4, ] vector tsvector, query tsquery, [ normalization int4 ]
) RETURNS float4

This function computes the cover density ranking for the given document vector and query, as described in Clarke, Cormack, and Tudhope's “Relevance Ranking for One to Three Term Queries” in the 1999 Information Processing and Management. The value K is one of the values from their formula, and defaults to K=4. The examples in their paper K=16; we can roughly describe the term as stating how far apart two search terms can fall before the formula begins penalizing them for lack of proximity.

Headlines

CREATE FUNCTION headline(
[ id int4, | ts_name text, ] document text, query tsquery, [ options text ]
) RETURNS text

Every form of the the headline() function accepts a document along with a query, and returns one or more ellipse-separated excerpts from the document in which terms from the query are highlighted. The configuration with which to parse the document can be specified by either its id or ts_name; if none is specified that the current configuration is used instead.

An options string if provided should be a comma-separated list of one or more ‘option=value’ pairs. The available options are:

• StartSel, StopSel — the strings with which query words appearing in the document should be delimited to distinguish them from other excerpted words.
• MaxWords, MinWords — limits on the shortest and longest headlines you will accept.
• ShortWord — this prevents your headline from beginning or ending with a word which has this many characters or less. The default value of 3 should eliminate most English conjunctions and articles.

Any unspecified options receive these defaults:

StartSel=<b>, StopSel=</b>, MaxWords=35, MinWords=15, ShortWord=3