|
Main
Community
Commercial services
Misc
 Subscribe in a reader
|
Documentation
Sphinx 0.9.7 reference manual
0.9.7-release, 02-Apr-2007
Sphinx 0.9.8-rc1 reference manual
Sphinx is a full-text search engine, distributed under GPL version 2.
Commercial licensing (eg. for embedded use) is also available upon request.
Generally, it's a standalone search engine, meant to provide fast,
size-efficient and relevant full-text search functions to other
applications. Sphinx was specially designed to integrate well with
SQL databases and scripting languages.
Currently built-in data source drivers support fetching data either via
direct connection to MySQL, or PostgreSQL, or from a pipe in a custom XML
format. Adding new drivers (eg. to natively support some other DBMSes)
is designed to be as easy as possible.
Search API is natively ported to PHP, Python, Perl, Ruby, Java, and
also available as a pluggable MySQL storage engine. API is very
lightweight so porting it to new language is known to take a few hours.
As for the name, Sphinx is an acronym which is officially decoded
as SQL Phrase Index. Yes, I know about CMU's Sphinx project.
- high indexing speed (upto 10 MB/sec on modern CPUs);
- high search speed (avg query is under 0.1 sec on 2-4 GB text collections);
- high scalability (upto 100 GB of text, upto 100 M documents on a single CPU);
- provides good relevance ranking through combination of phrase proximity ranking and statistical (BM25) ranking;
- provides distributed searching capabilities;
- provides document exceprts generation;
- provides searching from within MySQL through pluggable storage engine;
- supports boolean, phrase, and word proximity queries;
- supports multiple full-text fields per document (upto 32 by default);
- supports multiple additional attributes per document (ie. groups, timestamps, etc);
- supports stopwords;
- supports both single-byte encodings and UTF-8;
- supports English stemming, Russian stemming, and Soundex for morphology;
- supports MySQL natively (MyISAM and InnoDB tables are both supported);
- supports PostgreSQL natively.
Sphinx is available through its official Web site at http://www.sphinxsearch.com/.
Currently, Sphinx distribution tarball includes the following software:
indexer: an utility which creates fulltext indexes;search: a simple command-line (CLI) test utility which searches through fulltext indexes;searchd: a daemon which enables external software (eg. Web applications) to search through fulltext indexes;sphinxapi: a set of searchd client API libraries for popular Web scripting languages (PHP, Python, Perl, Ruby).
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License,
or (at your option) any later version. See COPYING file for details.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software Foundation, Inc.,
59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
If you don't want to be bound by GNU GPL terms (for instance,
if you would like to embed Sphinx in your software, but would not
like to disclose its source code), please contact
the author to obtain
a commercial license.
1.5. Author and contributorsAuthor
Sphinx initial author and current primary developer is:
ContributorsPeople who contributed to Sphinx and their contributions (in no particular order) are:
- Robert "coredev" Bengtsson (Sweden), initial version of PostgreSQL data source;
- Len Kranendonk, Perl API
- Dmytro Shteflyuk, Ruby API
Many other people have contributed ideas, bug reports, fixes, etc.
Thank you!
Sphinx development was started back in 2001, because I didn't manage
to find an acceptable search solution (for a database driven Web site)
which would meet my requirements. Actually, each and every important aspect was a problem:
- search quality (ie. good relevance)
- statistical ranking methods performed rather bad, especially on large collections of small documents (forums, blogs, etc)
- search speed
- especially if searching for phrases which contain stopwords, as in "to be or not to be"
- moderate disk and CPU requirements when indexing
- important in shared hosting enivronment, not to mention the indexing speed.
Despite the amount of time passed and numerous improvements made in the
other solutions, there's still no solution which I personally would
be eager to migrate to.
Considering that and a lot of positive feedback received from Sphinx users
during last years, the obvious decision is to continue developing Sphinx
(and, eventually, to take over the world).
Most modern UNIX systems with a C++ compiler should be able
to compile and run Sphinx without any modifications.
Currently known systems Sphinx has been successfully running on are:
- Linux 2.4.x, 2.6.x (various distributions)
- Windows 2000, XP
- FreeBSD 4.x, 5.x, 6.x
- NetBSD 1.6, 3.0
- Solaris 9, 11
- Mac OS X
CPU architectures known to work include X86, X86-64, SPARC64.
I hope Sphinx will work on other Unix platforms as well.
If the platform you run Sphinx on is not in this list,
please do report it.
At the moment, Windows version of Sphinx is not intended to be used
in production, but rather for testing and debugging only. Two most prominent
issues are missing concurrent queries support (client queries are stacked
on TCP connection level instead), and missing index data rotation support.
There are succesful production installations which workaround these issues.
However, running high-volume search service under Windows
is still not recommended.
On UNIX, you will need the following tools to build
and install Sphinx:
- a working C++ compiler. GNU gcc is known to work.
- a good make program. GNU make is known to work.
On Windows, you will need Microsoft Visual C/C++ Studio .NET 2003 or 2005.
Other compilers/environments will probably work as well, but for the
time being, you will have to build makefile (or other environment
specific project files) manually.
Extract everything from the distribution tarball (haven't you already?)
and go to the sphinx subdirectory:
$ tar xzvf sphinx-0.9.7.tar.gz
$ cd sphinx
Run the configuration program:
There's a number of options to configure. The complete listing may
be obtained by using --help switch. The most important ones are:
--prefix, which specifies where to install Sphinx;--with-mysql, which specifies where to look for MySQL
include and library files, if auto-detection fails;--with-pgsql, which specifies where to look for PostgreSQL
include and library files.
Build the binaries:
Install the binaries in the directory of your choice:
2.4. Known installation issues
If configure fails to locate MySQL headers and/or libraries,
try checking for and installing mysql-devel package. On some systems,
it is not installed by default.
If make fails with a message which look like
/bin/sh: g++: command not found
make[1]: *** [libsphinx_a-sphinx.o] Error 127
try checking for and installing gcc-c++ package.
If you are getting compile-time errors which look like
sphinx.cpp:67: error: invalid application of `sizeof' to
incomplete type `Private::SizeError<false>'
this means that some compile-time type size check failed.
The most probable reason is that off_t type is less than 64-bit
on your system. As a quick hack, you can edit sphinx.h and replace off_t
with DWORD in a typedef for SphOffset_t, but note that this will prohibit
you from using full-text indexes larger than 2 GB. Even if the hack helps,
please report such issues, providing the exact error message and
compiler/OS details, so I could properly fix them in next releases.
If you keep getting any other error, or the suggestions above
do not seem to help you, please don't hesitate to contact me.
2.5. Quick Sphinx usage tour
All the example commands below assume that you installed Sphinx
in /usr/local/sphinx.
To use Sphinx, you will need to:
Create a configuration file.
Default configuration file name is sphinx.conf.
All Sphinx programs look for this file in current working directory
by default.
Sample configuration file, sphinx.conf.dist, which has
all the options documented, is created by configure.
Copy and edit that sample file to make your own configuration:
$ cd /usr/local/sphinx/etc
$ cp sphinx.conf.dist sphinx.conf
$ vi sphinx.conf
Sample configuration file is setup to index documents
table from MySQL database test; so there's example.sql
sample data file to populate that table with a few documents for testing purposes:
$ mysql -u test < /usr/local/sphinx/etc/example.sql
Run the indexer to create full-text index from your data: $ cd /usr/local/sphinx/etc
$ /usr/local/sphinx/bin/indexer
Query your newly created index!
To query the index from command line, use search utility:
$ cd /usr/local/sphinx/etc
$ /usr/local/sphinx/bin/search test
To query the index from your PHP scripts, you need to:
Run the search daemon which your script will talk to: $ cd /usr/local/sphinx/etc
$ /usr/local/sphinx/bin/searchd
Run the attached PHP API test script (to ensure that the daemon
was succesfully started and is ready to serve the queries):
$ cd sphinx/api
$ php test.php test
Include the API (it's located in api/sphinxapi.php)
into your own scripts and use it.
Happy searching!
The data to be indexed can generally come from very different
sources: SQL databases, plain text files, HTML files, mailboxes,
and so on. From Sphinx point of view, the data it indexes is a
set of structured documents, each of which has the
same set of fields. This is biased towards SQL, where
each row correspond to a document, and each column to a field.
Depending on what source Sphinx should get the data from,
different code is required to fetch the data and prepare it for indexing.
This code is called data source driver (or simply
driver or data source for brevity).
At the time of this writing, there are drivers for MySQL and
PostgreSQL databases, which can connect to the database using
its native C/C++ API, run queries and fetch the data. There's
also a driver called xmlpipe, which runs a specified command
and reads the data from its stdout.
See Section 3.8, “xmlpipe data source” section for the format description.
There can be as many sources per index as necessary. They will be
sequentially processed in the very same order which was specifed in
index definition. All the documents coming from those sources
will be merged as if they were coming from a single source.
Attributes are additional values associated with each document
that can be used to perform additional filtering and sorting during search.
It is often desired to additionally process full-text search results
based not only on matching document ID and its rank, but on a number
of other per-document values as well. For instance, one might need to
sort news search results by date and then relevance,
or search through products within specified price range,
or limit blog search to posts made by selected users,
or group results by month. To do that efficiently, Sphinx allows
to attach a number of additional attributes
to each document, and store their values in the full-text index.
It's then possible to use stored values to filter, sort,
or group full-text matches.
A good example would be a forum posts table. Assume that
only title and contentfields need to be full-text searchable -
but that sometimes it is also required to limit search to a certain
author or a sub-forum (ie. search only those rows that have some
specific values of author_id or forum_id columns in the SQL table);
or to sort matches by post_date column; or to group matching posts
by month of the post_date and calculate per-group match counts.
This can be achieved by specifying all the mentioned columns
(excluding title and content, that are full-text fields) as
attributes, indexing them, and then using API calls to
setup filtering, sorting, and grouping. Here as an example.
Example sphinx.conf part:
...
sql_query = SELECT id, title, content, \
author_id, forum_id, post_date FROM my_forum_posts
sql_attr_uint = author_id
sql_attr_uint = forum_id
sql_attr_timestamp = post_date
...
Example application code (in PHP):
// only search posts by author whose ID is 123
$cl->SetFilter ( "author_id", array ( 123 ) );
// only search posts in sub-forums 1, 3 and 7
$cl->SetFilter ( "forum_id", array ( 1,3,7 ) );
// sort found posts by posting date in descending order
$cl->SetSortMode ( SPH_SORT_ATTR_DESC, "post_date" );
Attributes are named. Attribute names are case insensitive.
Attributes are not full-text indexed; they are stored in the index as is.
Currently supported attribute types are:
- unsigned integers (1-bit to 32-bit wide);
- UNIX timestamps;
- floating point values (32-bit, IEEE 754 single precision);
- string ordinals (specially computed integers);
- MVA, multi-value attributes (variable-length lists of 32-bit unsigned integers).
The complete set of per-document attribute values is sometimes
referred to as docinfo. Docinfos can either be
- stored separately from the main full-text index data ("extern" storage, in
.spa file), or - attached to each occurence of document ID in full-text index data ("inline" storage, in
.spd file).
When using extern storage, a copy of .spa file
(with all the attribute values for all the documents) is kept in RAM by
searchd at all times. This is for performance reasons;
random disk I/O would be too slow. On the contrary, inline storage does not
require any additional RAM at all, but that comes at the cost of greatly
inflating the index size: remember that it copies all
attribute value every time when the document ID
is mentioned, and that is exactly as many times as there are
different keywords in the document. Inline may be the only viable
option if you have only a few attributes and need to work with big
datasets in limited RAM. However, in most cases extern storage
makes both indexing and searching much more efficient.
Search-time memory requirements for extern storage are
(1+number_of_attrs)*number_of_docs*4 bytes, ie. 10 million docs with
2 groups and 1 timestamp will take (1+2+1)*10M*4 = 160 MB of RAM.
This is PER DAEMON, not per query. searchd
will allocate 160 MB on startup, read the data and keep it shared between queries.
The children will NOT allocate any additional
copies of this data.
3.3. MVA (multi-valued attributes)
MVAs, or multi-valued attributes, are an important special type of per-document attributes in Sphinx.
MVAs make it possible to attach lists of values to every document.
They are useful for article tags, product categories, etc.
Filtering and group-by (but not sorting) on MVA attributes is supported.
Currently, MVA list entries are limited to unsigned 32-bit integers.
The list length is not limited, you can have an arbitrary number of values
attached to each document as long as RAM permits (.spm file
that contains the MVA values will be precached in RAM by searchd).
The source data can be taken either from a separate query, or from a document field;
see source type in sql_attr_multi.
In the first case the query will have to return pairs of document ID and MVA values,
in the second one the field will be parsed for integer values.
There are absolutely no requirements as to incoming data order; the values will be
automatically grouped by document ID (and internally sorted within the same ID)
during indexing anyway.
When filtering, a document will match the filter on MVA attribute
if any of the values satisfy the filtering condition.
(Therefore, documents that pass through exclude filters will not
contain any of the forbidden values.)
When grouping by MVA attribute, a document will contribute to as
many groups as there are different MVA values associated with that document.
For instance, if the collection contains exactly 1 document having a 'tag' MVA
with values 5, 7, and 11, grouping on 'tag' will produce 3 groups with
'@count' equal to 1 and '@groupby' key values of 5, 7, and 11 respectively.
Also note that grouping by MVA might lead to duplicate documents in the result set:
because each document can participate in many groups, it can be chosen as the best
one in in more than one group, leading to duplicate IDs. PHP API historically
uses ordered hash on the document ID for the resulting rows; so you'll also need to use
SetArrayResult() in order
to employ group-by on MVA with PHP API.
To be able to answer full-text search queries fast, Sphinx needs
to build a special data structure optimized for such queries from
your text data. This structure is called index; and
the process of building index from text is called indexing.
Different index types are well suited for different tasks.
For example, a disk-based tree-based index would be easy to
update (ie. insert new documents to existing index), but rather
slow to search. Therefore, Sphinx architecture allows for different
index types to be implemented easily.
The only index type which is implemented in Sphinx at the moment is
designed for maximum indexing and searching speed. This comes at a cost
of updates being really slow; theoretically, it might be slower to
update this type of index than than to reindex it from scratch.
However, this very frequently could be worked around with
muiltiple indexes, see Section 3.10, “Live index updates” for details.
It is planned to implement more index types, including the
type which would be updateable in real time.
There can be as many indexes per configuration file as necessary.
indexer utility can reindex either all of them
(if --all option is specified), or a certain explicitly
specified subset. searchd utility will serve all
the specified indexes, and the clients can specify what indexes to
search in run time.
3.5. Restrictions on the source data
There are a few different restrictions imposed on the source data
which is going to be indexed by Sphinx, of which the single most
important one is:
ALL DOCUMENT IDS MUST BE UNIQUE UNSIGNED NON-ZERO INTEGER NUMBERS (32-BIT OR 64-BIT, DEPENDING ON BUILD TIME SETTINGS).
If this requirement is not met, different bad things can happen.
For instance, Sphinx can crash with an internal assertion while indexing;
or produce strange results when searching due to conflicting IDs.
Also, a 1000-pound gorilla might eventually come out of your
display and start throwing barrels at you. You've been warned.
3.6. Charsets, case folding, and translation tables
When indexing some index, Sphinx fetches documents from
the specified sources, splits the text into words, and does
case folding so that "Abc", "ABC" and "abc" would be treated
as the same word (or, to be pedantic, term).
To do that properly, Sphinx needs to know
- what encoding is the source text in;
- what characters are letters and what are not;
- what letters should be folded to what letters.
This should be configured on a per-index basis using
charset_type and
charset_table options.
charset_type
specifies whether the document encoding is single-byte (SBCS) or UTF-8.
charset_table
specifies the table that maps letter characters to their case
folded versions. The characters that are not in the table are considered
to be non-letters and will be treated as word separators when indexing
or searching through this index.
Note that while default tables do not include space character
(ASCII code 0x20, Unicode U+0020) as a letter, it's in fact
perfectly legal to do so. This can be
useful, for instance, for indexing tag clouds, so that space-separated
word sets would index as a single search query term.
Default tables currently include English and Russian characters.
Please do submit your tables for other languages!
3.7. SQL data sources (MySQL, PostgreSQL)
With all the SQL drivers, indexing generally works as follows.
- connection to the database is established;
- pre-query (see Section 8.1.9, “sql_query_pre”) is executed
to perform any necessary initial setup, such as setting per-connection encoding with MySQL;
- main query (see Section 8.1.10, “sql_query”) is executed and the rows it returns are indexed;
- post-query (see Section 8.1.19, “sql_query_post”) is executed
to perform any necessary cleanup;
- connection to the database is closed;
- indexer does the sorting phase (to be pedantic, index-type specific post-processing);
- connection to the database is established again;
- post-index query (see Section 8.1.20, “sql_query_post_index”) is executed
to perform any necessary final cleanup;
- connection to the database is closed again.
Most options, such as database user/host/password, are straightforward.
However, there are a few subtle things, which are discussed in more detail here.
Ranged queries
Main query, which needs to fetch all the documents, can impose
a read lock on the whole table and stall the concurrent queries
(eg. INSERTs to MyISAM table), waste a lot of memory for result set, etc.
To avoid this, Sphinx supports so-called ranged queries.
With ranged queries, Sphinx first fetches min and max document IDs from
the table, and then substitutes different ID intervals into main query text
and runs the modified query to fetch another chunk of documents.
Here's an example.
Example 1. Ranged query usage example
# in sphinx.conf
sql_query_range = SELECT MIN(id),MAX(id) FROM documents
sql_range_step = 1000
sql_query = SELECT * FROM documents WHERE id>=$start AND id<=$end
If the table contains document IDs from 1 to, say, 2345, then sql_query would
be run three times:
- with
$start replaced with 1 and $end replaced with 1000; - with
$start replaced with 1001 and $end replaced with 2000; - with
$start replaced with 2000 and $end replaced with 2345.
Obviously, that's not much of a difference for 2000-row table,
but when it comes to indexing 10-million-row MyISAM table,
ranged queries might be of some help.
sql_post vs. sql_post_index
The difference between post-query and post-index query is in that post-query
is run immediately when Sphinx received all the documents, but further indexing
may still fail for some other reason. On the contrary,
by the time the post-index query gets executed, it is guaranteed
that the indexing was succesful. Database connection is dropped and re-established
because sorting phase can be very lengthy and would just timeout otherwise.
xmlpipe data source was designed to enable users to plug data into
Sphinx without having to implement new data sources drivers themselves.
It is limited to 2 fixed fields and 2 fixed attributes, and is deprecated
in favor of Section 3.9, “xmlpipe2 data source” now. For new streams, use xmlpipe2.
To use xmlpipe, configure the data source in your configuration file
as follows:
source example_xmlpipe_source
{
type = xmlpipe
xmlpipe_command = perl /www/mysite.com/bin/sphinxpipe.pl
}
The indexer will run the command specified
in xmlpipe_command,
and then read, parse and index the data it prints to stdout.
More formally, it opens a pipe to given command and then reads
from that pipe.
indexer will expect one or more documents in custom XML format.
Here's the example document stream, consisting of two documents:
Example 2. XMLpipe document stream
<document>
<id>123</id>
<group>45</group>
<timestamp>1132223498</timestamp>
<title>test title</title>
<body>
this is my document body
</body>
</document>
<document>
<id>124</id>
<group>46</group>
<timestamp>1132223498</timestamp>
<title>another test</title>
<body>
this is another document
</body>
</document>
Legacy xmlpipe legacy driver uses a builtin parser
which is pretty fast but really strict and does not actually
fully support XML. It requires that all the fields must
be present, formatted exactly as in this example, and
occur exactly in the same order. The only optional
field is timestamp; it defaults to 1.
3.9. xmlpipe2 data source
xmlpipe2 lets you pass arbitrary full-text and attribute data to Sphinx
in yet another custom XML format. It also allows to specify the schema
(ie. the set of fields and attributes) either in the XML stream itself,
or in the source settings.
When indexing xmlpipe2 source, indexer runs the given command, opens
a pipe to its stdout, and expects well-formed XML stream. Here's sample
stream data:
Example 3. xmlpipe2 document stream
<?xml version="1.0" encoding="utf-8"?>
<sphinx:docset>
<sphinx:schema>
<sphinx:field name="subject"/>
<sphinx:field name="content"/>
<sphinx:attr name="published" type="timestamp"/>
<sphinx:attr name="author_id" type="int" bits="16" default="1"/>
</sphinx:schema>
<sphinx:document id="1234">
<content>this is the main content <![CDATA[[and this <cdata> entry must be handled properly by xml parser lib]]></content>
<published>1012325463</published>
<subject>note how field/attr tags can be in <b class="red">randomized</b> order</subject>
<misc>some undeclared element</misc>
</sphinx:document>
<!-- ... more documents here ... -->
</sphinx:docset>
Arbitrary fields and attributes are allowed.
They also can occur in the stream in arbitrary order within each document; the order is ignored.
There is a restriction on maximum field length; fields longer than 2 MB will be truncated to 2 MB (this limit can be changed in the source).
The schema, ie. complete fields and attributes list, must be declared
before any document could be parsed. This can be done either in the
configuration file using xmlpipe_field and xmlpipe_attr_XXX
settings, or right in the stream using <sphinx:schema> element.
<sphinx:schema> is optional. It is only allowed to occur as the very
first sub-element in <sphinx:docset>. If there is no in-stream
schema definition, settings from the configuration file will be used.
Otherwise, stream settings take precedence.
Unknown tags (which were not declared neither as fields nor as attributes)
will be ignored with a warning. In the example above, <misc> will be ignored.
All embedded tags and their attributes (such as <b> in <subject>
in the example above) will be silently ignored.
Support for incoming stream encodings depends on whether iconv
is installed on the system. xmlpipe2 is parsed using libexpat
parser that understands US-ASCII, ISO-8859-1, UTF-8 and a few UTF-16 variants
natively. Sphinx configure script will also check
for libiconv presence, and utilize it to handle
other encodings. libexpat also enforces the
requirement to use UTF-8 charset on Sphinx side, because the
parsed data it returns is always in UTF-8.
XML elements (tags) recognized by xmlpipe2 (and their attributes where applicable) are:
- sphinx:docset
- Mandatory top-level element, denotes and contains xmlpipe2 document set.
- sphinx:schema
- Optional element, must either occur as the very first child
of sphinx:docset, or never occur at all. Declares the document schema.
Contains field and attribute declarations. If present, overrides
per-source settings from the configuration file.
- sphinx:field
- Optional element, child of sphinx:schema. Declares a full-text field.
The only recognized attribute is "name", it specifies the element name
that should be treated as a full-text field in the subsequent documents.
- sphinx:attr
- Optional element, child of sphinx:schema. Declares an attribute.
Known attributes are:
- "name", specifies the element name that should be treated as an attribute in the subsequent documents.
- "type", specifies the attribute type. Possible values are "int", "timestamp", "str2ordinal", "bool", and "float".
- "bits", specifies the bit size for "int" attribute type. Valid values are 1 to 32.
- "default", specifies the default value for this attribute that should be used if the attribute's element is not present in the document.
- sphinx:document
- Mandatory element, must be a child of sphinx:docset.
Contains arbitrary other elements with field and attribute values
to be indexed, as declared either using sphinx:field and sphinx:attr
elements or in the configuration file. The only known attribute
is "id" that must contain the unique integer document ID.
There's a frequent situation when the total dataset is too big
to be reindexed from scratch often, but the amount of new records
is rather small. Example: a forum with a 1,000,000 archived posts,
but only 1,000 new posts per day.
In this case, "live" (almost real time) index updates could be
implemented using so called "main+delta" scheme.
The idea is to set up two sources and two indexes, with one
"main" index for the data which only changes rarely (if ever),
and one "delta" for the new documents. In the example above,
1,000,000 archived posts would go to the main index, and newly
inserted 1,000 posts/day would go to the delta index. Delta index
could then be reindexed very frequently, and the documents can
be made available to search in a matter of minutes.
Specifying which documents should go to what index and
reindexing main index could also be made fully automatical.
One option would be to make a counter table which would track
the ID which would split the documents, and update it
whenever the main index is reindexed.
Example 4. Fully automated live updates
# in MySQL
CREATE TABLE sph_counter
(
counter_id INTEGER PRIMARY KEY NOT NULL,
max_doc_id INTEGER NOT NULL
);
# in sphinx.conf
source main
{
# ...
sql_query_pre = REPLACE INTO sph_counter SELECT 1, MAX(id) FROM documents
sql_query = SELECT id, title, body FROM documents \
WHERE id<=( SELECT max_doc_id FROM sph_counter WHERE counter_id=1 )
}
source delta : main
{
sql_query_pre =
sql_query = SELECT id, title, body FROM documents \
WHERE id>( SELECT max_doc_id FROM sph_counter WHERE counter_id=1 )
}
index main
{
source = main
path = /path/to/main
# ... all the other settings
}
# note how all other settings are copied from main,
# but source and path are overridden (they MUST be)
index delta : main
{
source = delta
path = /path/to/delta
}
Merging two existing indexes can be more efficient that indexing the data
from scratch, and desired in some cases (such as merging 'main' and 'delta'
indexes instead of simply reindexing 'main' in 'main+delta' partitioning
scheme). So indexer has an option to do that.
Merging the indexes is normally faster than reindexing but still
not instant on huge indexes. Basically,
it will need to read the contents of both indexes once and write
the result once. Merging 100 GB and 1 GB index, for example,
will result in 202 GB of IO (but that's still likely less than
the indexing from scratch requires).
The basic command syntax is as follows:
indexer --merge DSTINDEX SRCINDEX [--rotate]
Only the DSTINDEX index will be affected: the contents of SRCINDEX will be merged into it.
--rotate switch will be required if DSTINDEX is already being served by searchd.
The initially devised usage pattern is to merge a smaller update from SRCINDEX into DSTINDEX.
Thus, when merging the attributes, values from SRCINDEX will win if duplicate document IDs are encountered.
Note, however, that the "old" keywords will not be automatically removed in such cases.
For example, if there's a keyword "old" associated with document 123 in DSTINDEX, and a keyword "new" associated
with it in SRCINDEX, document 123 will be found by both keywords after the merge.
You can supply an explicit condition to remove documents from DSTINDEX to mitigate that;
the relevant switch is --merge-dst-range:
indexer --merge main delta --merge-dst-range deleted 0 0
This switch lets you apply filters to the destination index along with merging.
There can be several filters; all of their conditions must be met in order
to include the document in the resulting mergid index. In the example above,
the filter passes only those records where 'deleted' is 0, eliminating all
records that were flagged as deleted (for instance, using
UpdateAttributes() call).
There are the following matching modes available:
- SPH_MATCH_ALL, matches all query words (default mode);
- SPH_MATCH_ANY, matches any of the query words;
- SPH_MATCH_PHRASE, matches query as a phrase, requiring perfect match;
- SPH_MATCH_BOOLEAN, matches query as a boolean expression (see Section 4.2, “Boolean query syntax”);
- SPH_MATCH_EXTENDED, matches query as an expression in Sphinx internal query language (see Section 4.3, “Extended query syntax”).
There also is a special "full scan" mode that will be automatically activated when the following conditions are met:
- The query string is empty (ie. its length is zero).
- docinfo storage is set to
extern.
In full scan mode, all the indexed documents will be considered as matching.
Such queries will still apply filters, sorting, or group by, but will not perform any real full-text searching.
This can be useful to unify full-text and non-full-text searching code, or to offload SQL server (there are cases when Sphinx scans will perform better than analogous MySQL queries).
4.2. Boolean query syntax
Boolean queries allow the following special operators to be used:
Here's an example query which uses all these operators:
Example 5. Boolean query example
( cat -dog ) | ( cat -mouse)
There always is implicit AND operator, so "hello world" query actually
means "hello & world".
OR operator precedence is higher than AND, so "looking for cat | dog | mouse"
means "looking for ( cat | dog | mouse )" and not
"(looking for cat) | dog | mouse".
Queries like "-dog", which implicitly include all documents from the
collection, can not be evaluated. This is both for technical and performance
reasons. Technically, Sphinx does not always keep a list of all IDs.
Performance-wise, when the collection is huge (ie. 10-100M documents),
evaluating such queries could take very long.
4.3. Extended query syntax
The following special operators can be used when using the extended matching mode:
Here's an example query which uses most of these operators:
Example 6. Extended query example
"hello world" @title "example program"~5 @body python -(php|perl)
There always is implicit AND operator, so "hello world" means that
both "hello" and "world" must be present in matching document.
OR operator precedence is higher than AND, so "looking for cat | dog | mouse"
means "looking for ( cat | dog | mouse )" and not
"(looking for cat) | dog | mouse".
Proximity distance is specified in words, adjusted for word count, and
applies to all words within quotes. For instance, "cat dog mouse"~5 query
means that there must be less than 8-word span which contains all 3 words,
ie. "CAT aaa bbb ccc DOG eee fff MOUSE" document will not
match this query, because this span is exactly 8 words long.
Quorum matching operator introduces a kind of fuzzy matching.
It will only match those documents that pass a given threshold of given words.
The example above ("the world is a wonderful place"/3) will match all documents
that have at least 3 of the 6 specified words.
Nested brackets, as in queries like
aaa | ( bbb ccc | ( ddd eee ) )
are not allowed yet, but this will be fixed.
Negation (ie. operator NOT) is only allowed on top level and not within
brackets (ie. groups). This isn't going to change, because supporting nested
negations would make phrase ranking implementation way too complicated.
Specific weighting function (currently) depends on the search mode.
There are these major parts which are used in the weighting functions:
- phrase rank,
- statistical rank.
Phrase rank is based on a length of longest common subsequence
(LCS) of search words between document body and query phrase. So if
there's a perfect phrase match in some document then its phrase rank
would be the highest possible, and equal to query words count.
Statistical rank is based on classic BM25 function which only takes
word frequencies into account. If the word is rare in the whole database
(ie. low frequency over document collection) or mentioned a lot in specific
document (ie. high frequency over matching document), it receives more weight.
Final BM25 weight is a floating point number between 0 and 1.
In all modes, per-field weighted phrase ranks are computed as
a product of LCS multiplied by per-field weight speficifed by user.
Per-field weights are integer, default to 1, and can not be set
lower than 1.
In SPH_MATCH_BOOLEAN mode, no weighting is performed at all, every match weight
is set to 1.
In SPH_MATCH_ALL and SPH_MATCH_PHRASE modes, final weight is a sum of weighted phrase ranks.
In SPH_MATCH_ANY mode, the idea is essentially the same, but it also
adds a count of matching words in each field. Before that, weighted
phrase ranks are additionally mutliplied by a value big enough to
guarantee that higher phrase rank in any field will make the
match ranked higher, even if it's field weight is low.
In SPH_MATCH_EXTENDED mode, final weight is a sum of weighted phrase
ranks and BM25 weight, multiplied by 1000 and rounded to integer.
This is going to be changed, so that MATCH_ALL and MATCH_ANY modes
use BM25 weights as well. This would improve search results in those
match spans where phrase ranks are equal; this is especially useful
for 1-word queries.
The key idea (in all modes, besides boolean) is that better subphrase
matches are ranked higher, and perfect matches are pulled to the top. Author's
experience is that this phrase proximity based ranking provides noticeably
better search quality than any statistical scheme alone (such as BM25,
which is commonly used in other search engines).
There are the following result sorting modes available:
- SPH_SORT_RELEVANCE mode, that sorts by relevance in descending order (best matches first);
- SPH_SORT_ATTR_DESC mode, that sorts by an attribute in descending order (bigger attribute values first);
- SPH_SORT_ATTR_ASC mode, that sorts by an attribute in ascending order (smaller attribute values first);
- SPH_SORT_TIME_SEGMENTS mode, that sorts by time segments (last hour/day/week/month) in descending order, and then by relevance in descending order;
- SPH_SORT_EXTENDED mode, that sorts by SQL-like combination of columns in ASC/DESC order;
- SPH_SORT_EXPR mode, that sorts by an arithmetic expression.
SPH_SORT_RELEVANCE ignores any additional parameters and always sorts matches
by relevance rank. All other modes require an additional sorting clause, with the
syntax depending on specific mode. SPH_SORT_ATTR_ASC, SPH_SORT_ATTR_DESC and
SPH_SORT_TIME_SEGMENTS modes require simply an attribute name.
SPH_SORT_RELEVANCE is equivalent to sorting by "@weight DESC, @id ASC" in extended mode,
SPH_SORT_ATTR_ASC is equivalent to "attribute ASC, @weight DESC, @id ASC",
and SPH_SORT_ATTR_DESC to "attribute DESC, @weight DESC, @id ASC" respectively.
SPH_SORT_TIME_SEGMENTS mode
In SPH_SORT_TIME_SEGMENTS mode, attribute values are split into so-called
time segments, and then sorted by time segment first, and by relevance second.
The segments are calculated according to the current timestamp
at the time when the search is performed, so the results would change over time.
The segments are as follows:
- last hour,
- last day,
- last week,
- last month,
- last 3 months,
- everything else.
These segments are hardcoded, but it is trivial to change them if necessary.
This mode was added to support searching through blogs, news headlines, etc.
When using time segments, recent records would be ranked higher because of segment,
but withing the same segment, more relevant records would be ranked higher -
unlike sorting by just the timestamp attribute, which would not take relevance
into account at all.
SPH_SORT_EXTENDED mode
In SPH_SORT_EXTENDED mode, you can specify an SQL-like sort expression
with up to 5 attributes (including internal attributes), eg:
@relevance DESC, price ASC, @id DESC
Both internal attributes (that are computed by the engine on the fly)
and user attributes that were configured for this index are allowed.
Internal attribute names must start with magic @-symbol; user attribute
names can be used as is. In the example above, @relevance
and @id are internal attributes and price is user-specified.
Known internal attributes are:
- @id (match ID)
- @weight (match weight)
- @rank (match weight)
- @relevance (match weight)
@rank and @relevance are just additional
aliases to @weight.
SPH_SORT_EXPR mode
Expression sorting mode lets you sort the matches by an arbitrary arithmetic
expression, involving attribute values, internal attributes (@id and @weight),
arithmetic operations, and a number of built-in functions. Here's an example:
$cl->SetSortMode ( SPH_SORT_EXPR,
"@weight + ( user_karma + ln(pageviews) )*0.1" );
The following operators and functions are supported. They are mimiced after MySQL.
The functions take a number of arguments depending on the specific function.
- Operators: +, -, *, /, <, > <=, >=, =, <>.
- Unary (1-argument) functions: abs(), ceil(), floor(), sin(), cos(), ln(), log2(), log10(), exp(), sqrt().
- Binary (2-argument) functions: min(), max(), pow().
- Ternary (3-argument) functions: if().
All calculations are performed in single-precision, 32-bit IEEE 754 floating point format.
Comparison operators (eg. = or <=) return 1.0 when the condition is true and 0.0 otherwise.
For instance, (a=b)+3 will evaluate to 4 when attribute 'a' is equal to attribute 'b', and to 3 when 'a' is not.
Unlike MySQL, the equality comparisons (ie. = and <> operators) introduce a small equality threshold (1e-6 by default).
If the difference between compared values is within the threshold, they will be considered equal.
All unary and binary functions are straightforward, they behave just like their mathematical counterparts.
But IF() behavior needs to be explained in more detail.
It takes 3 arguments, check whether the 1st argument is equal to 0.0, returns the 2nd argument if it is not zero, or the 3rd one when it is.
Note that unlike comparison operators, IF() does not use a threshold!
Therefore, it's safe to use comparison results as its 1st argument, but arithmetic operators might produce unexpected results.
For instance, the following two calls will produce different results even though they are logically equivalent:
IF ( sqrt(3)*sqrt(3)-3<>0, a, b )
IF ( sqrt(3)*sqrt(3)-3, a, b )
In the first case, the comparison operator <> will return 0.0 (false)
because of a threshold, and IF() will always return 'b' as a result.
In the second one, the same sqrt(3)*sqrt(3)-3 expression will be compared
with zero without threshold by the IF() function itself.
But its value will be slightly different from zero because of limited floating point
calculations precision. Because of that, the comparison with 0.0 done by IF()
will not pass, and the second variant will return 'a' as a result.
4.6. Grouping (clustering) search results
Sometimes it could be useful to group (or in other terms, cluster)
search results and/or count per-group match counts - for instance,
to draw a nice graph of how much maching blog posts were there per
each month; or to group Web search results by site; or to group
matching forum posts by author; etc.
In theory, this could be performed by doing only the full-text search
in Sphinx and then using found IDs to group on SQL server side. However,
in practice doing this with a big result set (10K-10M matches) would
typically kill performance.
To avoid that, Sphinx offers so-called grouping mode. It is enabled
with SetGroupBy() API call. When grouping, all matches are assigned to
different groups based on group-by value. This value is computed from
specified attribute using one of the following built-in functions:
- SPH_GROUPBY_DAY, extracts year, month and day in YYYYMMDD format from timestamp;
- SPH_GROUPBY_WEEK, extracts year and first day of the week number (counting from year start) in YYYYNNN format from timestamp;
- SPH_GROUPBY_MONTH, extracts month in YYYYMM format from timestamp;
- SPH_GROUPBY_YEAR, extracts year in YYYY format from timestamp;
- SPH_GROUPBY_ATTR, uses attribute value itself for grouping.
The final search result set then contains one best match per group.
Grouping function value and per-group match count are returned along
as "virtual" attributes named
@group and
@count respectively.
The result set is sorted by group-by sorting clause, with the syntax similar
to SPH_SORT_EXTENDED sorting clause
syntax. In addition to @id and @weight,
group-by sorting clause may also include:
- @group (groupby function value),
- @count (amount of matches in group).
The default mode is to sort by groupby value in descending order,
ie. by "@group desc".
On completion, total_found result parameter would
contain total amount of matching groups over he whole index.
WARNING: grouping is done in fixed memory
and thus its results are only approximate; so there might be more groups reported
in total_found than actually present. @count might also
be underestimated. To reduce inaccuracy, one should raise max_matches.
If max_matches allows to store all found groups, results will be 100% correct.
For example, if sorting by relevance and grouping by "published"
attribute with SPH_GROUPBY_DAY function, then the result set will
contain
- one most relevant match per each day when there were any
matches published,
- with day number and per-day match count attached,
- sorted by day number in descending order (ie. recent days first).
4.7. Distributed searching
To scale well, Sphinx has distributed searching capabilities.
Distributed searching is useful to improve query latency (ie. search
time) and throughput (ie. max queries/sec) in multi-server, multi-CPU
or multi-core environments. This is essential for applications which
need to search through huge amounts data (ie. billions of records
and terabytes of text).
The key idea is to horizontally partition (HP) searched data
accross search nodes and then process it in parallel.
Partitioning is done manually. You should
- setup several instances
of Sphinx programs (
indexer and searchd)
on different servers; - make the instances index (and search) different parts of data;
- configure a special distributed index on some of the
searchd
instances; - and query this index.
This index only contains references to other
local and remote indexes - so it could not be directly reindexed,
and you should reindex those indexes which it references instead.
When searchd receives a query against distributed index,
it does the following:
- connects to configured remote agents;
- issues the query;
- sequentially searches configured local indexes (while the remote agents are searching);
- retrieves remote agents' search results;
- merges all the results together, removing the duplicates;
- sends the merged resuls to client.
From the application's point of view, there are no differences
between usual and distributed index at all.
Any searchd instance could serve both as a master
(which aggregates the results) and a slave (which only does local searching)
at the same time. This has a number of uses:
- every machine in a cluster could serve as a master which
searches the whole cluster, and search requests could be balanced between
masters to achieve a kind of HA (high availability) in case any of the nodes fails;
-
if running within a single multi-CPU or multi-core machine, there
would be only 1 searchd instance quering itself as an agent and thus
utilizing all CPUs/core.
It is scheduled to implement better HA support which would allow
to specify which agents mirror each other, do health checks, keep track
of alive agents, load-balance requests, etc.
4.8. searchd query log format
searchd logs all succesfully executed search queries
into query log file. Here's an example:
[Fri Jun 29 21:17:58 2007] 0.004 sec [all/0/rel 35254 (0,20)] [lj] test
[Fri Jun 29 21:20:34 2007] 0.024 sec [all/0/rel 19886 (0,20) @channel_id] [lj] test
This log format is as follows:
[query-date] query-time [match-mode/filters-count/sort-mode
total-matches (offset,limit) @groupby-attr] [index-name] query
Match mode can take one of the following values:
- "all" for SPH_MATCH_ALL mode;
- "any" for SPH_MATCH_ANY mode;
- "phr" for SPH_MATCH_PHRASE mode;
- "bool" for SPH_MATCH_BOOLEAN mode;
- "ext" for SPH_MATCH_EXTENDED mode.
Sort mode can take one of the following values:
- "rel" for SPH_SORT_RELEVANCE mode;
- "attr-" for SPH_SORT_ATTR_DESC mode;
- "attr+" for SPH_SORT_ATTR_ASC mode;
- "tsegs" for SPH_SORT_TIME_SEGMENTS mode;
- "ext" for SPH_SORT_EXTENDED mode.
There is a number of native searchd client API implementations
for Sphinx. As of time of this writing, we officially support our own
PHP, Python, and Java implementations. There also are third party
free, open-source API implementations for Perl, Ruby, and C++.
The reference API implementation is in PHP, because (we believe)
Sphinx is most widely used with PHP than any other language.
This reference documentation is in turn based on reference PHP API,
and all code samples in this section will be given in PHP.
However, all other APIs provide the same methods and implement
the very same network protocol. Therefore the documentation does
apply to them as well. There might be minor differences as to the
method naming conventions or specific data structures used.
But the provided functionality must not differ across languages.
5.1. General API functionsPrototype: function GetLastError()
Returns last error message, as a string, in human readable format.
If there were no errors during the previous API call, empty string is returned.
You should call it when any other function (such as Query())
fails (typically, the failing function returns false). The returned string will
contain the error description.
The error message is not reset by this call; so you can safely
call it several times if needed.
Prototype: function GetLastWarning ()
Returns last warning message, as a string, in human readable format.
If there were no warnings during the previous API call, empty string is returned.
You should call it to verify whether your request
(such as Query()) was completed but with warnings.
For instance, search query against a distributed index might complete
succesfully even if several remote agents timed out. In that case,
a warning message would be produced.
The warning message is not reset by this call; so you can safely
call it several times if needed.
Prototype: function SetServer ( $host, $port )
Sets searchd host name and TCP port.
All subsequent requests will use the new host and port settings.
Default host and port are 'localhost' and 3312, respectively.
Prototype: function SetRetries ( $count, $delay=0 )
Sets distributed retry count and delay.
On temporary failures searchd will attempt up to
$count retries per agent. $delay is the delay
between the retries, in milliseconds. Retries are disabled by default.
Note that this call will not make the API itself retry on
temporary failure; it only tells searchd to do so.
Currently, the list of temporary failures includes all kinds of connect()
failures and maxed out (too busy) remote agents.
Prototype: function SetArrayResult ( $arrayresult )
PHP specific. Controls matches format in the search results set
(whether matches should be returned as an array or a hash).
$arrayresult argument must be boolean. If $arrayresult is false
(the default mode), matches will returned in PHP hash format with
document IDs as keys, and other information (weight, attributes)
as values. If $arrayresult is true, matches will be returned
as a plain array with complete per-match information including
document ID.
Introduced along with GROUP BY support on MVA attributes.
Group-by-MVA result sets may contain duplicate document IDs.
Thus they need to be returned as plain arrays, because hashes
will only keep one entry per document ID.
5.2. General query settingsPrototype: function SetLimits ( $offset, $limit, $max_matches=0, $cutoff=0 )
Sets offset into server-side result set ($offset) and amount of matches
to return to client starting from that offset ($limit). Can additionally
control maximum server-side result set size for current query ($max_matches)
and the threshold amount of matches to stop searching at ($cutoff).
All parameters must be non-negative integers.
First two parameters to SetLimits() are identical in behavior to MySQL
LIMIT clause. They instruct searchd to return at
most $limit matches starting from match number $offset.
The default offset and limit settings are 0 and 20, that is, to return
first 20 matches.
max_matches setting controls how much matches searchd
will keep in RAM while searching. All matching documents will be normally
processed, ranked, filtered, and sorted even if max_matches is set to 1.
But only best N documents are stored in memory at any given moment for performance
and RAM usage reasons, and this setting controls that N. Note that there are
two places where max_matches limit is enforced. Per-query
limit is controlled by this API call, but there also is per-server limit
controlled by max_matches setting in the config file. To prevent
RAM usage abuse, server will not allow to set per-query limit
higher than the per-server limit.
You can't retrieve more than max_matches matches to the client application.
The default limit is set to 1000. Normally, you must not have to go over
this limit. One thousand records is enough to present to the end user.
And if you're thinking about pulling the results to application
for further sorting or filtering, that would be much more efficient
if performed on Sphinx side.
$cutoff setting is intended for advanced performance control.
It tells searchd to forcibly stop search query
once $cutoff matches had been found and processed.
Prototype: function SetMaxQueryTime ( $max_query_time )
Sets maximum search query time, in milliseconds. Parameter must be
a non-negative integer. Default valus is 0 which means "do not limit".
Similar to $cutoff setting from SetLimits(),
but limits elapsed query time instead of processed matches count. Local search queries
will be stopped once that much time has elapsed. Note that if you're performing
a search which queries several local indexes, this limit applies to each index
separately.
5.3. Full-text search query settingsPrototype: function SetMatchMode ( $mode )
Sets full-text query matching mode, as described in Section 4.1, “Matching modes”.
Parameter must be a constant specifying one of the known modes.
WARNING: (PHP specific) you must not take the matching mode
constant name in quotes, that syntax specifies a string and is incorrect:
$cl->SetMatchMode ( "SPH_MATCH_ANY" ); // INCORRECT! will not work as expected
$cl->SetMatchMode ( SPH_MATCH_ANY ); // correct, works OK
Prototype: function SetRankingMode ( $ranker )
Sets ranking mode. Only available in SPH_MATCH_EXTENDED2 matching
mode at the time of this writing. Parameter must be a constant
specifying one of the known modes.
By default, Sphinx computes two factors which contribute to the final
match weight. The major part is query phrase proximity to document text.
The minor part is so-called BM25 statistical function, which varies
from 0 to 1 depending on the keyword frequency within document
(more occurrences yield higher weight) and within the whole index
(more rare keywords yield higher weight).
However, in some cases you'd want to compute weight differently -
or maybe avoid computing it at all for performance reasons because
you're sorting the result set by something else anyway. This can be
accomplished by setting the appropriate ranking mode.
Currently implemented modes are:
- SPH_RANK_PROXIMITY_BM25, default ranking mode which uses and combines
both phrase proximity and BM25 ranking.
- SPH_RANK_BM25, statistical ranking mode which uses BM25 ranking only (similar to
most other full-text engines). This mode is faster but may result in worse quality
on queries which contain more than 1 keyword.
- SPH_RANK_NONE, disabled ranking mode. This mode is the fastest.
It is essentially equivalent to boolean searching. A weight of 1 is assigned
to all matches.
Prototype: function SetSortMode ( $mode, $sortby="" )
Set matches sorting mode, as described in Section 4.5, “Sorting modes”.
Parameter must be a constant specifying one of the known modes.
WARNING: (PHP specific) you must not take the matching mode
constant name in quotes, that syntax specifies a string and is incorrect:
$cl->SetSortMode ( "SPH_SORT_ATTR_DESC" ); // INCORRECT! will not work as expected
$cl->SetSortMode ( SPH_SORT_ATTR_ASC ); // correct, works OK
Prototype: function SetWeights ( $weights )
Binds per-field weights in the order of appearance in the index.
DEPRECATED, use SetFieldWeights() instead.
Prototype: function SetFieldWeights ( $weights )
Binds per-field weights by name. Parameter must be a hash (associative array)
mapping string field names to integer weights.
Match ranking can be affected by per-field weights. For instance,
see Section 4.4, “Weighting” for an explanation how phrase proximity
ranking is affected. This call lets you specify what non-default
weights to assign to different full-text fields.
The weights must be positive 32-bit integers. The final weight
will be a 32-bit integer too. Default weight value is 1. Unknown
field names will be silently ignored.
There is no enforced limit on the maximum weight value at the
moment. However, beware that if you set it too high you can start
hitting 32-bit wraparound issues. For instance, if you set
a weight of 10,000,000 and search in extended mode, then
maximum possible weight will be equal to 10 million (your weight)
by 1 thousand (internal BM25 scaling factor, see Section 4.4, “Weighting”)
by 1 or more (phrase proximity rank). The result is at least 10 billion
that does not fit in 32 bits and will be wrapped around, producing
unexpected results.
Prototype: function SetIndexWeights ( $weights )
Sets per-index weights, and enables weighted summing of match weights
across different indexes. Parameter must be a hash (associative array)
mapping string index names to integer weights. Default is empty array
that means to disable weighting summing.
When a match with the same document ID is found in several different
local indexes, by default Sphinx simply chooses the match from the index
specified last in the query. This is to support searching through
partially overlapping index partitions.
However in some cases the indexes are not just partitions, and you
might want to sum the weights across the indexes instead of picking one.
SetIndexWeights() lets you do that. With summing enabled,
final match weight in result set will be computed as a sum of match
weight coming from the given index multiplied by respective per-index
weight specified in this call. Ie. if the document 123 is found in
index A with the weight of 2, and also in index B with the weight of 3,
and you called SetIndexWeights ( array ( "A"=>100, "B"=>10 ) ),
the final weight return to the client will be 2*100+3*10 = 230.
5.4. Result set filtering settingsPrototype: function SetIDRange ( $min, $max )
Sets an accepted range of document IDs. Parameters must be integers.
Defaults are 0 and 0; that combination means to not limit by range.
After this call, only those records that have document ID
between $min and $max (including IDs
exactly equal to $min or $max)
will be matched.
Prototype: function SetFilter ( $attribute, $values, $exclude=false )
Adds new integer values set filter.
On this call, additional new filter is added to the existing
list of filters. $attribute must be a string with
attribute name. $values must be a plain array
containing integer values. $exclude must be a boolean
value; it controls whether to accept the matching documents
(default mode, when $exclude is false) or reject them.
Only those documents where $attribute column value
stored in the index matches any of the values from $values
array will be matched (or rejected, if $exclude is true).
Prototype: function SetFilterRange ( $attribute, $min, $max, $exclude=false )
Adds new integer range filter.
On this call, additional new filter is added to the existing
list of filters. $attribute must be a string with
attribute name. $min and $max must be
integers that define the acceptable attribute values range
(including the boundaries). $exclude must be a boolean
value; it controls whether to accept the matching documents
(default mode, when $exclude is false) or reject them.
Only those documents where $attribute column value
stored in the index is between $min and $max
(including values that are exactly equal to $min or $max)
will be matched (or rejected, if $exclude is true).
5.4.4. SetFilterFloatRangePrototype: function SetFilterFloatRange ( $attribute, $min, $max, $exclude=false )
Adds new float range filter.
On this call, additional new filter is added to the existing
list of filters. $attribute must be a string with
attribute name. $min and $max must be
floats that define the acceptable attribute values range
(including the boundaries). $exclude must be a boolean
value; it controls whether to accept the matching documents
(default mode, when $exclude is false) or reject them.
Only those documents where $attribute column value
stored in the index is between $min and $max
(including values that are exactly equal to $min or $max)
will be matched (or rejected, if $exclude is true).
Prototype: function SetGeoAnchor ( $attrlat, $attrlong, $lat, $long )
Sets anchor point for and geosphere distance (geodistance) calculations, and enable them.
$attrlat and $attrlong must be strings that contain the names
of latitude and longitude attributes, respectively. $lat and $long
are floats that specify anchor point latitude and longitude, in radians.
Once an anchor point is set, you can use magic "@geodist" attribute
name in your filters and/or sorting expressions. Sphinx will compute geosphere distance
between the given anchor point and a point specified by latitude and lognitude
attributes from each full-text match, and attach this value to the resulting match.
The latitude and longitude values both in SetGeoAnchor and the index
attribute data are expected to be in radians. The result will be returned in meters,
so geodistance value of 1000.0 means 1 km. 1 mile is approximately 1609.344 meters.
Prototype: function SetGroupBy ( $attribute, $func, $groupsort="@group desc" )
Sets grouping attribute, function, and groups sorting mode; and enables grouping
(as described in Section 4.6, “Grouping (clustering) search results ”).
$attribute is a string that contains group-by attribute name.
$func is a constant that chooses a function applied to the attribute value in order to compute group-by key.
$groupsort is a clause that controls how the groups will be sorted. Its syntax is similar
to that described in Section 4.5, “SPH_SORT_EXTENDED mode”.
Grouping feature is very similar in nature to GROUP BY clause from SQL.
Results produces by this function call are going to be the same as produced
by the following pseudo code:
SELECT ... GROUP BY $func($attribute) ORDER BY $groupsort
Note that it's $groupsort that affects the order of matches
in the final result set. Sorting mode (see Section 5.3.3, “SetSortMode”)
affect the ordering of matches within group, ie.
what match will be selected as the best one from the group.
So you can for instance order the groups by matches count
and select the most relevant match within each group at the same time.
Prototype: function SetGroupDistinct ( $attribute )
Sets attribute name for per-group distinct values count calculations.
Only available for grouping queries.
$attribute is a string that contains the attribute name.
For each group, all values of this attribute will be stored (as RAM limits
permit), then the amount of distinct values will be calculated and returned
to the client. This feature is similar to COUNT(DISTINCT)
clause in standard SQL; so these Sphinx calls:
$cl->SetGroupBy ( "category", SPH_GROUPBY_ATTR, "@count desc" );
$cl->SetGroupDistinct ( "vendor" );
can be expressed using the following SQL clauses:
SELECT id, weight, all-attributes,
COUNT(DISTINCT vendor) AS @distinct,
COUNT(*) AS @count
FROM products
GROUP BY category
ORDER BY @count DESC
In the sample pseudo code shown just above, SetGroupDistinct() call
corresponds to COUNT(DISINCT vendor) clause only.
GROUP BY, ORDER BY, and COUNT(*)
clauses are all an equivalent of SetGroupBy() settings. Both queries
will return one matching row for each category. In addition to indexed attributes,
matches will also contain total per-category matches count, and the count
of distinct vendor IDs within each category.
Prototype: function Query ( $query, $index="*" )
Connects to searchd server, runs given search query
with current settings, obtains and returns the result set.
$query is a query string. $index is an index name (or names) string.
Returns false and sets GetLastError() message on general error.
Returns search result set on success.
Default value for $index is "*" that means
to query all local indexes. Characters allowed in index names include
Latin letters (a-z), numbers (0-9), minus sign (-), and underscore (_);
everything else is considered a separator. Therefore, all of the
following samples calls are valid and will search the same
two indexes:
$cl->Query ( "test query", "main delta" );
$cl->Query ( "test query", "main;delta" );
$cl->Query ( "test query", "main, delta" );
Index specification order matters. If document with identical IDs are found
in two or more indexes, weight and attribute values from the very last matching
index will be used for sorting and returning to client (unless explicitly
overridden with SetIndexWeights()). Therefore,
in the example above, matches from "delta" index will always win over
matches from "main".
On success, Query() returns a result set that contains
some of the found matches (as requested by SetLimits())
and additional general per-query statistics. The result set is a hash
(PHP specific; other languages might utilize other structures instead
of hash) with the following keys and values:
- "matches":
- Hash which maps found document IDs to another small hash containing document weight and attribute values
(or an array of the similar small hashes if SetArrayResult() was enabled).
- "total":
- Total amount of matches retrieved on server (ie. to the server side result set) by this query.
You can retrieve up to this amount of matches from server for this query text with current query settings.
- "total_found":
- Total amount of matching documents in index (that were found and procesed on server).
- "words":
- Hash which maps query keywords (case-folded, stemmed, and otherwise processed) to a small hash with per-keyword statitics ("docs", "hits").
- "error":
- Query error message reported by
searchd (string, human readable). Empty if there were no errors. - "warning":
- Query warning message reported by
searchd (string, human readable). Empty if there were no warnings.
Prototype: function AddQuery ( $query, $index="*" )
Adds additional query with current settings to multi-query batch.
$query is a query string. $index is an index name (or names) string.
Returns index to results array returned from RunQueries().
Batch queries (or multi-queries) enable searchd to perform internal
optimizations if possible. They also reduce network connection overheads and search process
creation overheads in all cases. They do not result in any additional overheads compared
to simple queries. Thus, if you run several different queries from your web page,
you should always consider using multi-queries.
For instance, running the same full-text query but with different
sorting or group-by settings will enable searchd
to perform expensive full-text search and ranking operation only once,
but compute multiple group-by results from its output.
This can be a big saver when you need to display not just plain
search results but also some per-category counts, such as the amount of
products grouped by vendor. Without multi-query, you would have to run several
queries which perform essentially the same search and retrieve the
same matches, but create result sets differently. With multi-query,
you simply pass all these querys in a single batch and Sphinx
optimizes the redundant full-text search internally.
AddQuery() internally saves full current settings state
along with the query, and you can safely change them afterwards for subsequent
AddQuery() calls. Already added queries will not be affected;
there's actually no way to change them at all. Here's an example:
$cl->SetSortMode ( SPH_SORT_RELEVANCE );
$cl->AddQuery ( "hello world", "documents" );
$cl->SetSortMode ( SPH_SORT_ATTR_DESC, "price" );
$cl->AddQuery ( "ipod", "products" );
$cl->AddQuery ( "harry potter", "books" );
$results = $cl->RunQueries ();
With the code above, 1st query will search for "hello world" in "documents" index
and sort results by relevance, 2nd query will search for "ipod" in "products"
index and sort results by price, and 3rd query will search for "harry potter"
in "books" index while still sorting by price. Note that 2nd SetSortMode() call
does not affect the first query (because it's already added) but affects both other
subsequent queries.
AddQuery() does not modify the current state. That is,
all current sorting, filtering, and grouping settings will not be affected by
this call; so subsequent queries can easily reuse current query settings.
AddQuery() returns an index into an array of results
that will be returned from RunQueries() call. It is simply
a sequentially increasing 0-based integer, ie. first call will return 0,
second will return 1, and so on. Just a small helper so you won't have
to track the indexes manualy if you need then.
Prototype: function RunQueries ()
Connect to searchd, runs a batch of all queries added using AddQuery(),
obtains and returns the result sets. Returns false and sets GetLastError()
message on general error (such as network I/O failure). Returns a plain array
of result sets on success.
Each result set in the returned array is exactly the same as
the result set returned from Query().
Note that the batch query request itself almost always succeds -
unless there's a network error, blocking index rotation in progress,
or another general failure which prevents the whole request from being
processed.
However individual queries within the batch might very well fail.
In this case their respective result sets will contain non-empty "error" message,
but no matches or query statistics. In the extreme case all queries within the batch
could fail. There still will be no general error reported, because API was able to
succesfully connect to searchd, submit the batch, and receive
the results - but every result set will have a specific error message.
Prototype: function ResetFilters ()
Clears all currently set filters.
This call is only normally required when using multi-queries. You might want
to set different filters for different queries in the batch. To do that,
you should call ResetFilters() and add new filters using
the respective calls.
Prototype: function ResetGroupBy ()
Clears all currently group-by settings, and disables group-by.
This call is only normally required when using multi-queries.
You can change individual group-by settings using SetGroupBy()
and SetGroupDistinct() calls, but you can not disable
group-by using those calls. ResetGroupBy()
fully resets previous group-by settings and disables group-by mode
in the current state, so that subsequent AddQuery()
calls can perform non-grouping searches.
5.7. Additional functionalityPrototype: function BuildExcerpts ( $docs, $index, $words, $opts=array() )
Excerpts (snippets) builder function. Connects to searchd,
asks it to generate excerpts (snippets) from given documents, and returns the results.
$docs is a plain array of strings that carry the documents' contents.
$index is an index name string. Different settings (such as charset,
morphology, wordforms) from given index will be used.
$words is a string that contains the keywords to highlight. They will
be processed with respect to index settings. For instance, if English stemming
is enabled in the index, "shoes" will be highlighted even if keyword is "shoe".
$opts is a hash which contains additional optional highlighting parameters:
- "before_match":
- A string to insert before a keyword match. Default is "<b>".
- "after_match":
- A string to insert after a keyword match. Default is "<b>".
- "chunk_separator":
- A string to insert between snippet chunks (passages). Default is " ... ".
- "limit":
- Maximum snippet size, in symbols (codepoints). Integer, default is 256.
- "around":
- How much words to pick around each matching keywords block. Integer, default is 5.
- "exact_phrase":
- Whether to highlight exact query phrase matches only instead of individual keywords. Boolean, default is false.
- "single_passage":
- whether to extract single best passage only. Boolean, default is false.
Returns false on failure. Returns a plain array of strings with excerpts (snippets) on success.
Prototype: function UpdateAttributes ( $index, $attrs, $values )
Instantly updates given attribute values in given documents.
Returns number of actually updated documents (0 or more) on success, or -1 on failure.
$index is a name of the index (or indexes) to be updated.
$attrs is a plain array with string attribute names, listing attributes that are updated.
$values is a hash where key is document ID, and value is a plain array of new attribute values.
$index can be either a single index name or a list, like in Query().
Unlike Query(), wildcard is not allowed and all the indexes
to update must be specified explicitly. The list of indexes can include
distributed index names. Updates on distributed indexes will be pushed
to all agents.
The updates only work with docinfo=extern storage strategy.
They are very fast because they're working fully in RAM, but they can also
be made persistent: updates are saved on disk on clean searchd
shutdown initiated by SIGTERM signal.
Usage example:
$cl->UpdateAttributes ( "test1", array("group_id"), array(1=>array(456)) );
$cl->UpdateAttributes ( "products", array ( "price", "amount_in_stock" ),
array ( 1001=>array(123,5), 1002=>array(37,11), 1003=>(25,129) ) );
The first sample statement will update document 1 in index "test1", setting "group_id" to 456.
The second one will update documents 1001, 1002 and 1003 in index "products". For document 1001,
the new price will be set to 123 and the new amount in stock to 5; for document 1002, the new price
will be 37 and the new amount will be 11; etc.
6. MySQL storage engine (SphinxSE)
SphinxSE is MySQL storage engine which can be compiled
into MySQL server 5.x using its pluggable architecure.
It is not available for MySQL 4.x series. It also requires
MySQL 5.0.22 or higher in 5.0.x series, or MySQL 5.1.12
or higher in 5.1.x series.
Despite the name, SphinxSE does not
actually store any data itself. It is actually a built-in client
which allows MySQL server to talk to searchd,
run search queries, and obtain search results. All indexing and
searching happen outside MySQL.
|
