The Lucene project lets you index the documents on your filesystem or
web server so you can run combined full text and metadata searches. A full
text search takes one or more
words of a human language as a query and should return documents which
are the "most relevant" for those words. Web searches are a classic
example of full text searches. Metadata searches should be familiar
to anyone who has used the find command; for example,
looking for all files that have been modified in the last week.
The primary goal of Lucene is to provide a fast index and query
implementation and to specify an interface to the index implementation
-- how to send queries to it and get your results back as fast as
possible. Lucene is not, by itself, designed to be a complete user-facing
index solution but rather to provide the heart of such a
system. There are also higher level projects which use one of the
Lucene implementations to provide search capabilities, for example,
KDE4's strigi desktop search. If you just want to add a search
capability to something then you might like to explore these higher level
tools to see if you can save the time of writing a program that uses
the Lucene API directly.
It is tempting to think of adding full text to an index as just a
filesystem traversal where you read each file and shove the byte
contents into the index. Normally you want to extend this to allow
conversions too, such as extracting the plain text of PDF files
and indexing the extracted human readable text instead of the bytes
that comprise the PDF file. The metadata associated with a document is
entirely up to you, for example, extracting the Vorbis artist, album
and track comments from FLAC audio files and adding them as metadata.
Using Lucene to index your Web site lets you offer a text search feature - like
a Google search box - for servicing searches like "Wakelocks embedded".
This is only the beginning though, because you can also offer advanced
searches by combining metadata into the search. If you build a Lucene
index for each registered user, the personalized search you can offer
is hard to beat. For example, finding pages about "locking" that
contain a link to a specific web site in the article comments. Or any
article on "locking" that contains a comment by any one of your
Lucene is actually an umbrella project which has many implementations
in Java, C++, Ruby and PHP among others.
Probably the most widely known implementation of Lucene is the original one
that is done in Java. In recent times, implementations in C++ (CLucene) and PHP
have become available. There are also implementations in Perl and
Ruby, see the
full list for details.
The CLucene page states that its primary goal is to be
faster than the Java version. It would appear that the PHP
by the desire to be homogeneous with the PHP environment.
The implementation of these full text and metadata search types
normally call for different queries and thus different
implementations to best resolve the queries. For example, it might be
quite common to want to search for a range in a metadata query, like
all the documents added to the index in December, whereas a full text
query might demand ranking of documents that contain the strings
"DDR3" and "latency".
You don't really need to know what Lucene does on its side of the API
to build and search indexes with it, though a high level knowledge of
what happens in the implementation can help you understand how to make
efficient use of the API.
Abstractly, a Lucene index consists of many Document objects, each of
which contains one or more fields. A
field is a key-value pair, for example, the key of "indexed-on" and a
value "Wed Dec 17, 2008 @ 3:58 PM". The full text content of a
document is also added to the Lucene index as a field property of a
Fields can be stored verbatim in the index, or have an index created
for them, or both. You might want to index and store the URL that a
document was retrieved from, but might want to only index the document
text because storing it verbatim might make the index too large for
your application. An index on the contents of a file is likely to be
much smaller than the file itself. If you have access to the original
file you don't really want to store it in the Lucene index verbatim
too. A field can also be tokenized or stored atomically (a so called
would want to tokenize the text content of a file but probably want
the date it was indexed to remain an atomic value.
Normally you would have Lucene tokenize the text of a file and build
for the tokens. For example, the word "token" would have a list of
which document numbers contain that word along with other metadata
relating to how often that term appears in each document relative to
the length of the document. This way queries looking for "token" and
"lucene" can be resolved by merging the two lists for each token.
A great deal of attention has been paid to not
locking data in the index with Lucene. This way,
the index can undergo updating in the background while it is
actively being used to service searches.
This eliminates the need to wait on the background process.
You can only have a single update running for an index at
any time, but many clients can be reading the index while that update
A Lucene index is made up of one or more segments. Each segment is
fully independent of any other segment and is stored in one or more
files. Concurrency without locking is achieved by writing any new or
changed data to a new Segment. One way to speed up indexing documents
and create fewer segments is to have Lucene cache as many of the added
documents in RAM and flush out a single, large segment on a less
For Java Lucene the setRAMBufferSizeMB
is used to set how much RAM can be used before a new segment is
written, its default is only 16Mb. Creating larger segments during
indexing means it will take slightly longer before clients can see new
documents (because the new segment is not written and is thus not
accessible) but will make for fewer, larger segments and thus less
need to merge segments later.
Instead of flushing a new segment when enough RAM has been used, you
can force a segment to be flushed every X documents with setMaxBufferedDocs. By
default, flushing is done when the buffered RAM size is reached and
there is no default maximum number of documents before a flush.
Segments are merged either periodically during the adding of documents or by
calling one of many optimize
methods. If an index is to remain constant for a period of time it is
a good idea to optimize it so that multiple segments are converted
into a single segment. Optimization has the additional side benefit
that if your filesystem is not full, writing a new single-segment
Lucene index should also mean that the index is stored in a single
Adding segments and merging segments are very similar operations.
To merge segments, all of the data is copied from the old segments
into a new segment and the old segments are then discarded.
The currently active segments are listed in the "segments" file.
Depending on how the
implementation of Lucene you are using operates, the segments file
might use a commit lock to protect it while it is being updated.
At any rate, as the segments file just lists the file names and other
metadata about segments, it can be updated very quickly.
I mentioned at the outset that Lucene specializes in full text
indexing. There are some issues when using Lucene for numerical
metadata which make using those datatypes a more complex task than
just shoving full text into the index.
Knowing the Lucene API and how to include and search for information
in a Lucene index can allow you to develop many applications.
Hopefully the glimpse behind
the API that I've included can help you get started writing
applications that use Lucene efficiently. Because there are
implementations of Lucene in PHP, C++, C#, Java and other languages
you can apply general knowledge of Lucene to applications ranging from
Web development to embedded coding.
to post comments)