Package org.apache.lucene.analysis.ngram
Class NGramTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.apache.lucene.analysis.ngram.NGramTokenizer
-
- All Implemented Interfaces:
Closeable,AutoCloseable
- Direct Known Subclasses:
EdgeNGramTokenizer
public class NGramTokenizer extends Tokenizer
Tokenizes the input into n-grams of the given size(s).On the contrary to
NGramTokenFilter, this class sets offsets so that characters between startOffset and endOffset in the original stream are the same as the term chars.For example, "abcde" would be tokenized as (minGram=2, maxGram=3):
ngram tokens example Term ab abc bc bcd cd cde de Position increment 1 1 1 1 1 1 1 Position length 1 1 1 1 1 1 1 Offsets [0,2[ [0,3[ [1,3[ [1,4[ [2,4[ [2,5[ [3,5[ This tokenizer changed a lot in Lucene 4.4 in order to:
- tokenize in a streaming fashion to support streams which are larger than 1024 chars (limit of the previous version),
- count grams based on unicode code points instead of java chars (and never split in the middle of surrogate pairs),
- give the ability to
pre-tokenizethe stream before computing n-grams.
Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
-
Field Summary
Fields Modifier and Type Field Description static intDEFAULT_MAX_NGRAM_SIZEstatic intDEFAULT_MIN_NGRAM_SIZE-
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
-
Constructor Summary
Constructors Constructor Description NGramTokenizer()Creates NGramTokenizer with default min and max n-grams.NGramTokenizer(int minGram, int maxGram)Creates NGramTokenizer with given min and max n-grams.NGramTokenizer(AttributeFactory factory, int minGram, int maxGram)Creates NGramTokenizer with given min and max n-grams.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidend()booleanincrementToken()protected booleanisTokenChar(int chr)Only collect characters which satisfy this condition.voidreset()-
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset, setReader, setReaderTestPoint
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
-
-
-
Field Detail
-
DEFAULT_MIN_NGRAM_SIZE
public static final int DEFAULT_MIN_NGRAM_SIZE
- See Also:
- Constant Field Values
-
DEFAULT_MAX_NGRAM_SIZE
public static final int DEFAULT_MAX_NGRAM_SIZE
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
NGramTokenizer
public NGramTokenizer(int minGram, int maxGram)Creates NGramTokenizer with given min and max n-grams.- Parameters:
minGram- the smallest n-gram to generatemaxGram- the largest n-gram to generate
-
NGramTokenizer
public NGramTokenizer(AttributeFactory factory, int minGram, int maxGram)
Creates NGramTokenizer with given min and max n-grams.- Parameters:
factory-AttributeFactoryto useminGram- the smallest n-gram to generatemaxGram- the largest n-gram to generate
-
NGramTokenizer
public NGramTokenizer()
Creates NGramTokenizer with default min and max n-grams.
-
-
Method Detail
-
incrementToken
public final boolean incrementToken() throws IOException- Specified by:
incrementTokenin classTokenStream- Throws:
IOException
-
isTokenChar
protected boolean isTokenChar(int chr)
Only collect characters which satisfy this condition.
-
end
public final void end() throws IOException- Overrides:
endin classTokenStream- Throws:
IOException
-
reset
public final void reset() throws IOException- Overrides:
resetin classTokenizer- Throws:
IOException
-
-