Package org.apache.lucene.analysis.nl
Class DutchAnalyzer
- java.lang.Object
-
- org.apache.lucene.analysis.Analyzer
-
- org.apache.lucene.analysis.ReusableAnalyzerBase
-
- org.apache.lucene.analysis.nl.DutchAnalyzer
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public final class DutchAnalyzer extends ReusableAnalyzerBase
Analyzer
for Dutch language.Supports an external list of stopwords (words that will not be indexed at all), an external list of exclusions (word that will not be stemmed, but indexed) and an external list of word-stem pairs that overrule the algorithm (dictionary stemming). A default set of stopwords is used unless an alternative list is specified, but the exclusion list is empty by default.
You must specify the required
Version
compatibility when creating DutchAnalyzer:- As of 3.6,
DutchAnalyzer(Version, Set)
andDutchAnalyzer(Version, Set, Set)
also populate the default entries for the stem override dictionary - As of 3.1, Snowball stemming is done with SnowballFilter, LowerCaseFilter is used prior to StopFilter, and Snowball stopwords are used by default.
- As of 2.9, StopFilter preserves position increments
NOTE: This class uses the same
Version
dependent settings asStandardAnalyzer
.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.analysis.ReusableAnalyzerBase
ReusableAnalyzerBase.TokenStreamComponents
-
-
Field Summary
Fields Modifier and Type Field Description static String
DEFAULT_STOPWORD_FILE
File containing default Dutch stopwords.static String[]
DUTCH_STOP_WORDS
Deprecated.usegetDefaultStopSet()
instead
-
Constructor Summary
Constructors Constructor Description DutchAnalyzer(Version matchVersion)
Builds an analyzer with the default stop words (getDefaultStopSet()
) and a few default entries for the stem exclusion table.DutchAnalyzer(Version matchVersion, File stopwords)
Deprecated.useDutchAnalyzer(Version, Set)
insteadDutchAnalyzer(Version matchVersion, String... stopwords)
Deprecated.useDutchAnalyzer(Version, Set)
insteadDutchAnalyzer(Version matchVersion, HashSet<?> stopwords)
Deprecated.useDutchAnalyzer(Version, Set)
insteadDutchAnalyzer(Version matchVersion, Set<?> stopwords)
DutchAnalyzer(Version matchVersion, Set<?> stopwords, Set<?> stemExclusionTable)
DutchAnalyzer(Version matchVersion, Set<?> stopwords, Set<?> stemExclusionTable, CharArrayMap<String> stemOverrideDict)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description protected ReusableAnalyzerBase.TokenStreamComponents
createComponents(String fieldName, Reader aReader)
Returns a (possibly reused)TokenStream
which tokenizes all the text in the providedReader
.static Set<?>
getDefaultStopSet()
Returns an unmodifiable instance of the default stop-words set.void
setStemDictionary(File stemdictFile)
Deprecated.This prevents reuse of TokenStreams.void
setStemExclusionTable(File exclusionlist)
Deprecated.useDutchAnalyzer(Version, Set, Set)
insteadvoid
setStemExclusionTable(String... exclusionlist)
Deprecated.useDutchAnalyzer(Version, Set, Set)
insteadvoid
setStemExclusionTable(HashSet<?> exclusionlist)
Deprecated.useDutchAnalyzer(Version, Set, Set)
instead-
Methods inherited from class org.apache.lucene.analysis.ReusableAnalyzerBase
initReader, reusableTokenStream, tokenStream
-
Methods inherited from class org.apache.lucene.analysis.Analyzer
close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setPreviousTokenStream
-
-
-
-
Field Detail
-
DUTCH_STOP_WORDS
@Deprecated public static final String[] DUTCH_STOP_WORDS
Deprecated.usegetDefaultStopSet()
insteadList of typical Dutch stopwords.
-
DEFAULT_STOPWORD_FILE
public static final String DEFAULT_STOPWORD_FILE
File containing default Dutch stopwords.- See Also:
- Constant Field Values
-
-
Constructor Detail
-
DutchAnalyzer
public DutchAnalyzer(Version matchVersion)
Builds an analyzer with the default stop words (getDefaultStopSet()
) and a few default entries for the stem exclusion table.
-
DutchAnalyzer
public DutchAnalyzer(Version matchVersion, Set<?> stopwords, Set<?> stemExclusionTable)
-
DutchAnalyzer
public DutchAnalyzer(Version matchVersion, Set<?> stopwords, Set<?> stemExclusionTable, CharArrayMap<String> stemOverrideDict)
-
DutchAnalyzer
@Deprecated public DutchAnalyzer(Version matchVersion, String... stopwords)
Deprecated.useDutchAnalyzer(Version, Set)
insteadBuilds an analyzer with the given stop words.- Parameters:
matchVersion
-stopwords
-
-
DutchAnalyzer
@Deprecated public DutchAnalyzer(Version matchVersion, HashSet<?> stopwords)
Deprecated.useDutchAnalyzer(Version, Set)
insteadBuilds an analyzer with the given stop words.- Parameters:
stopwords
-
-
DutchAnalyzer
@Deprecated public DutchAnalyzer(Version matchVersion, File stopwords)
Deprecated.useDutchAnalyzer(Version, Set)
insteadBuilds an analyzer with the given stop words.- Parameters:
stopwords
-
-
-
Method Detail
-
getDefaultStopSet
public static Set<?> getDefaultStopSet()
Returns an unmodifiable instance of the default stop-words set.- Returns:
- an unmodifiable instance of the default stop-words set.
-
setStemExclusionTable
@Deprecated public void setStemExclusionTable(String... exclusionlist)
Deprecated.useDutchAnalyzer(Version, Set, Set)
insteadBuilds an exclusionlist from an array of Strings.- Parameters:
exclusionlist
-
-
setStemExclusionTable
@Deprecated public void setStemExclusionTable(HashSet<?> exclusionlist)
Deprecated.useDutchAnalyzer(Version, Set, Set)
insteadBuilds an exclusionlist from a Hashtable.
-
setStemExclusionTable
@Deprecated public void setStemExclusionTable(File exclusionlist)
Deprecated.useDutchAnalyzer(Version, Set, Set)
insteadBuilds an exclusionlist from the words contained in the given file.
-
setStemDictionary
@Deprecated public void setStemDictionary(File stemdictFile)
Deprecated.This prevents reuse of TokenStreams. If you wish to use a custom stem dictionary, create your own Analyzer withStemmerOverrideFilter
Reads a stemdictionary file , that overrules the stemming algorithm This is a textfile that contains per line word\tstem, i.e: two tab seperated words
-
createComponents
protected ReusableAnalyzerBase.TokenStreamComponents createComponents(String fieldName, Reader aReader)
Returns a (possibly reused)TokenStream
which tokenizes all the text in the providedReader
.- Specified by:
createComponents
in classReusableAnalyzerBase
- Parameters:
fieldName
- the name of the fields content passed to theReusableAnalyzerBase.TokenStreamComponents
sink as a readeraReader
- the reader passed to theTokenizer
constructor- Returns:
- A
TokenStream
built from aStandardTokenizer
filtered withStandardFilter
,LowerCaseFilter
,StopFilter
,KeywordMarkerFilter
if a stem exclusion set is provided,StemmerOverrideFilter
, andSnowballFilter
-
-