ch.ethz.mxquery.sms.ftstore
Class LinguisticTokenGenerator

java.lang.Object
  ch.ethz.mxquery.sms.ftstore.LinguisticTokenGenerator

public class LinguisticTokenGenerator
extends java.lang.Object

A class for preprocessing the documents: Tokenization and conversion of XDM tokens into Linguistic Tokens

Constructor Summary

LinguisticTokenGenerator()

Method Summary

java.util.Vector	getLinguisticToken(Token textToken)
void	incrementParagraphCounter()
java.util.Vector	tokenize(java.lang.String text)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LinguisticTokenGenerator

public LinguisticTokenGenerator()
                         throws MXQueryException

Throws:

MXQueryException

Method Detail

getLinguisticToken

public java.util.Vector getLinguisticToken(Token textToken)

Returns:

vector of Linguistic Tokens (of the XDM TextToken)

incrementParagraphCounter

public void incrementParagraphCounter()

tokenize

public java.util.Vector tokenize(java.lang.String text)