Tokenizer.cc File Reference

#include "spear/Tokenizer.h"

Macros
#define	MAX_MULTI_WORD_SIZE   20

Functions
void	unitSplit (const Char *buffer, unsigned int size, std::vector< String > &tokens)

Detailed Description

Author

Mihai Surdeanu $Id$

Macro Definition Documentation

#define MAX_MULTI_WORD_SIZE   20

Function Documentation

void unitSplit	(	const Char *	buffer,
		unsigned int	size,
		std::vector< String > &	tokens
	)