How to set the dictionary for text analysis using neural networks

manpreet Best Answer 3 years ago

I want to use a neural network to do text analysis.

If I use a large dictionary, then it will contain all the words in training and test set, but the size of the dictionary is too large which will make the neural network training hard and time consuming. Another option is using the words in training set only, and make it a dictionary, for test set, all the words that are not in dictionary will be disregarded. Which option is better?

In general, I think for a bag of words model, the second option doesn't harm too much, and we may use it. But for other models, for example, if we treat the text as a series (time series), and we use a recurrent neural network, disregarding some words may break the relationship in this series or create a wrong relationship. So how to set the dictionary for the above situations?