Combining different features as input to Neural Network

Question

I use two different sources of information as input to my neural model. The model takes a word as input and produces binary [1/0] output. I have represented each word by using its word embedding (1024 dimensional vector) and its Valence, Arousal, and Dominance Lexicon (3 dimensional vector). I concatenate them and obtain 1027 dimensional vector for each word.

My questions are:

1. Should I normalize/preprocess those vector before concatenating them? My lexicon values are between [0,1] and my embeddings are the ELMO embeddings produced depending on a context. Therefore I am not sure about the range of the embeddings.

2. If yes, should I preprocess one of them before merging, what should I do ?

3. Is there any good resource that might have an answer for my question?

Answer 1

You should have all of your features in a comparable range. You can leave lexicon values in the [0, 1] range but normalize embedding to have approximately the same range (with using, for example, MinMaxScaler).

Answer 2

So, there are a number of questions that needs to be answered in this question. First of all, I should say that there is no need to normalize the dataset before feeding it into the network as the embeddings are always normalized if the dataset model architecture is properly defined.

Just before merging the values, there is certainly a need to bring the values to fixed range having appropriate dimensions.

I believe that just to be double sure about the embedding representations, you must refer to Deep contextualized word representations and GloVe original papers.