Two-dimensional text: visualising contextualized language models

Input a phrase or a sentence (5-15 words):

Choose the model layer

We will use the ELMo model trained on English Wikipedia to infer contextualized embeddings for words in your query. Then, for each embeddings, we will find most similar words among 10 000 most frequent other words in this model's vocabulary. Since contextualized architectures do not store non-contextual word embeddings, we generated them beforehand by averaging contextualized embeddings of all occurrences of these words in the English Wikipedia.

Lexical substitutes are also known as paradigmatic replacements. These are the words which can in theory replace the corresponding word in your input sentence.

Substitutes will change depending on the context. The larger is the substitute font size, the more certain ELMo is about this word.