{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "In our previous example, we operated on high-dimensional bag-of-words vectors with length `vocab_size`, and we were explicitly converting from low-dimensional positional representation vectors into sparse one-hot representation. This one-hot representation is not memory-efficient, in addition, each word is treated independently from each other, i.e. one-hot encoded vectors do not express any semantic similarity between words.\n",
    "\n",
    "In this unit, we will continue exploring **News AG** dataset. To begin, let's load the data and get some definitions from the previous notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading dataset...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\train.csv: 29.5MB [00:01, 18.8MB/s]                            \n",
      "d:\\WORK\\ai-for-beginners\\5-NLP\\14-Embeddings\\data\\test.csv: 1.86MB [00:00, 11.2MB/s]                          \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building vocab...\n",
      "Vocab size =  95812\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torchtext\n",
    "import numpy as np\n",
    "from torchnlp import *\n",
    "train_dataset, test_dataset, classes, vocab = load_dataset()\n",
    "vocab_size = len(vocab)\n",
    "print(\"Vocab size = \",vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## What is embedding?\n",
    "\n",
    "The idea of **embedding** is to represent words by lower-dimensional dense vectors, which somehow reflect semantic meaning of a word. We will later discuss how to build meaningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector. \n",
    "\n",
    "So, embedding layer would take a word as an input, and produce an output vector of specified `embedding_size`. In a sense, it is very similar to `Linear` layer, but instead of taking one-hot encoded vector, it will be able to take a word number as an input.\n",
    "\n",
    "By using embedding layer as a first layer in our network, we can switch from bag-or-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`.  \n",
    "\n",
    "![Image showing an embedding classifier for five sequence words.](images/embedding-classifier-example.png)\n",
    "\n",
    "Our classifier neural network will start with embedding layer, then aggregation layer, and linear classifier on top of it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EmbedClassifier(torch.nn.Module):\n",
    "    def __init__(self, vocab_size, embed_dim, num_class):\n",
    "        super().__init__()\n",
    "        self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
    "        self.fc = torch.nn.Linear(embed_dim, num_class)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.embedding(x)\n",
    "        x = torch.mean(x,dim=1)\n",
    "        return self.fc(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dealing with variable sequence size\n",
    "\n",
    "As a result of this architecture, minibatches to our network would need to be created in a certain way. In the previous unit, when using bag-of-words, all BoW tensors in a minibatch had equal size `vocab_size`, regardless of the actual length of our text sequence. Once we move to word embeddings, we would end up with variable number of words in each text sample, and when combining those samples into minibatches we would have to apply some padding.\n",
    "\n",
    "This can be done using the same technique of providing `collate_fn` function to the datasource:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def padify(b):\n",
    "    # b is the list of tuples of length batch_size\n",
    "    #   - first element of a tuple = label, \n",
    "    #   - second = feature (text sequence)\n",
    "    # build vectorized sequence\n",
    "    v = [encode(x[1]) for x in b]\n",
    "    # first, compute max length of a sequence in this minibatch\n",
    "    l = max(map(len,v))\n",
    "    return ( # tuple of two tensors - labels and features\n",
    "        torch.LongTensor([t[0]-1 for t in b]),\n",
    "        torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])\n",
    "    )\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training embedding classifier\n",
    "\n",
    "Now that we have defined proper dataloader, we can train the model using the training function we have defined in the previous unit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3200: acc=0.6415625\n",
      "6400: acc=0.6865625\n",
      "9600: acc=0.7103125\n",
      "12800: acc=0.726953125\n",
      "16000: acc=0.739375\n",
      "19200: acc=0.75046875\n",
      "22400: acc=0.7572321428571429\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.889799795315499, 0.7623160588611644)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
    "train_epoch(net,train_loader, lr=1, epoch_size=25000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note**: We are only training for 25k records here (less than one full epoch) for the sake of time, but you can continue training, write a function to train for several epochs, and experiment with learning rate parameter to achieve higher accuracy. You should be able to go to the accuracy of about 90%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EmbeddingBag Layer and Variable-Length Sequence Representation\n",
    "\n",
    "In the previous architecture, we needed to pad all sequences to the same length in order to fit them into a minibatch. This is not the most efficient way to represent variable length sequences - another apporach would be to use **offset** vector, which would hold offsets of all sequences stored in one large vector.\n",
    "\n",
    "![Image showing an offset sequence representation](images/offset-sequence-representation.png)\n",
    "\n",
    "> **Note**: On the picture above, we show a sequence of characters, but in our example we are working with sequences of words. However, the general principle of representing sequences with offset vector remains the same.\n",
    "\n",
    "To work with offset representation, we use [`EmbeddingBag`](https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html) layer. It is similar to `Embedding`, but it takes content vector and offset vector as input, and it also includes averaging layer, which can be `mean`, `sum` or `max`.\n",
    "\n",
    "Here is modified network that uses `EmbeddingBag`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EmbedClassifier(torch.nn.Module):\n",
    "    def __init__(self, vocab_size, embed_dim, num_class):\n",
    "        super().__init__()\n",
    "        self.embedding = torch.nn.EmbeddingBag(vocab_size, embed_dim)\n",
    "        self.fc = torch.nn.Linear(embed_dim, num_class)\n",
    "\n",
    "    def forward(self, text, off):\n",
    "        x = self.embedding(text, off)\n",
    "        return self.fc(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To prepare the dataset for training, we need to provide a conversion function that will prepare the offset vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def offsetify(b):\n",
    "    # first, compute data tensor from all sequences\n",
    "    x = [torch.tensor(encode(t[1])) for t in b]\n",
    "    # now, compute the offsets by accumulating the tensor of sequence lengths\n",
    "    o = [0] + [len(t) for t in x]\n",
    "    o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
    "    return ( \n",
    "        torch.LongTensor([t[0]-1 for t in b]), # labels\n",
    "        torch.cat(x), # text \n",
    "        o\n",
    "    )\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, that unlike in all previous examples, our network now accepts two parameters: data vector and offset vector, which are of different sizes. Sililarly, our data loader also provides us with 3 values instead of 2: both text and offset vectors are provided as features. Therefore, we need to slightly adjust our training function to take care of that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3200: acc=0.6153125\n",
      "6400: acc=0.6615625\n",
      "9600: acc=0.6932291666666667\n",
      "12800: acc=0.715078125\n",
      "16000: acc=0.7270625\n",
      "19200: acc=0.7382291666666667\n",
      "22400: acc=0.7486160714285715\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(22.771553103007037, 0.7551983365323096)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = EmbedClassifier(vocab_size,32,len(classes)).to(device)\n",
    "\n",
    "def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):\n",
    "    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
    "    loss_fn = loss_fn.to(device)\n",
    "    net.train()\n",
    "    total_loss,acc,count,i = 0,0,0,0\n",
    "    for labels,text,off in dataloader:\n",
    "        optimizer.zero_grad()\n",
    "        labels,text,off = labels.to(device), text.to(device), off.to(device)\n",
    "        out = net(text, off)\n",
    "        loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total_loss+=loss\n",
    "        _,predicted = torch.max(out,1)\n",
    "        acc+=(predicted==labels).sum()\n",
    "        count+=len(labels)\n",
    "        i+=1\n",
    "        if i%report_freq==0:\n",
    "            print(f\"{count}: acc={acc.item()/count}\")\n",
    "        if epoch_size and count>epoch_size:\n",
    "            break\n",
    "    return total_loss.item()/count, acc.item()/count\n",
    "\n",
    "\n",
    "train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Semantic Embeddings: Word2Vec\n",
    "\n",
    "In our previous example, the model embedding layer learnt to map words to vector representation, however, this representation did not have much semantical meaning. It would be nice to learn such vector representation, that similar words or symonims would correspond to vectors that are close to each other in terms of some vector distance (eg. euclidian distance).\n",
    "\n",
    "To do that, we need to pre-train our embedding model on a large collection of text in a specific way. One of the first ways to train semantic embeddings is called [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). It is based on two main architectures that are used to produce a distributed representation of words:\n",
    "\n",
    " - **Continuous bag-of-words** (CBoW) — in this architecture, we train the model to predict a word from surrounding context. Given the ngram $(W_{-2},W_{-1},W_0,W_1,W_2)$, the goal of the model is to predict $W_0$ from $(W_{-2},W_{-1},W_1,W_2)$.\n",
    " - **Continuous skip-gram** is opposite to CBoW. The model uses surrounding window of context words to predict the current word.\n",
    "\n",
    "CBoW is faster, while skip-gram is slower, but does a better job of representing infrequent words.\n",
    "\n",
    "![Image showing both CBoW and Skip-Gram algorithms to convert words to vectors.](./images/example-algorithms-for-converting-words-to-vectors.png)\n",
    "\n",
    "To experiment with word2vec embedding pre-trained on Google News dataset, we can use **gensim** library. Below we find the words most similar to 'neural'\n",
    "\n",
    "> **Note:** When you first create word vectors, downloading them can take some time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim.downloader as api\n",
    "w2v = api.load('word2vec-google-news-300')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "neuronal -> 0.7804799675941467\n",
      "neurons -> 0.7326500415802002\n",
      "neural_circuits -> 0.7252851724624634\n",
      "neuron -> 0.7174385190010071\n",
      "cortical -> 0.6941086649894714\n",
      "brain_circuitry -> 0.6923246383666992\n",
      "synaptic -> 0.6699118614196777\n",
      "neural_circuitry -> 0.6638563275337219\n",
      "neurochemical -> 0.6555314064025879\n",
      "neuronal_activity -> 0.6531826257705688\n"
     ]
    }
   ],
   "source": [
    "for w,p in w2v.most_similar('neural'):\n",
    "    print(f\"{w} -> {p}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also compute vector embeddings from the word, to be used in training classification model (we only show first 20 components of the vector for clarity):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.01226807,  0.06225586,  0.10693359,  0.05810547,  0.23828125,\n",
       "        0.03686523,  0.05151367, -0.20703125,  0.01989746,  0.10058594,\n",
       "       -0.03759766, -0.1015625 , -0.15820312, -0.08105469, -0.0390625 ,\n",
       "       -0.05053711,  0.16015625,  0.2578125 ,  0.10058594, -0.25976562],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.word_vec('play')[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great thing about semantical embeddings is that you can manipulate vector encoding to change the semantics. For example, we can ask to find a word, whose vector representation would be as close as possible to words *king* and *woman*, and as far away from the word *man*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('queen', 0.7118192911148071)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both CBOW and Skip-Grams are “predictive” embeddings, in that they only take local contexts into account. Word2Vec does not take advantage of global context. \n",
    "\n",
    "**FastText**, builds on Word2Vec by learning vector representations for each word and the charachter n-grams found within each word. The values of the representations are then averaged into one vector at each training step. While this adds a lot of additional computation to pre-training it enables word embeddings to encode sub-word information. \n",
    "\n",
    "Another method, **GloVe**, leverages the idea of co-occurence matrix, uses neural methods to decompose co-occurrence matrix into more expressive and non linear word vectors.\n",
    "\n",
    "You can play with the example by changing embeddings to FastText and GloVe, since gensim supports several different word embedding models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Pre-Trained Embeddings in PyTorch\n",
    "\n",
    "We can modify the example above to pre-populate the matrix in our embedding layer with semantical embeddings, such as Word2Vec. We need to take into account that vocabularies of pre-trained embedding and our text corpus will likely not match, so we will initialize weights for the missing words with random values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedding size: 300\n",
      "Populating matrix, this will take some time...Done, found 41080 words, 54732 words missing\n"
     ]
    }
   ],
   "source": [
    "embed_size = len(w2v.get_vector('hello'))\n",
    "print(f'Embedding size: {embed_size}')\n",
    "\n",
    "net = EmbedClassifier(vocab_size,embed_size,len(classes))\n",
    "\n",
    "print('Populating matrix, this will take some time...',end='')\n",
    "found, not_found = 0,0\n",
    "for i,w in enumerate(vocab.itos):\n",
    "    try:\n",
    "        net.embedding.weight[i].data = torch.tensor(w2v.get_vector(w))\n",
    "        found+=1\n",
    "    except:\n",
    "        net.embedding.weight[i].data = torch.normal(0.0,1.0,(embed_size,))\n",
    "        not_found+=1\n",
    "\n",
    "print(f\"Done, found {found} words, {not_found} words missing\")\n",
    "net = net.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train our model. Note that the time it takes to train the model is significantly larger than in the previous example, due to larger embedding layer size, and thus much higher number of parameters. Also, because of this, we may need to train our model on more examples if we want to avoid overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3200: acc=0.6359375\n",
      "6400: acc=0.68109375\n",
      "9600: acc=0.7067708333333333\n",
      "12800: acc=0.723671875\n",
      "16000: acc=0.73625\n",
      "19200: acc=0.7463541666666667\n",
      "22400: acc=0.7560714285714286\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(214.1013875559821, 0.7626759436980166)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our case we do not see huge increase in accuracy, which is likely to quite different vocalularies. \n",
    "To overcome the problem of different vocabularies, we can use one of the following solutions:\n",
    "* Re-train word2vec model on our vocabulary\n",
    "* Load our dataset with the vocabulary from the pre-trained word2vec model. Vocabulary used to load the dataset can be specified during loading.\n",
    "\n",
    "The latter approach seems easiter, especially because PyTorch `torchtext` framework contains built-in support for embeddings. We can, for example, instantiate GloVe-based vocabulary in the following manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████▉| 399999/400000 [00:15<00:00, 25411.14it/s]\n"
     ]
    }
   ],
   "source": [
    "vocab = torchtext.vocab.GloVe(name='6B', dim=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loaded vocabulary has the following basic operations:\n",
    "* `vocab.stoi` dictionary allows us to convert word into its dictionary index\n",
    "* `vocab.itos` does the opposite - converts number into word\n",
    "* `vocab.vectors` is the array of embedding vectors, so to get the embedding of a word `s` we need to use `vocab.vectors[vocab.stoi[s]]`\n",
    "\n",
    "Here is the example of manipulating embeddings to demonstrate the equation **kind-man+woman = queen** (I had to tweak the coefficient a bit to make it work):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'queen'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the vector corresponding to kind-man+woman\n",
    "qvec = vocab.vectors[vocab.stoi['king']]-vocab.vectors[vocab.stoi['man']]+1.3*vocab.vectors[vocab.stoi['woman']]\n",
    "# find the index of the closest embedding vector \n",
    "d = torch.sum((vocab.vectors-qvec)**2,dim=1)\n",
    "min_idx = torch.argmin(d)\n",
    "# find the corresponding word\n",
    "vocab.itos[min_idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the classifier using those embeddings, we first need to encode our dataset using GloVe vocabulary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def offsetify(b):\n",
    "    # first, compute data tensor from all sequences\n",
    "    x = [torch.tensor(encode(t[1],voc=vocab)) for t in b] # pass the instance of vocab to encode function!\n",
    "    # now, compute the offsets by accumulating the tensor of sequence lengths\n",
    "    o = [0] + [len(t) for t in x]\n",
    "    o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
    "    return ( \n",
    "        torch.LongTensor([t[0]-1 for t in b]), # labels\n",
    "        torch.cat(x), # text \n",
    "        o\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen above, all vector embeddings are stored in `vocab.vectors` matrix. It makes it super-easy to load those weights into weights of embedding layer using simple copying:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = EmbedClassifier(len(vocab),len(vocab.vectors[0]),len(classes))\n",
    "net.embedding.weight.data = vocab.vectors\n",
    "net = net.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train our model and see if we get better results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3200: acc=0.6271875\n",
      "6400: acc=0.68078125\n",
      "9600: acc=0.7030208333333333\n",
      "12800: acc=0.71984375\n",
      "16000: acc=0.7346875\n",
      "19200: acc=0.7455729166666667\n",
      "22400: acc=0.7529464285714286\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(35.53972978646833, 0.7575175943698017)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)\n",
    "train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the reasons we are not seeing significant increase in accuracy is due to the fact that some words from our dataset are missing in the pre-trained GloVe vocabulary, and thus they are essentially ignored. To overcome this fact, we can train our own embeddings on our dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contextual Embeddings\n",
    "\n",
    "One key limitation of tradition pretrained embedding representaitons such as Word2Vec is the problem of word sense disambigioution. While pretrained embeddings can capture some of the meaning of words in context, every possible meaning of a word is encoded into the same embedding. This can cause problems in downstream models, since many words such as the word 'play' have different meanings depending on the context they are used in.\n",
    "\n",
    "For example word 'play' in those two different sentences have quite different meaning:\n",
    "- I went to a **play** at the theature.\n",
    "- John wants to **play** with his friends.\n",
    "\n",
    "The pretrained embeddings above represent both of these meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models in the next unit.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
  },
  "kernelspec": {
   "display_name": "py37_pytorch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
