"
]
},
{
"cell_type": "markdown",
"id": "2",
"metadata": {},
"source": [
"\n",
"This notebook reproduces the experiments done in the paper that introduced the ComplEx algorithm: Complex Embeddings for Simple Link Prediction, Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier and Guillaume Bouchard, ICML 2016. http://jmlr.org/proceedings/papers/v48/trouillon16.pdf\n",
"\n",
"In table 2, the paper reports five metrics measured on the WN18 and FB15K datasets: \"raw\" MRR (mean reciprocal rank), \"filtered\" MRR and filtered Hits at {1, 3, 10}. This notebook measures all of these, as well as raw Hits at {1, 3, 10}."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "3",
"metadata": {
"nbsphinx": "hidden",
"tags": [
"CloudRunner"
]
},
"outputs": [],
"source": [
"# install StellarGraph if running on Google Colab\n",
"import sys\n",
"if 'google.colab' in sys.modules:\n",
" %pip install -q stellargraph[demos]==1.3.0b"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "4",
"metadata": {
"nbsphinx": "hidden",
"tags": [
"VersionCheck"
]
},
"outputs": [],
"source": [
"# verify that we're using the correct version of StellarGraph for this notebook\n",
"import stellargraph as sg\n",
"\n",
"try:\n",
" sg.utils.validate_notebook_version(\"1.3.0b\")\n",
"except AttributeError:\n",
" raise ValueError(\n",
" f\"This notebook requires StellarGraph version 1.3.0b, but a different version {sg.__version__} is installed. Please see .\"\n",
" ) from None"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "5",
"metadata": {},
"outputs": [],
"source": [
"from stellargraph import datasets, utils\n",
"from tensorflow.keras import callbacks, optimizers, losses, metrics, regularizers, Model\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from stellargraph.mapper import KGTripleGenerator\n",
"from stellargraph.layer import ComplEx\n",
"\n",
"from IPython.display import HTML"
]
},
{
"cell_type": "markdown",
"id": "6",
"metadata": {},
"source": [
"## Initialisation\n",
"\n",
"We need to set up our model parameters, like the number of epochs to train for, and the dimension of the embedding vectors we compute for each node and for each edge type.\n",
"\n",
"The evaluation is performed in three steps:\n",
"\n",
"1. Load the data\n",
"2. Train a model\n",
"3. Evaluate the model\n",
"\n",
"The paper says that it used:\n",
"- the AdaGrad optimiser for 1000 epochs with an early stopping criterion evaluated every 50 epochs, but we've found using the Adam optimiser allows for much fewer epochs\n",
"- an embedding dimension of 150 or 200, since they had close results\n",
"- 10 negative samples (corrupted edges) per positive edge, which gives noticeably improved performance on FB15k compared to using 1, and but not for WN18 (the paper evaluated 1, 2, 5 and 10 negative samples)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7",
"metadata": {
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"epochs = 50\n",
"embedding_dimension = 200\n",
"negative_samples = 10"
]
},
{
"cell_type": "markdown",
"id": "8",
"metadata": {},
"source": [
"## WN18\n",
"\n",
"The paper uses the WN18 and FB15k datasets for validation. These datasets are not good for evaluating algorithms because they contain \"inverse relations\", where `(s, r1, o)` implies `(o, r2, s)` for a pair of relation types `r1` and `r2` (for instance, `_hyponym` (\"is more specific than\") and `_hypernym` (\"is more general than\") in WN18), however, they work fine to demonstrate StellarGraph's functionality, and are appropriate to compare against the published results.\n",
"\n",
"### Load the data\n",
"\n",
"The dataset comes with a defined train, test and validation split, each consisting of subject, relation, object triples. We can load a `StellarGraph` object with all of the triples, as well as the individual splits as Pandas DataFrames, using the `load` method of the `WN18` dataset."
]
},
{
"cell_type": "markdown",
"id": "9",
"metadata": {
"tags": [
"DataLoadingLinks"
]
},
"source": [
"(See [the \"Loading from Pandas\" demo](../basics/loading-pandas.ipynb) for details on how data can be loaded.)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "10",
"metadata": {
"tags": [
"DataLoading"
]
},
"outputs": [
{
"data": {
"text/html": [
"The WN18 dataset consists of triplets from WordNet 3.0 (http://wordnet.princeton.edu). There are 40,943 synsets and 18 relation types among them. The training set contains 141442 triplets, the validation set 5000 and the test set 5000. Antoine Bordes, Xavier Glorot, Jason Weston and Yoshua Bengio “A Semantic Matching Energy Function for Learning with Multi-relational Data” (2014).\n",
"\n",
"Note: this dataset contains many inverse relations, and so should only be used to compare against published results. Prefer WN18RR. See: Kristina Toutanova and Danqi Chen “Observed versus latent features for knowledge base and text inference” (2015), and Dettmers, Tim, Pasquale Minervini, Pontus Stenetorp and Sebastian Riedel “Convolutional 2D Knowledge Graph Embeddings” (2017)."
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"wn18 = datasets.WN18()\n",
"display(HTML(wn18.description))\n",
"wn18_graph, wn18_train, wn18_test, wn18_valid = wn18.load()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "11",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"StellarDiGraph: Directed multigraph\n",
" Nodes: 40943, Edges: 151442\n",
"\n",
" Node types:\n",
" default: [40943]\n",
" Features: none\n",
" Edge types: default-_also_see->default, default-_derivationally_related_form->default, default-_has_part->default, default-_hypernym->default, default-_hyponym->default, ... (13 more)\n",
"\n",
" Edge types:\n",
" default-_hyponym->default: [37221]\n",
" default-_hypernym->default: [37221]\n",
" default-_derivationally_related_form->default: [31867]\n",
" default-_member_meronym->default: [7928]\n",
" default-_member_holonym->default: [7928]\n",
" default-_part_of->default: [5148]\n",
" default-_has_part->default: [5142]\n",
" default-_member_of_domain_topic->default: [3341]\n",
" default-_synset_domain_topic_of->default: [3335]\n",
" default-_instance_hyponym->default: [3150]\n",
" default-_instance_hypernym->default: [3150]\n",
" default-_also_see->default: [1396]\n",
" default-_verb_group->default: [1220]\n",
" default-_member_of_domain_region->default: [983]\n",
" default-_synset_domain_region_of->default: [982]\n",
" default-_member_of_domain_usage->default: [675]\n",
" default-_synset_domain_usage_of->default: [669]\n",
" default-_similar_to->default: [86]\n"
]
}
],
"source": [
"print(wn18_graph.info())"
]
},
{
"cell_type": "markdown",
"id": "12",
"metadata": {},
"source": [
"### Train a model\n",
"\n",
"The ComplEx algorithm consists of some embedding layers and a scoring layer, but the `ComplEx` object means these details are invisible to us. The `ComplEx` model consumes \"knowledge-graph triples\", which can be produced in the appropriate format using `KGTripleGenerator`."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "13",
"metadata": {},
"outputs": [],
"source": [
"wn18_gen = KGTripleGenerator(\n",
" wn18_graph, batch_size=len(wn18_train) // 100 # ~100 batches per epoch\n",
")\n",
"\n",
"wn18_complex = ComplEx(\n",
" wn18_gen,\n",
" embedding_dimension=embedding_dimension,\n",
" embeddings_regularizer=regularizers.l2(1e-7),\n",
")\n",
"\n",
"wn18_inp, wn18_out = wn18_complex.in_out_tensors()\n",
"\n",
"wn18_model = Model(inputs=wn18_inp, outputs=wn18_out)\n",
"\n",
"wn18_model.compile(\n",
" optimizer=optimizers.Adam(lr=0.001),\n",
" loss=losses.BinaryCrossentropy(from_logits=True),\n",
" metrics=[metrics.BinaryAccuracy(threshold=0.0)],\n",
")"
]
},
{
"cell_type": "markdown",
"id": "14",
"metadata": {},
"source": [
"Inputs for training are produced by calling the `KGTripleGenerator.flow` method, this takes a dataframe with `source`, `label` and `target` columns, where each row is a true edge in the knowledge graph. The `negative_samples` parameter controls how many random edges are created for each positive edge to use as negative examples for training."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "15",
"metadata": {},
"outputs": [],
"source": [
"wn18_train_gen = wn18_gen.flow(\n",
" wn18_train, negative_samples=negative_samples, shuffle=True\n",
")\n",
"wn18_valid_gen = wn18_gen.flow(wn18_valid, negative_samples=negative_samples)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "16",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train for 101 steps, validate for 4 steps\n",
"Epoch 1/50\n",
"101/101 [==============================] - 24s 240ms/step - loss: 0.6971 - binary_accuracy: 0.5005 - val_loss: 0.6970 - val_binary_accuracy: 0.5013\n",
"Epoch 2/50\n",
"101/101 [==============================] - 23s 230ms/step - loss: 0.6967 - binary_accuracy: 0.5067 - val_loss: 0.6966 - val_binary_accuracy: 0.4989\n",
"Epoch 3/50\n",
"101/101 [==============================] - 23s 230ms/step - loss: 0.6963 - binary_accuracy: 0.5180 - val_loss: 0.6961 - val_binary_accuracy: 0.5077\n",
"Epoch 4/50\n",
"101/101 [==============================] - 23s 231ms/step - loss: 0.6958 - binary_accuracy: 0.5308 - val_loss: 0.6957 - val_binary_accuracy: 0.5162\n",
"Epoch 5/50\n",
"101/101 [==============================] - 24s 242ms/step - loss: 0.6953 - binary_accuracy: 0.5430 - val_loss: 0.6953 - val_binary_accuracy: 0.5318\n",
"Epoch 6/50\n",
"101/101 [==============================] - 24s 236ms/step - loss: 0.6942 - binary_accuracy: 0.5969 - val_loss: 0.6930 - val_binary_accuracy: 0.7030\n",
"Epoch 7/50\n",
"101/101 [==============================] - 24s 237ms/step - loss: 0.6784 - binary_accuracy: 0.8850 - val_loss: 0.6488 - val_binary_accuracy: 0.9090\n",
"Epoch 8/50\n",
"101/101 [==============================] - 24s 242ms/step - loss: 0.5640 - binary_accuracy: 0.9092 - val_loss: 0.4654 - val_binary_accuracy: 0.9091\n",
"Epoch 9/50\n",
"101/101 [==============================] - 24s 239ms/step - loss: 0.3702 - binary_accuracy: 0.9107 - val_loss: 0.3183 - val_binary_accuracy: 0.9108\n",
"Epoch 10/50\n",
"101/101 [==============================] - 24s 241ms/step - loss: 0.2636 - binary_accuracy: 0.9190 - val_loss: 0.2561 - val_binary_accuracy: 0.9176\n",
"Epoch 11/50\n",
"101/101 [==============================] - 24s 237ms/step - loss: 0.1915 - binary_accuracy: 0.9382 - val_loss: 0.2023 - val_binary_accuracy: 0.9316\n",
"Epoch 12/50\n",
"101/101 [==============================] - 24s 236ms/step - loss: 0.1316 - binary_accuracy: 0.9671 - val_loss: 0.1555 - val_binary_accuracy: 0.9520\n",
"Epoch 13/50\n",
"101/101 [==============================] - 24s 234ms/step - loss: 0.0928 - binary_accuracy: 0.9854 - val_loss: 0.1226 - val_binary_accuracy: 0.9695\n",
"Epoch 14/50\n",
"101/101 [==============================] - 24s 235ms/step - loss: 0.0711 - binary_accuracy: 0.9939 - val_loss: 0.1019 - val_binary_accuracy: 0.9817\n",
"Epoch 15/50\n",
"101/101 [==============================] - 24s 235ms/step - loss: 0.0597 - binary_accuracy: 0.9971 - val_loss: 0.0897 - val_binary_accuracy: 0.9882\n",
"Epoch 16/50\n",
"101/101 [==============================] - 23s 229ms/step - loss: 0.0535 - binary_accuracy: 0.9982 - val_loss: 0.0825 - val_binary_accuracy: 0.9917\n",
"Epoch 17/50\n",
"101/101 [==============================] - 24s 236ms/step - loss: 0.0496 - binary_accuracy: 0.9988 - val_loss: 0.0784 - val_binary_accuracy: 0.9927\n",
"Epoch 18/50\n",
"101/101 [==============================] - 24s 240ms/step - loss: 0.0472 - binary_accuracy: 0.9991 - val_loss: 0.0754 - val_binary_accuracy: 0.9936\n",
"Epoch 19/50\n",
"101/101 [==============================] - 24s 236ms/step - loss: 0.0454 - binary_accuracy: 0.9993 - val_loss: 0.0741 - val_binary_accuracy: 0.9936\n",
"Epoch 20/50\n",
"101/101 [==============================] - 24s 235ms/step - loss: 0.0444 - binary_accuracy: 0.9993 - val_loss: 0.0728 - val_binary_accuracy: 0.9938\n",
"Epoch 21/50\n",
"101/101 [==============================] - 24s 235ms/step - loss: 0.0434 - binary_accuracy: 0.9994 - val_loss: 0.0717 - val_binary_accuracy: 0.9939\n",
"Epoch 22/50\n",
"101/101 [==============================] - 24s 237ms/step - loss: 0.0427 - binary_accuracy: 0.9994 - val_loss: 0.0702 - val_binary_accuracy: 0.9941\n",
"Epoch 23/50\n",
"101/101 [==============================] - 24s 238ms/step - loss: 0.0420 - binary_accuracy: 0.9994 - val_loss: 0.0697 - val_binary_accuracy: 0.9940\n",
"Epoch 24/50\n",
"101/101 [==============================] - 24s 237ms/step - loss: 0.0414 - binary_accuracy: 0.9994 - val_loss: 0.0690 - val_binary_accuracy: 0.9941\n",
"Epoch 25/50\n",
"101/101 [==============================] - 24s 237ms/step - loss: 0.0408 - binary_accuracy: 0.9994 - val_loss: 0.0693 - val_binary_accuracy: 0.9938\n",
"Epoch 26/50\n",
"101/101 [==============================] - 24s 238ms/step - loss: 0.0404 - binary_accuracy: 0.9994 - val_loss: 0.0684 - val_binary_accuracy: 0.9938\n",
"Epoch 27/50\n",
"101/101 [==============================] - 24s 234ms/step - loss: 0.0398 - binary_accuracy: 0.9995 - val_loss: 0.0680 - val_binary_accuracy: 0.9938\n",
"Epoch 28/50\n",
"101/101 [==============================] - 24s 237ms/step - loss: 0.0394 - binary_accuracy: 0.9994 - val_loss: 0.0667 - val_binary_accuracy: 0.9941\n",
"Epoch 29/50\n",
"101/101 [==============================] - 24s 236ms/step - loss: 0.0390 - binary_accuracy: 0.9994 - val_loss: 0.0666 - val_binary_accuracy: 0.9940\n",
"Epoch 30/50\n",
"101/101 [==============================] - 24s 239ms/step - loss: 0.0385 - binary_accuracy: 0.9994 - val_loss: 0.0661 - val_binary_accuracy: 0.9940\n",
"Epoch 31/50\n",
"101/101 [==============================] - 24s 238ms/step - loss: 0.0381 - binary_accuracy: 0.9994 - val_loss: 0.0659 - val_binary_accuracy: 0.9941\n",
"Epoch 32/50\n",
"101/101 [==============================] - 24s 235ms/step - loss: 0.0377 - binary_accuracy: 0.9994 - val_loss: 0.0657 - val_binary_accuracy: 0.9939\n",
"Epoch 33/50\n",
"101/101 [==============================] - 24s 235ms/step - loss: 0.0373 - binary_accuracy: 0.9994 - val_loss: 0.0642 - val_binary_accuracy: 0.9941\n",
"Epoch 34/50\n",
"101/101 [==============================] - 24s 238ms/step - loss: 0.0369 - binary_accuracy: 0.9994 - val_loss: 0.0640 - val_binary_accuracy: 0.9940\n",
"Epoch 35/50\n",
"101/101 [==============================] - 24s 234ms/step - loss: 0.0365 - binary_accuracy: 0.9994 - val_loss: 0.0638 - val_binary_accuracy: 0.9940\n",
"Epoch 36/50\n",
"101/101 [==============================] - 24s 239ms/step - loss: 0.0362 - binary_accuracy: 0.9994 - val_loss: 0.0639 - val_binary_accuracy: 0.9938\n",
"Epoch 37/50\n",
"101/101 [==============================] - 25s 246ms/step - loss: 0.0359 - binary_accuracy: 0.9994 - val_loss: 0.0629 - val_binary_accuracy: 0.9941\n",
"Epoch 38/50\n",
"101/101 [==============================] - 24s 237ms/step - loss: 0.0355 - binary_accuracy: 0.9994 - val_loss: 0.0622 - val_binary_accuracy: 0.9940\n",
"Epoch 39/50\n",
"101/101 [==============================] - 24s 233ms/step - loss: 0.0351 - binary_accuracy: 0.9994 - val_loss: 0.0624 - val_binary_accuracy: 0.9939\n",
"Epoch 40/50\n",
"101/101 [==============================] - 24s 234ms/step - loss: 0.0347 - binary_accuracy: 0.9994 - val_loss: 0.0610 - val_binary_accuracy: 0.9941\n",
"Epoch 41/50\n",
"101/101 [==============================] - 23s 231ms/step - loss: 0.0344 - binary_accuracy: 0.9994 - val_loss: 0.0617 - val_binary_accuracy: 0.9941\n",
"Epoch 42/50\n",
"101/101 [==============================] - 23s 229ms/step - loss: 0.0340 - binary_accuracy: 0.9994 - val_loss: 0.0608 - val_binary_accuracy: 0.9941\n",
"Epoch 43/50\n",
"101/101 [==============================] - 23s 232ms/step - loss: 0.0335 - binary_accuracy: 0.9994 - val_loss: 0.0599 - val_binary_accuracy: 0.9943\n",
"Epoch 44/50\n",
"101/101 [==============================] - 23s 231ms/step - loss: 0.0331 - binary_accuracy: 0.9995 - val_loss: 0.0610 - val_binary_accuracy: 0.9938\n",
"Epoch 45/50\n",
"101/101 [==============================] - 23s 232ms/step - loss: 0.0329 - binary_accuracy: 0.9994 - val_loss: 0.0601 - val_binary_accuracy: 0.9940\n",
"Epoch 46/50\n",
"101/101 [==============================] - 23s 232ms/step - loss: 0.0326 - binary_accuracy: 0.9994 - val_loss: 0.0596 - val_binary_accuracy: 0.9940\n",
"Epoch 47/50\n",
"101/101 [==============================] - 23s 230ms/step - loss: 0.0324 - binary_accuracy: 0.9994 - val_loss: 0.0596 - val_binary_accuracy: 0.9939\n",
"Epoch 48/50\n",
"101/101 [==============================] - 24s 238ms/step - loss: 0.0320 - binary_accuracy: 0.9994 - val_loss: 0.0597 - val_binary_accuracy: 0.9939\n",
"Epoch 49/50\n",
"101/101 [==============================] - 24s 241ms/step - loss: 0.0317 - binary_accuracy: 0.9994 - val_loss: 0.0595 - val_binary_accuracy: 0.9938\n",
"Epoch 50/50\n",
"101/101 [==============================] - 24s 241ms/step - loss: 0.0315 - binary_accuracy: 0.9994 - val_loss: 0.0580 - val_binary_accuracy: 0.9942\n"
]
}
],
"source": [
"wn18_es = callbacks.EarlyStopping(monitor=\"val_loss\", patience=10)\n",
"wn18_history = wn18_model.fit(\n",
" wn18_train_gen, validation_data=wn18_valid_gen, epochs=epochs, callbacks=[wn18_es]\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "17",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAI4CAYAAACV/7uiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdeXxU5dn/8c81k4QsZMOENayKKAq4UNy31vq4ULVPq6LWqn0qPK1b1bZqrVRtbe1Ta0uttkVtXepStbZipT+r1tZK1YqioCAKCJKwJQgkBLLMzP3740zCZBggA5OZzJzv+/Wa15xz5p4zVw5hrtz3uRdzziEiIiLZJZDpAERERCR5SuAiIiJZSAlcREQkCymBi4iIZCElcBERkSyUl+kA9lRVVZUbMWJEpsMQERHpEW+++WaDc646/njWJ/ARI0Ywd+7cTIchIiLSI8xsRaLjakIXERHJQkrgIiIiWUgJXEREJAspgYuIiGQhJXAREZEspAQuIiKShdKWwM3st2a2zsze3cHrZma/MLMlZjbfzA5JV2wiIiLZJp018PuBk3fy+inA6OhjKvCrNMQkIiKSldKWwJ1zLwOf7KTIGcCDzvMaUGFmg9ITnYiISHbpTTOxDQFWxuzXRo+tzkw4kktC4QhNLSEaW9q9563tbG4N0RKK0NIeprU9TEu7t90S8rZbQ2HCEUco7AhFoo9whFDEEY442sMRnINwxBFxHQ+854i37XAAOLctlo5t17nviLetzPavJS4Xe2zb3nbvdjvdTRhLBzPznjsPdD2Ri77fkTh+i74hehos0TkTSCbG+Fh3eM64c+z6jN3X8ck7vV7b7+5SKmPsjtj4Yq9nx7XrjCf2dzuuTNzLCX8vdvXPGf9Pmej3KDZG61o40WYXu/o/svPgtj+3mXHrmQdy2Ki9kjjR7ulNCbzbzGwqXjM7w4YNy3A00hu0tIdZsm4z769p4v3Vjby/pom1jS2dCXtLW7jb5woGjMK8AAV5AfKCAfIDRjBo5AcCBANGMGDkB7dtBwwCZuQFAgQC3rb38LY7WIJvlu2/hGJL2HbHEtl2jsTfVvFvj09u27++/Wfs7I8Ox7ZkbNH3xyfn+Pd1TfK71p0Y42ONj2+7c+5hQk342XEx7OiPtIRJoxsBpCLG7nAJdhxu2+/YDpJn12Nsdyy27I4S7Y4DSfx75O1vXzzhHxE7uM67+j+SMLT4P2Ridkr6pCe19qYEXgcMjdmviR7bjnNuJjATYOLEien+w1QyzDnHq0vXM2/lRhZFk/VHDc2EI96vQp+8AGMGlrJ3dV/KivIoK8yntDCfsqI877nQe+7bJ4/C/ACF+UH6RJ+L8oPkB3dwZ8k5CLVCWzO0bYb2LdFvDrftdW9j276LbHtEwtHtcNx+9ByxZbd7uJjyMefo+EwzwBI/d54/0TNgAQgEIZAHFsT7K6RjO+i97hXs+lkxh7q+FrfvHERCEG73niPt3s8Sbt+23RFHbNwW6PqzxIr/gu1yDaPPsftmMT9PzM/VsW8Wvb7hmGsd7npsV9c54fWJuR5mcZ/fEUPA23aRmOvUDuHoterY704Mndcs9loGtsXR+XsXjnuO/l5tZxfXPd6ufocBgvkQ7BN9Log+8iGvj/d751zM73fH/52Y7djf3e3+78XEmfBaWNzPsIN/JwvEPYIx58T7Nwm3QrgNQm3ec8cj1ApFlUD5zq9VCvSmBD4LuMzMHgMOAzY559R8Lp1C4QjPLljNr/6xlPfXNAEwtF8R+w0s49QDB7LfoDLGDCxlxF4lBAM7+aIJtUH9Itj4MbRsij4at223Nm57bmuOeWzewZeciEiMAQdCv5E9/jFpS+Bm9ihwPFBlZrXA94B8AOfcr4HZwKnAEmALcHG6YpPerTUU5qm36vj1P5eyYv0W9unfl5+eNYGTDhhAaWH+Lt7cBGvehTXzYfV8WPMOrHvfq9HE61MGheXbnvsOhD59oaAECvpCfvG27YISyC/0/jKHxDVPiPnrPaam1Vnzi/vLPv4v/47aQ2fZuEcgGP28HdWu3bba5w5rjWxfK+vcDkVrx65rG2Rs7adLzWcHLREAgXwI5nk1rEC+V+MKBL3tQHDb+7rUsGJq0F3Et6s6Eta4YmvwHbW6zp8tsn0NO7523qWGHtj1dU54Tej6sySq9XYcs0DMdeq4RnnR5/xuxhDfChF3XTtaWLZrjej4PbS4uHfDjn5fLeCds7O22h59bo3Zbo+79vHniKkF76jVJ+G1iNnf7ueL+73t0noTV/vveK2jBSGvT0yLQgHkdbQo9Nm9a5ektCVw59y5u3jdAZemKRzJAs2tIR79z8fc869lrG1sZXxNOb/+0qGcNHYAgR3VsCMRWPEKzP8DrHgVPllG53/Q4r1g4Hg44uvec9VoKKyAwjIvaXckEhHpOcE8oDjTUeSE3tSELgJAezjCr/+xlN/O+YgNW9o5YtRe3H7WBI7ep2rHnUvqP4D5j8H8x2HTSigohVHHwfhzYNB4L2GXDd71PTwRkSyhBC69Sigc4crH5jF7wRpO3L8/Xzt+Hw4dXpm4cPN6ePePXuKue9NrWtv703DiTTDmVCjQX/kikruUwKXXCEccVz/+DrMXrOG7p+3PV48Zlbhg4yp47juw6BnvPu2AcXDSrTDui1A6ML1Bi4hkiBK49AqRiONbT77DrHdWce3J+yVO3s55TeR//ZbX2eWw/4UJU2DguPQHLCKSYUrgknGRiOM7f1rAU2/VcfVn9+Vrx++9faHN9fCXb8D7f4Ghh8OZd8NeCcqJiPiEErhklHOO6bPe5bE3VnL5p/fhis+M3r7Qwlle8m5tgs9+H464VD3GRcT3lMAlY5xz3PKXhfz+tY+Zdtworv7svl0LbN0As78NCx6HQRPg87+B/vtnJlgRkV5GCVwywjnHj/76Pr+bs5yvHDWS607er+sQsQ9fgFmXQXM9HH89HHONN2GCiIgASuCSAc45bv/bYma+vIwLDh/OjZP375q8X/s1/L9roXo/OPdRGHxw5oIVEemllMAl7R6fu5K7XlrKuZOGcvPpB3RN3v/+JfztBthvMnzhPm+6UhER2c4Oll0S6RkNm1u59dlFHDayH7eeOa7rlKiv/MxL3mPPhLPuV/IWEdkJ1cAlrX747CK2toe59fMHdk3eL/8E/v4DOPAL8PmZ0fmSRURkR1QDl7T599IGnppXx7Rj92af/qXeQefgH7d5yXv8OUreIiLdpG9KSYvWUJjv/vldhvUr5rJP7+MddA5eutWrfR90Ppx+p8Z3i4h0kxK4pMXMfy5jWX0z91/8KQrzg17yfvFm7773IV+GyTO8dYpFRKRblMClxy1vaObOl5Zw2vhBHD+mv5e8//ZdePWXMPErcOpPlbxFRJKkb03pUc45bnz6XQqCAaZPHusdfOlWL3lPmgqn3aHkLSKyG/TNKT3q2QWr+deHDXzzpH0ZUFYIb97v3fM++AI45f8gdgy4iIh0mxK49JjGlnZueWYh44aUc8ERI+DD5+EvV8M+J8Lknyl5i4jsAd0Dlx5zx98+oH5zK/deOJHgmnfg8QthwAHeJC2a11xEZI+oBi49YkHtJh58dTlfPnw44/s2wiNnQ3E/OP8J6FOa6fBERLKeauCScuGI44Y/L2Cvvn245rgB8PvToL0FvjwLSgdmOjwRkZygBC4pd++/ljG/dhN3nX0AZX+6CDZ8BF96Cvrvl+nQRERyhhK4pNSf59Xxo7++zylj+3Pqsu/Dile8VcVGHpPp0EREcorugUvKvLhoLdc88Q6Hj+rHnQP/gr37JJx4E4z7YqZDExHJOUrgkhKvLVvP1x9+iwMGlfLAvnPI+/fPYeL/wFHfyHRoIiI5SU3ossferdvEVx+Yy+hK44mqe+jzj6fhgM9rohYRkR6kBC57ZMm6zXz5t/9hXJ91PJj/C/I/WOI1mx/1DSVvEZEepAQuu61u41a+fN/rnOD+w0/c3QRa+sAFf4JRx2c6NBGRnKcELrulYXMrF97zby5ufZBL+DMMPBjOfggqhmY6NBERX1ACl6Q1trRz+b3Pc/Pm2zjKFsAhF3r3u/MLMx2aiIhvKIFLt7WHI8yaV8fLL87i9i13MCjYCJPvhEO+nOnQRER8RwlcdmlLW4in57xD/ZwH+a+2F/hCoJaWksEEzn8OhhyS6fBERHxJCVx2aEPTFl6e/Qh9F/2BL7o3ybcwm6on4A6/hsJxX4TCskyHKCLiW0rg0smFQ2zZuJb1q5ax8pVH2XfNs5xhG2kMVLB+/68w8LivUq75zEVEegUl8FzT3gIL/wztW8A5cJGY5wjhSJi3VzTQsrGe/JZ6ClvX0ze0norwBspdIyXmKAEGuwDvlx5B29FfYcinzqBM63eLiPQqSuC5ZvGz8KdpO3w5CBwKtLp8PglU0BSsZFPBINYVjiNUXI0r6U+wdAAjD/kMBw4ZnrawRUQkOWlN4GZ2MjADL4/c65y7Le714cBvgWrgE+BLzrnadMaY9ZrWeM+X/gcKK8AC0Yfx6rINTHv4LT43oYZbzzmcQWYMymy0IiKym9K2mImZBYG7gFOAscC5ZjY2rtjtwIPOufHALcCP0hVfzmiuh0A+VO0LpQOgbzWU7EV9uITL//wR/av7c8N/f0rTnIqIZLl0rkY2CVjinFvmnGsDHgPOiCszFvh7dPulBK/Lrmyuh5LqLgk6EnFc/fjbNLW088vzDqa4QHdORESyXToT+BBgZcx+bfRYrHeA/45ufx4oNbO94k9kZlPNbK6Zza2vr++RYLNWcz2UVHU59JuXl/GvDxuY/rmx7DdQQ79ERHJBb1sP/JvAcWY2DzgOqAPC8YWcczOdcxOdcxOrq6vTHWPv1hytgUe9uWIDt/9tMaeNG8R5k4ZlMDAREUmldLal1gGxK13URI91cs6tIloDN7O+wBeccxvTFmEuaG7w7n8Dm7a0c8Wj8xhcUciPvjAO031vEZGckc4a+BvAaDMbaWYFwBRgVmwBM6sys46YrsfrkS7d5VxnE7pzjm//8R3WNrZw57mHUFaocdwiIrkkbQncORcCLgOeAxYBjzvn3jOzW8zs9Gix44HFZvYBMAC4NV3x5YS2zRDaCiXVPPTaCp57by3fPnkMBw2tyHRkIiKSYmntjuycmw3Mjjs2PWb7SeDJdMaUU5q9Dn21ob784K+LOH5MNV89elSGgxIRkZ7Q2zqxyZ5obgDgl69torIkn5+eNYFAQPe9RURykRJ4LonWwN/dVMBt/z2evfr2yXBAIiLSU5TAc0k0ga935RwwROO9RURymRJ4Lokm8E8oVa9zEZEcpwSeSzbX0xLsi8srpDA/mOloRESkBymB55LmejYHK1T7FhHxAa1qkUua69kUqKCsj/5ZRURynWrguaS5gU+sXDVwEREfUALPJc31rHdllBUpgYuI5LqkEriZ3WNmE3sqGNkD4RBsWc/acBmlhWpCFxHJdcnWwPcGXjezeWb2NTPTYOPeYusngGNNWEPIRET8IKkE7pz7NDAG+H/AjcAqM/udmR3RE8FJEqJjwOva+1JWpBq4iEiuS/oeuHNuiXPuery1vb8EVAP/NLMFZnZZdB1vSbdoAl8bUg1cRMQP9qQTWwQIRZ8NaAeuAz42szNSEJskI7qQSQPqxCYi4gdJJ3AzG25mtwAfA48Aa4AjnHOHACOAnwF3pTJI6YZoDbzBlVOmTmwiIjkv2V7ozwFLgTOAHwFDnHNTnXNzAZxzIeA3wOBUByq7sHkdzvJopFg1cBERH0i2qrYKONo599pOytQDI3c/JNktzfW0FfbDbQ2oBi4i4gNJfdM75y7uRhkHrNjtiGT3NDfQUtAPQJ3YRER8INkm9BlmdmWC41eY2R2pC0uS1lzPlrxKADWhi4j4QLKd2L4AvJrg+KvAF/c8HNltzfU0BSsA1cBFRPwg2QRehXePO956oP+ehyO7rbmejcFK8oNGYb6muBcRyXXJftPXAolmXTsCr4ObZEJbM7Rv4RPKKCvMx8wyHZGIiPSwZLsrPwTcYWZbgeejx04Cfoo3fEwyIXYMuO5/i4j4QrIJ/AfAKOCPgIseM+Bh4JYUxiXJiM7Cti5cqpXIRER8ItlhZGHgQjP7PnBw9PBbzrmlKY9Mui9aA18T6ktZiWrgIiJ+sFvVNefcEmBJimOR3RVN4LXtpVRoJTIREV9I+tvezPYBzgKGAwWxrznnvpKiuCQZm9cBsLKliGEaQiYi4gtJJXAz+y/gaeB9YCzwDt498QDwRsqjk+5pboCCUuq3BtSJTUTEJ5IdRvZ94P+ccwcBrcA5wDDgZeCpFMcm3dVcjyupoqU9onnQRUR8ItkEvj/wYHQ7BBQ555qB7wHfTmVgkoTmekJFewGaRlVExC+STeBb2NbsvgZv/W/wkvmAFMUkyWpuoK1PFaBpVEVE/CLZ9tY3gUl498BfAn5oZjXA+cC8FMcm3dW8jq2VEwA0DlxExCeSrYHfwLalQqfjTa36E6AImJbCuKS7ImHYsp4t+VqJTETET7pdXTOzAF4T+gcAzrkG4LQeiku6a+sGcBGtRCYi4jPJ1MAdsAAY1EOxyO6ITuKy0aIJXBO5iIj4QrcTuHPOAUuByp4LR5IWTeDrKQdUAxcR8Ytk74F/D/g/MxvSE8HIbuhciayUYMAoLghmOCAREUmHZBP4rXi90FeYWZ2ZfRD72NWbzexkM1tsZkvM7LoErw8zs5fMbJ6ZzTezU5OMz382ewl8bbiMssI8rQUuIuITyd4w/f3ufpCZBYG7gM/i9V5/w8xmOecWxhT7LvC4c+5XZjYWmM22seaSSHM9WIC17UWUFrZnOhoREUmTZJcTvXkPPmsSsMQ5twzAzB4DzgBiE7gDyqLb5cCqPfg8f2iuh+IqGlsj6sAmIuIjyTah74khwMqY/drosVg3AV8ys1q82vfliU5kZlPNbK6Zza2vr++JWLNHcwOUVNO4tV0d2EREfCSpBG5mETML7+iRgnjOBe53ztUApwIPRcefd+Gcm+mcm+icm1hdXZ2Cj81izfXQt5rGFiVwERE/SbbN9ct4zdwd8oFD8dYH31Xzeh0wNGa/Jnos1v8AJwM45141s0KgCliXZJz+0bwOKj9FY21ITegiIj6S7D3wRJ3Y7jezd4ATgF/t5O1vAKPNbCRe4p4CnBdX5mPgM9Fz7g8UAj5vI9+FjiZ01cBFRHwlVffA/w58bmcFnHMh4DLgOWARXm/z98zsFjM7PVrsGuCS6B8EjwIXRSeQkUTatkDbZsJFe7GlLax50EVEfCRVba4nA5t2Vcg5Nxuvc1rssekx2wuBo1IUU+7b0gDA1oJ+gFYiExHxk6S+8c3sb/GHgMHAfnhjuCWdorOwbcn3Eria0EVE/CPZKlt8p7MIMBe4wjn3YmpCkm6LzsLWGKgENqsJXUTER5LtxHZxTwUiu6FjJbJAObCZMjWhi4j4RrLjwEea2b4Jjo82sxGpCkq6KZrAP3GlAKqBi4j4SLK90H9L4k5mRwL37Xk4kpTmBsgvYUPIS9xK4CIi/pFsAj8YmJPg+KvAIXsejiSlYxa2rSEANaGLiPhIsgk8H+iT4HgfoGDPw5GkNNd3TuISMCgpUAIXEfGLZBP4m8BFCY5/BXh7j6OR5EQTeFNLiL598ggEtBa4iIhfJFtluwX4f2a2D/B89NhJwCl4i49IOjXXw5BDvJXIdP9bRMRXkqqBO+dewJt1rRL4cfRRAZzqnHt+Z++VFItENA+6iIiPJX3TNJrEX+iBWCQZLRvBhaGkP41btRKZiIjfJDsO/FNmdliC44eZ2cTUhSW7FB0DTkmVauAiIj6UbCe2O4ERCY4PBX6xx9FI922OLpFeUq174CIiPpRsAj8Ab+7zeG9FX5N06ayBV9PYElINXETEZ5JN4BGgLMHxyt04l+yJZm8p0XBxFZtbQ1pKVETEZ5JNuv8Grklw/Bq82dgkXZrrwQJsNu/vKTWhi4j4S7LVtu8C/zSzeUDH8qGfAUYDx6cwLtmV5noo3ovGtgigaVRFRPwm2XHgbwKHAe/iTd5yCrAAONw5l+jeuPSU6Cxsm7a2A6qBi4j4ze6MA38PuKAHYpFkNNd3DiED1IlNRMRndrvd1cwGEreAiXPu4z2OSLqnuR4GH7xtJTJN5CIi4itJfeubWRkwA5hC4tXHgqkISrohZhpVUA1cRMRvku2F/mO8e+DnAi14K5PdCKwCzktpZLJj7S3Q2tg5iQsogYuI+E2y7a6nARc6514yswjwqnPuITOrxbsv/oeURyjb2+KNAaekmqZPvCb0vuqFLiLiK8nWwPcClka3G/EmcAH4F3BcqoKSXYidRrWlndI+eQS1FriIiK8km8BXADXR7SXA5Oj2CcDmVAUlu9C8rQburUSm5nMREb9JNoE/xbYJW2YAN5jZamBm9CHpELcSmaZRFRHxn6S++Z1z343ZfsrMjgSOBhY7555NdXCyAx0JvG9/GreuUQ1cRMSH9qjq5pz7D/Cf+ONm9izwVefc6j05v+xAcz3kF0NBCY0tIYZUFGU6IhERSbOeWkHsWEBZpac0N0BJFYC3Fria0EVEfEdLgGaj5nVQUg1AU0u7mtBFRHxICTwbRRcyiUQcTa0h1cBFRHxICTwbRadR3dwWwjmtRCYi4kdK4NnGuc4auKZRFRHxLyXwbNOyESKhzklcQCuRiYj4UU8l8H8BW3vo3P62uWMSF61EJiLiZ0klcDN73sy+aGY7rfI5507VGPAeEjsLW0cTuu6Bi4j4TrI18JXA74A6M/uxme3TAzHJzjTH1sC9JnRNpSoi4j9JJXDn3FeAQcB04NPAYjN7ycymmNkuq4FmdrKZLTazJWZ2XYLXf2Zmb0cfH5jZxmTi84WYaVSb1IQuIuJbSd8Dd85tds79xjn3KeAQ4D28WvkqM7vdzEYkep+ZBYG7gFOAscC5ZjY27txXOecOcs4dBNyJt3iKxGpuAAyK+nV2YlMNXETEf3a7E5uZlQPH4E2bGsTruHYM8IGZTUvwlknAEufcMudcG/AYcMZOPuJc4NHdjS9nNa32plEN5tHY0k5JQZC8oAYTiIj4TdLf/GZ2jJk9CKwCvgn8ARjmnPtv59xhwFTg1gRvHYJ3D71DbfRYos8YDowE/r6D16ea2Vwzm1tfX5/sj5DdmlZD6SAgOg+6OrCJiPhSsr3Q3wdeBMqBs4CRzrlbnXNrYoo9A/Tbw7imAE8658KJXnTOzXTOTXTOTayurt7Dj8oyTauhbDAAjS3tuv8tIuJTyd48fRS41zlXt6MCzrn1JP7DoA4YGrNfEz2WyBTg0iRj84fG1TDkUG9za0iTuIiI+FS3a+DRXubTgIrd/Kw3gNFmNtLMCvCS9KwEn7MfUAm8upufk7tCrbClAUq31cBLVQMXEfGlbidw51x7dDNhs3Y33h8CLgOeAxYBjzvn3jOzW8zs9JiiU4DHnHNudz4np21e6z2XeffAm1q0EpmIiF8l++1/L3AF8PXd+TDn3Gxgdtyx6XH7N+3OuX2hMTq5XUwNXJ3YRET8KdkEPhg4y8w+DbwJNMe+6JybmqrAJIGmVd5z6UCcc14vdDWhi4j4UrIJfG/grej24LjX1OTd05qinf3LBtPcFibitBKZiIhfJfXt75w7oacCkW5oXAXBPlBUSeOmFkDTqIqI+JWm8MomTauhdCCYbVtKVPfARUR8Ken2VzM7DjgPGA4UxL7mnPt0iuKSRBpjJnHRPOgiIr6W7ExsXwKeBwYAJwCNwEC8RU2Wpjw66SpmGlWtRCYi4m/JNqF/G7jKOXcm0AZcDRwAPIE3t7n0FOe6zoOuJnQREV9LNoHvzbZx3G1ASXTClZ/hLWIiPaVlE7Rv6ZzEpaMJXRO5iIj4U7IJfCNQEt1eDewb3S4BylIVlCTQMYQsZiUyQFOpioj4VLLVt9fw1v9+F/gL8DMzm4i3rvcrKY5NYnVO4rKtCb0oP0hBngYSiIj4UbIJ/Bqgb3T7ZqAUOA14D7gqhXFJvI5pVGOa0DWJi4iIfyU7kcvymO2taMnP9GnqmAd9Ww1czeciIv6l9tds0bQaCisgvwiILmSiDmwiIr6V7DjwajN7wMzqzCxkZuHYR08FKXSZxAWiS4lqCJmIiG/tznKi44AZwCq0gEn6xIwBB68X+oi9SnbyBhERyWXJJvDjgZOcc6/3QCyyM02rYcDYzt3GFnViExHxs2TvgW8AmnoiENmJcAg2r+2sgWstcBERSTaB3wrcYGaq+qVTcz24SGcC39oeJhRxugcuIuJjySbis4FPAbVm9j7edKqdnHMnpSowidExiUvcSmSqgYuI+FeyCbwWLVqSfh2TuJQO9HZbOqZRVUOIiIhfJTuRy8U9FYjsROckLl4NvEkrkYmI+J4mcskGTavBglBSDWglMhER6UYN3Mz+BpzlnNsU3d4h3QPvIY2rvebzgPf3ltYCFxGR7lTh6oBIzLakW9Oq7SZxAXViExHxs10m8Nj73h3bZlYE7B09vDS6sIn0lKY1UDW6c7exxWtCVyc2ERH/SnYu9AIz+ymwHngn+lhvZneYWZ+eCFCINqF3rYH3yQtQmB/MYFAiIpJJyVbh7gTOBK4E5kSPHQXcApQA01IXmgDQ1gytm7omcC0lKiLie8km8CnAec65Z2OOLTSzVcAjKIGnXtMa7zlmJTLNgy4iIskOI2sDliQ4vhRo3/NwZDuN0VnYopO4AJoHXUREkk7g9wJXm5l1HIhuXwHcl8rAJCpuEhfoqIErgYuI+Fl3xoHPjNkNAGcBnzWz/0SPfQrYC3gy9eFJZwIv23YPvGlrO0MrizIUkIiI9AbduZE6Om7/rejzgOjzx9HHqFQFJTEaV0NBKfQp3XaopV01cBERn+vOOPAT0hGI7EDTqi73v721wEO6By4i4nOaC723a1zdpfm8NRShLRzRJC4iIj6nBN7bNa3p2oFtq+ZBFxERJfDeLRLxOrHFDiFr0UpkIiKiBN67bVkPkfa4SVxUAxcRkTQncDM72cwWm9kSM7tuB2XONrOFZvaemT2Szvh6nc4x4FqJTEREukpbO6yZBYG7gM8CtcAbZjbLObcwpsxo4HrgKOfcBjPrn674eqVECWOd5vEAACAASURBVDzahF6uqVRFRHwtnTXwScAS59wy51wb8BhwRlyZS4C7nHMbAJxz69IYX+/TMY1qmWrgIiLSVToT+BBgZcx+bfRYrH2Bfc1sjpm9ZmYnJzqRmU01s7lmNre+vr6Hwu0FmtYABn0HdB7SPXAREYHe14ktD2/mt+OBc4F7zKwivpBzbqZzbqJzbmJ1dXWaQ0yjplVQUg3Bbcl69cYWCvMD9Mnrbf90IiKSTunMAnXA0Jj9muixWLXALOdcu3PuI+ADtp/K1T/iJnGJRBzPL1zLMaOriVlPRkREfCidCfwNYLSZjTSzAry1xWfFlfkzXu0bM6vCa1JflsYYe5e4SVzmrdzAmsYWThs3aCdvEhERP0hbV2bnXMjMLgOeA4LAb51z75nZLcBc59ys6GsnmdlCIAx8yzm3Pl0x9jpNq6BmYufus/PXUJAX4DP7+7tzvohfRCIRamtraW5uznQo0kNKSkqoqakhEEi+Pp3WsUjOudnA7Lhj02O2HXB19OFvoVZvIpfoJC6RiOOv767m2NHVlKoHuogvNDQ0YGaMGTNmt77gpXeLRCLU1dXR0NBA//7JV8z0G9FbxY0Bf7t2I6s3tXDquIE7eZOI5JKNGzcyYMAAJe8cFQgEGDBgAJs2bdq996c4HkmVpjXec7QT2+z5qykIBjhx7ICdvElEckk4HCY/Xy1uuSw/P59QKLRb71UC7606JnEpHYRzjr++u4ZjRldpAhcRn9GIk9y2J/++SuC9VUwT+tsrN1K3cSunqve5iIhEKYH3Vk2rIa8QiiqZvWA1+UFT87mIiHRSAu+tGr11wB0we8Eajt6ninJNnyoiPnT//feTl6cFnOIpgfdWTauhdDDzazep+VxEss6JJ57IRRddlJJznXPOOdTVxU/cKUrgvVWTN41qR/P5SWM1fExEcktbW1u3yhUVFTFggG4hxlMC742cg8bVuL4DeXbBao7ap4ryYjWfi0h2uOiii3jxxRd54IEHMDPMjPvvvx8z4+GHH+bUU0+lpKSEG2+8Eeccl1xyCXvvvTdFRUWMGjWK73znO7S2tnaeL74JvWN/zpw5HHLIIRQXF3PooYfyxhtvZOLHzRjdVOiNWjZCaCurXSW1G7Zyxaf9u56LiGxz8zPvsXBVY9o/d+zgMr73uQO6XX7GjBksW7aMQYMGMWPGDAAaG724r732Wn784x9z1113AeCco3///jzyyCMMGDCA+fPnM23aNPLz87n55pt3+BmRSITrr7+eGTNmUF1dzVVXXcXZZ5/Nhx9+6Jv75f74KbNNdBKX/6wvJC9gnHSAmo5EJHuUl5dTUFBAUVERAwd6t/9aWloAmDZtGueff36X8rfeemvn9ogRI1i6dCl33333ThO4c46f//znHHLIIQDcdNNNHH744SxdupQxY8ak+kfqlZTAe6PoJC4vrDSO3KeKiuKCDAckIr1BMrXg3mrSpEnbHbvnnnu49957Wb58Oc3NzYRCISKRyE7PY2ZMmDChc3/wYG/diLVr1/omgeseeG8UncTlncZiTtPc5yKSQ0pKSrrsP/HEE1x66aWcc845zJ49m3nz5jF9+nTa29t3ep5AIEAwGOzc75jRbFeJP5eoBt4bRRN4g/Xjs+p9LiJZqKCggHA4vMtyL7/8MgcffDBXX71tEcrly5f3YGS5QzXwXsg1rmYTpUzcexD9StR8LiLZZ+TIkbz55pssXbqUhoaGHdaox4wZw4IFC3j66adZunQpM2bM4KmnnkpztNlJCbwXaqr/mFWRCk3eIiJZ65prrqGqqooJEyZQXV3NnDlzEpabNm0aF1xwARdffDEHH3wwr7/+OjfddFN6g81S5pzLdAx7ZOLEiW7u3LmZDiOl1vzkMBY39WHctS+oBi7iY4sWLWL//ffPdBjSw3b172xmbzrnJsYfVw28l3HOkbdlLZG+aj4XEZEdUwLvZRbVbaAyspGqQcMzHYqIiPRiSuC9zItzFxA0x8hR+2Q6FBER6cWUwHuJxpZ2vvnEO7zw+tsA9K0aluGIRESkN9M48F7g30sa+NaT81m9aSt3jOsDHwClGv8tIiI7php4Bm1tC3PTrPc4797X6ZMX4I9fO5Iz947+k5QNzmxwIiLSq6kGniHzPt7ANY+/w7KGZi46cgTXnrwfRQVB+GA1BPKguCrTIYqISC+mBJ5mbaEIv3jxQ+7+xxIGlhXy8FcP46h9YpJ102roOxACahwREZEdUwJPoyXrmrji0bdZuLqRLx5aw/TPjaWsMH9bgQ/+BgtnwbDDMxekiIhkBVXz0sA5x8Ovr2Dyna+wetNWfnPBodx+1oSuyXvu7+DRKVC1D5x5d+aCFRHpBe6//37y8rbVMf/xj39gZtTW1u70fWbG73//+z3+/IsuuogTTzxxj8/Tk1QD72GfNLdx7R/n8/zCtRwzuoqfnjWB/mWF2wpEIvD378Mrd8Dok+CLv4M+fTMXsIhIL3TkkUeyevVq+vfvn9Lz/v73v+eCCy4gflrxGTNm9PqlSZXAe9CcJQ1c/fjbfNLcxndP25+vHDWSQMC2FQi1wp+/Du8+CYdeDKfeDkH9k4iIxCsoKGDgwPQNry0vL0/bZ+0uNaH3gLZQhB/NXsSX7nudvn3y+NPXj+Krx4zqmry3fAIPfd5L3ifeBJN/puQtIjnhnnvuoby8nJaWli7Hf/zjHzNs2DDC4TCXXHIJe++9N0VFRYwaNYrvfOc7tLa27vCciZrQX3rpJcaPH09hYSHjx4/npZde2u59N9xwA/vvvz/FxcUMHTqU//3f/2XTpk2d57zgggsAr+ndzLjooouA7ZvQnXPcfvvtjBo1ioKCAvbee29+/vOfd/msESNGMH36dK688kr69evHgAEDuOqqqwiFQsldwG5SxkixpfWb+cZjb7OgbhPnHTaMG08b6w0Pi7VhOTx8lvf8hftg3BczEaqIZJu/XgdrFqT/cweOg1Nu63bxs88+myuuuIKnn36ac845p/P4gw8+yJe+9CXMjP79+/PII48wYMAA5s+fz7Rp08jPz+fmm2/u1mesWrWKyZMnc/bZZ/PYY49RV1fHlVdeuV25oqIiZs6cydChQ1m6dCmXXnopV1xxBQ888ABHHnkkv/zlL7nssstYvXp1Z/lE7r77bm688UZmzJjBCSecwIsvvsg3vvENSktL+Z//+Z/OcnfeeSfXXnstr7/+OvPmzeP888/nwAMP7FImVZTAUyQUjvDbOR/xs+c/pE9+gN9ccCj/dUCC5p66N+GRKRBuhQv+DCOOSn+wIiI9qLy8nDPOOIMHH3ywM4HPnTuXhQsX8tRTTxEIBLj11ls7y48YMYKlS5dy9913dzuB33333VRVVXHPPfeQl5fH2LFj+eEPf8jnPve5LuW++93vdvmcH/3oR0yZMoXf/e53FBQUdDaV76p5/rbbbuPyyy9n6tSpAIwePZrFixdz6623dknOxxxzDNddd11nmd/97ne88MILSuC91fzajVz/1ALeW9XIifv35wdnjmNgeWHXQmve9TqqvfcnKK+Bi/4C1WMyE7CIZKckasGZduGFF3L66aezbt06+vfvz4MPPsikSZMYM8b73rvnnnu49957Wb58Oc3NzYRCoaQ6jS1cuJBJkyZ16al+9NFHb1fuqaee4uc//zlLliyhsbGRSCRCW1sba9asYfDg7s142djYSG1tLccee2yX48cddxwzZsxgy5YtFBcXA3DQQQd1KTN48GA++uijbv9cydA98D3Q3BrilmcWcuZdc6hvauVX5x/CPV+e2DV5r/wPPHIO/Poo+OA5OPJyuOQfSt4iktNOOukkqqqqeOSRR2hvb+exxx7jwgsvBOCJJ57g0ksv5ZxzzmH27NnMmzeP6dOn097entIYXn/9dc466yyOPfZY/vSnP/HWW2/x61//GoC2traUflaHgoKCLvtm1mO92VUD300vLFzL9KffZXVjC+cfNoxvn7zftnHdzsGyl+Bfd8Dyf0FRPzjhBph0CRRVZjZwEZE0CAaDnH/++Tz00EOMGjWKTZs2MWXKFABefvllDj74YK6++urO8suXL0/q/GPHjuWhhx4iHA4TDHr9jObMmdOlzCuvvEJVVRU/+MEPOo89+eSTXcp0JNzY88QrKyujpqaGl19+mcmTJ3ce/+c//8nIkSM7a9/ppgSepHWNLdz0zHvMXrCGfQf05cnzjuDQ4f28FyMRWPws/OunsGoelA6C//ohHHoRFJRkNG4RkXT78pe/zE9/+lO+973vMXnyZPr1874rx4wZw3333cfTTz/NgQceyF/+8heeeuqppM79ta99jTvuuIOpU6fyzW9+k1WrVnHDDTd0KTNmzBjq6+u57777OOGEE3jllVe4++6uE2WNHDkSgFmzZnH00UdTVFRE377bz8Vx/fXXc8011zB69GiOP/54/v73v/OrX/2Ku+66K6m4U0kJvBsiEcfcFRuY9U4dT89bRWs4wjdP2pepx+5NQV4AImFY+DS8/BNYtxAqR8LnfgETpkBen0yHLyKSEePHj+eggw7i7bff5qabbuo8Pm3aNBYsWMDFF19MKBRi8uTJ3HTTTVx++eXdPveQIUN45pln+MY3vsFBBx3E6NGj+cUvfsFnPvOZzjKTJ0/mhhtu4Dvf+Q6bN2/muOOO4yc/+QnnnXdeZ5lPfepTXHnllUybNo36+nouvPBC7r///u0+72tf+xrNzc388Ic/5Otf/zpDhw7ltttu65HOad1l8bPP9OiHmZ0MzACCwL3OudviXr8I+AlQFz30S+fcvTs758SJE93cuXNTHqtzjnfrGpn1Th1/mb+a1ZtaKMwPcOL+A7jmpDGMrCrxEve7T3mJu2ExVO0Lx34LDvhvjekWkT22aNEi9t9//0yHIT1sV//OZvamc25i/PG0ZRkzCwJ3AZ8FaoE3zGyWc25hXNE/OOcuS1dcXax5lzVL5vHSumKeWBLkrU/yyAsEOHbfaq49eT8+O3YAJX3yIByCtx+Ff90O65dA/7HeFKhjz4BA4nsoIiIiqZTOauIkYIlzbhmAmT0GnAHEJ/CMeW32Axz+8UzOBc4F2kuKCPQbSbBwFKwbAe0jvA5qr93lTcIyYByc/RDsN1nLf4qISFqlM4EPAVbG7NcChyUo9wUzOxb4ALjKObcyvoCZTQWmAgwbNixlAfb99Df54/v/xacHbqGyZRX5G5bDho+8WvaSFyAUnRZw0EEw5VEYcwqY7fScIiIiPaG33ah9BnjUOddqZtOAB4BPxxdyzs0EZoJ3DzxVH37giAEcOGJA4hedg81rYesGqN5PiVtERDIqne2+dcDQmP0atnVWA8A5t9451zGb/b3AoWmKbdfMoHQg9N9fyVtE0iadHY0l/fbk3zedCfwNYLSZjTSzAmAKMCu2gJkNitk9HViUxvhERHqVYDCY8tnJpHdpb2/vMh1sMtKWwJ1zIeAy4Dm8xPy4c+49M7vFzE6PFrvCzN4zs3eAK4CL0hWfiEhvU1FRwdq1a3tsKk7JrEgkwtq1a3d77fG0jgPvCT01DlxEJNMikQi1tbU0NzdnOhTpISUlJdTU1BDYyUimjI8DFxGR5AQCgZSOtJHcosHLIiIiWUgJXEREJAspgYuIiGQhJXAREZEspAQuIiKShbJ+GJmZ1QMrUnjKKqAhhefzK13H1NB1TA1dx9TQdUyNZK/jcOdcdfzBrE/gqWZmcxONt5Pk6Dqmhq5jaug6poauY2qk6jqqCV1ERCQLKYGLiIhkISXw7c3MdAA5QtcxNXQdU0PXMTV0HVMjJddR98BFRESykGrgIiIiWUgJXEREJAspgYuIiGQhJXAREZEspAQuIiKShZTARUREspASuIiISBZSAhcREclCeZkOYE9VVVW5ESNGZDoMERGRHvHmm282JFqNLOsT+IgRI5g7d26mwxAREekRZpZwyWw1oYuIiGQhJXAREZEspAQuIiKShZTARUREspASuIiISBZSAhcREclCaU3gZnaymS02syVmdl2C139mZm9HHx+Y2cZ0xiciIpIt0jYO3MyCwF3AZ4Fa4A0zm+WcW9hRxjl3VUz5y4GD0xWfiIhINklnDXwSsMQ5t8w51wY8Bpyxk/LnAo+mJbKoUDhCeziCcy6dHysiIpK0dM7ENgRYGbNfCxyWqKCZDQdGAn/fwetTgakAw4YNS1mA/3n4Jg5e+itaCBImQJhg9BEgZHlECBI271jEvEeYPMKWRySQR8TycBYkEt2n4zmQh4t5tkAeBPMhkO89B/MIBPOxYAEE86BPGZT0x8oGEiwbRGFJGcUFQYoL8ijuE6SkII9gwFL2c4uISPbprVOpTgGedM6FE73onJsJzASYOHFiyqrLe42exKK2TyAcAheCSBiLeNsWCWMuhEVCmAsTiIQwFyIvEqLAhQi4LQQiYQIuRNCFCLiw90yYoPP+DMgjTJAQeS5MviX80RLa7ApZ5ypYQQX1roIF+eOY8PlrOHXcoFT96CIikmXSmcDrgKEx+zXRY4lMAS7t8YjijDliMhwxOT0f5hxEwoRDbbS3t9Le3ka4rY1QeyvtWxpp37SKSOMaXNNarHktBVvWMWLLOsZuXcHkltc499H+PLvgRL5/xoH0KylIT8wiItJrpDOBvwGMNrOReIl7CnBefCEz2w+oBF5NY2zpZwbBPILBPIJ9iincrsAhid/X3oK78xB+HvkTR793AK8tXc8PzjyQU1QbFxHxlbR1YnPOhYDLgOeARcDjzrn3zOwWMzs9pugU4DGnnmSJ5RdiJ9zAgM0LeenURgaWF/K1h9/i8kfn8UlzW6ajExGRNLFsz5MTJ050vltONBKGXx0FkXbap/2bu1/+mDv//iEVxfn84MxxnHzgwExHKCIiKWJmbzrnJsYf10xs2SgQhBO/B+uXkD//Ya48cTSzLjua/qWF/O/v3+SqP7xNOJLdf5iJiMjOKYFnq31PhmFHwD9ug7Zmxg4u4+nLjuKrR4/kT/PqmLv8k0xHKCIiPUgJPFuZwYk3w+a18NrdAOQHA0w7bm8A5tduymR0IiLSw5TAs9mww2DMaTDnF9C8HoDq0j4MLi/knVpNIy8iksuUwLPdZ6ZD22b41087D42vqVANXEQkxymBZ7v++8FB58Mb98CGFQCMH1rOx59sYeMWDSsTEclVSuC54PjrwQLw0g8BGD+kAtB9cBGRXKYEngvKh8Bh02D+H2DNu4yrKQdgvu6Di4jkLCXwXHH0VVBYBi/eTHlRPiOrSnhHNXARkZylBJ4riirh6Kvhw7/B8lcYX1OuGriISA5TAs8lh02D0sHw/PcYP6SctY2trG1syXRUIiLSA5TAc0l+ERxzNdTN5fC+awB4Z6Vq4SIiuUgJPNcMORSAffIaCAZMPdFFRHKUEniuqRwBQJ/NKxndvy/z65TARURykRJ4rimqhIJS2LCisyNbti8ZKyIi21MCzzVmUDkcNq5gfE0FG7e0s/KTrZmOSkREUkwJPBdVDIcNK5hQ483IpoVNRERyjxJ4LqocARtXMGZAXwqCAY0HFxHJQUrguahyOLRvoaB1PfsPLtOMbCIiOUgJPBdVDPeeN6xgQk0579ZtIhxRRzYRkVyiBJ6LKqMJPNqRbUtbmKX1mzMbk4iIpJQSeC6qGOY9b1jOhM6VydSMLiKSS5TAc1FBCZRUw8YVjKruS3FBUB3ZRERyjBJ4rooOJQsGjAOHlKsjm4hIjlECz1XRyVwAJtSUs2hVI22hSIaDEhGRVElrAjezk81ssZktMbPrdlDmbDNbaGbvmdkj6Ywvp1QMh021EAkzvqaCtnCExWuaMh2ViIikSNoSuJkFgbuAU4CxwLlmNjauzGjgeuAo59wBwDfSFV/OqRwBkRA01mlGNhGRHJTOGvgkYIlzbplzrg14DDgjrswlwF3OuQ0Azrl1aYwvt3QMJduwnKH9iqgszldHNhGRHJLOBD4EWBmzXxs9FmtfYF8zm2Nmr5nZyYlOZGZTzWyumc2tr6/voXCzXMxkLmbGuJoKDSUTEckhva0TWx4wGjgeOBe4x8wq4gs552Y65yY65yZWV1enOcQsUV4DFujSke2DtU1saQtlODAREUmFdCbwOmBozH5N9FisWmCWc67dOfcR8AFeQpdkBfOhrAY2eAl83JByIg4WrmrMcGAiIpIK6UzgbwCjzWykmRUAU4BZcWX+jFf7xsyq8JrUl6UxxtwSO5RsaEdHNjWji4jkgrQlcOdcCLgMeA5YBDzunHvPzG4xs9OjxZ4D1pvZQuAl4FvOufXpijHnRCdzARhQVsiAsj7qyCYikiPy0vlhzrnZwOy4Y9Njth1wdfQhe6pyOGxeA+1bIb+I8erIJiKSM3pbJzZJpcoR3vNGr/P/hJpyPmpoZtPW9szFJCIiKaEEnssqto0FBxgfndBlgWrhIiJZTwk8l8WsCw4wPrq0qGZkExHJfkrguazvAMgr7KyBVxQXMHyvYnVkExHJAUrgucwMKoZ11sDBa0ZXE7qISPZTAs91MUPJAMYPKWfVphbqm1ozGJSIiOwpJfBcFzOZC2y7D65mdBGR7KYEnusqhkPLJtjqJezRA0oBWL5+SyajEhGRPaQEnus6x4J7tfDK4nyK8oPUblACFxHJZkrgua6y61hwM2NIZRF1G7ZmLiYREdljSuC5LmZd8A41lUXUbVQCFxHJZkrgua6oAgrLu3RkG1KhBC4iku2UwP0gbijZkMoiNm5pZ3NrKINBiYjInlAC94O4oWQ1lcUAug8uIpLFlMD9oGI4bPwYnAO8JnSAuo3qiS4ikq2UwP2gcgSEWmDzWsDrxAZQqxq4iEjWUgL3g7ie6NV9+1AQDKgJXUQkiymB+0HHZC7RseCBgDG4opBa9UQXEclaSuB+UDHMe97YtSe6mtBFRLKXErgf5BdC34FdJ3OpKFYTuohIFlMC94u4oWRDKoto2NxKS3s4g0GJiMjuUgL3i/jJXDqHkqkWLiKSjZTA/aJyODTWQrgd2DaUTM3oIiLZSQncLyqGg4vAplrAa0IH1cBFRLKVErhfdCwrGr0PPrCskGDAtC64iEiWUgL3i7ix4HnBAAPLCtWELiKSpdKawM3sZDNbbGZLzOy6BK9fZGb1ZvZ29PHVdMaX08qGQCBvu1XJ1IQuIpKd8tL1QWYWBO4CPgvUAm+Y2Szn3MK4on9wzl2Wrrh8IxCE8pquq5JVFPHqsvUZDEpERHZXOmvgk4Alzrllzrk24DHgjDR+vsQNJaupLGJtYwvt4UgGgxIRkd2RzgQ+BFgZs18bPRbvC2Y238yeNLOhiU5kZlPNbK6Zza2vr++JWHNTgslcIg7WbGrJYFAiIrI7elsntmeAEc658cDzwAOJCjnnZjrnJjrnJlZXV6c1wKxWMRya66GtGYAhFcUArFRPdBGRrJPOBF4HxNaoa6LHOjnn1jvnWqO79wKHpik2f+joib7xY0CTuYiIZLN0JvA3gNFmNtLMCoApwKzYAmY2KGb3dGBRGuPLfXHrgg+qKAQ0mYuISDZKWy9051zIzC4DngOCwG+dc++Z2S3AXOfcLOAKMzsdCAGfABelKz5f6KyBewm8T16Q/qV9VAMXEclCaUvgAM652cDsuGPTY7avB65PZ0y+UlIF+cWdk7mA14yudcFFRLJPb+vEJj3JbPtVySqL1YQuIpKFlMD9Jn4oWUURqzdtJRxxGQxKRESSpQTuNx01cOcl7JrKItrDjnVNGgsuIpJNlMD9pnI4tDXB1g1AzLKiug8uIpJVlMD9pnMo2XLAmw8dNJRMRCTbKIH7Tdyyoh01cPVEFxHJLkrgftNvFGDQ8CEAxQV59CspUAIXEckySuB+U1AMFcOgYXHnoSEVWhdcRCTbKIH7UfUYqN+WwL3JXLSgiYhINlEC96Oqfb0m9EgY8GrgqzZuxTmNBRcRyRZK4H5UvR+EWzsndBlSWURLe4T1zW0ZDkxERLpLCdyPqsd4z9Fm9JpKb11wdWQTEckeSuB+VLWv9xxN4EMqNJmLiEi2UQL3o6IK6DsQGj4AYmZj26iObCIi2UIJ3K+q94X69wEoL8qntDBPTegiIllECdyvqsZA/Qedi5oMqShSE7qISBZRAver6jHeoiZNqwFvLLgmcxERyR5K4H7V2RPda0avqSymdoPGgouIZAslcL+q6kjg0Y5sFUVsbg3RuDWUwaBERKS7lMD9qm9/KKzonBO9c1Uy9UQXEckKSuB+ZdZlTvSaSo0FFxHJJkrgfla173aTuWgomYhIdlAC97Pq/WBLAzSvp19JAYX5AfVEFxHJEkrgftbRE71hMWZGTWWxmtBFRLKEErifJZgTXZ3YRESyQ1oTuJmdbGaLzWyJmV23k3JfMDNnZhPTGZ/vlA+F/OIuc6KrBi4ikh3SlsDNLAjcBZwCjAXONbOxCcqVAlcCr6crNt8KBKBqdMxkLkVs2NJOc6vGgouI9HbprIFPApY455Y559qAx4AzEpT7PvBjoCWNsflXx5zoxCwrqo5sIiK9XjoT+BBgZcx+bfRYJzM7BBjqnHt2Zycys6lmNtfM5tbX16c+Uj+pHgONtdDapLHgIiJZpNd0YjOzAHAHcM2uyjrnZjrnJjrnJlZXV/d8cLmssyf6B9RUFgNQqxq4iEivt0cJ3Mz6mtlpZja6G8XrgKEx+zXRYx1KgQOBf5jZcuBwYJY6svWwmDnRq/v2oSAYoHaDeqKLiPR2SSVwM3vEzK6IbufjdTR7BnjPzCbv4u1vAKPNbKSZFQBTgFkdLzrnNjnnqpxzI5xzI4DXgNOdc3OTiVGS1G8kBPKhYTGBgDGoolBN6CIiWSDZGvjxwJzo9ufwas2DgJuAG3f2RudcCLgM/n979x4nR1nne/zz6+7pmcwkM5PL5EJuBBISEiAXQryxCKxiAA+wgkoEdQFl9QiLZ3ePsurRlVXPUXddLwsoi6jrBRRBREUUEAUFImOAlgAAIABJREFUhBDCJQlJBoRcIMnkMkkmk7l01+/8UdXTNcMEZpJOX2a+79erX/XUUzXVvy7o/J566umn+A2wBviJu68ys2vM7JxBxiGFkqyCsUf3mhNdg9hERMpfapD7jwG2RuW3Are7+1Yz+xFwwN9157j7XcBdfeo+fYB9Tx1kbHKwxh0DW1cB4Uj0+9dqYKCISLkb7BV4CzAjKr8VuD8q1wJBoYKSImuaA7v+AplOJjfW0rK3k47ubKmjEhGRVzHYBH4r8EMzuxeoB+6J6hcA6wsZmBRR02zwAHY09/yU7OXd+hm+iEg5G2wC/xjwVeAZ4K3unhuufATwX4UMTIooNif65NG5x4pqJLqISDkb1D3waCDaV/qp/7eCRSTFN24WYLB9HZNPWApoMhcRkXI32J+RzTezebH1s8zsVjP7FzMb7IA4KRdVI2D0dGh5lkkNNSQTxkZdgYuIlLXBdqF/CzgewMymAD8FRgIfBD5X2NCkqKI50VPJBLPGj+SpTbtLHZGIiLyKwSbw2cATUfkdwGPufibwPuDdhQxMiqxpNuxohmyGhdNGs3JjK0HgpY5KREQOYLAJPE3+KWGnAr+OyuuAiQWKSUqhaTZkO6H1RRZNa2RvR4bmlrZSRyUiIgcw2AS+FrjAzKYR/g783qh+ErCrkIFJkfXMib6WRdNHA7DiRf0nFREpV4NN4J8FvgD8BfhjbJ7yM8h3rUslaop+SrZ9LUeNq6OxtooVG5TARUTK1WB/Rvbz6Op7EvBUbNN9wO2FDEyKrKYBRk2ClrWYGQunNrJiQ2upoxIRkQMY9ONE3X2ru68E0mZWE9U97O6rCx6dFNe4Y3oeanLi9NE0b2tjd3t3iYMSEZH+DDqBm9klZtYMtAFtZrbezP624JFJ8TXNge3rwJ1F08L74E9sVDe6iEg5GuxELlcB1xE+x/v86PVL4Dozu7Lw4UlRNR0DXW2wZzPzpzaSMNSNLiJSpgY7e9qVwFXufkOs7udm9izwv4FvFCwyKb7YSPS6mVOYPbFeI9FFRMrUYLvQpxIOWOvrvmibVLKmOeFy+zoAFk1rZOXGVrKa0EVEpOwMNoFvIpzApa9To21SyerGwYjR0PIsAIumjaatM8P6bXtLHJiIiPQ12C7064Gvm9lM4MGo7hTCrvVPFzIwKQGznjnRgdiELq3MmVhfyshERKSPQV2BR48N/RhwEeHgtV8C7wH+yd3/vfDhSdE1zYbt4U/Jjhxby5i6tCZ0EREpQwfzO/Br3X0a0AA0uPs0d7++8KFJSTTNhvYdsG97bEIXJXARkXLzml3oZvbb19jeU3b3MwoQk5RSbCQ6deNYNH009z27jdb2Lhpr06WNTUREegzkHvjmwx6FlI+mKIFvXwtHvik/ocuGVk6bM76EgYmISNxrJnB3v6QYgUiZaJgSzou+4RFYfCnzpzaQTBgrNuxSAhcRKSODvgcuQ5wZzPsbWH0ndOyhNp1izsRRPK4JXUREyooSuLzSwvdCZj+s+hkQ/h78SU3oIiJSVpTA5ZUmnxgOZnviBwAsmt7Ivq4sa7doQhcRkXJR1ARuZkvNbK2ZNZvZ1f1s/5CZPW1mK83sj2Y2t5jxScQMFl4Mmx6FlrU9A9n0czIRkfJRtARuZkngWuBMYC6wrJ8E/SN3P97dFwBfAr5SrPikj/kXgiVh5Q+ZNqaWsZrQRUSkrBTzCnwJ0Ozuz7t7F3ALcG58B3ffE1utA3TTtVRGjodj3gZP3oIFWRZOG80TerSoiEjZKGYCnwxsjK1viup6MbOPmNlzhFfgf9/fgczscjNbbmbLW1paDkuwQtiN3rYVmu/lxOmj+cv2fezc11XqqEREhDIcxBZN1Xo08HHgUwfY5wZ3X+zui5uamoob4HAy6wyoa4Invs+iaY0APKFudBGRslDMBL6Z3s8Mn8Krz/J2C3DeYY1IXl2yCk54N6y7mxNGd5OKJnQREZHSK2YCfwyYZWYzzCwNXAjcGd/BzGbFVs8G1hcxPunPwoshyDDi2ds4dlI9K17UfXARkXJQtATu7hngCuA3wBrgJ+6+ysyuMbNzot2uMLNVZrYS+Afg/cWKTw5g/LEweTGs+D6LpjawcmMrmWxQ6qhERIa9gTzMpGDc/S7grj51n46VrypmPDJACy+CX/4vTp+7me91Z3l2y16Om9xQ6qhERIa1shvEJmXouPMhVcOJO8O2lwayiYiUnhK4vLaaBph7LnXrfsbkOlih34OLiJScErgMzIKLsM49XDL2GY1EFxEpA0rgMjBH/hU0TmNp9328uKOd7W2dpY5IRGRYUwKXgUkkYMHFTN71KFOsRdOqioiUmBK4DNyCZQC8M/mAutFFREpMCVwGrnEadtSbuTD9Rx5p1hz0IiKlpAQug7PwvUwItlLz0sO81Lq/1NGIiAxbSuAyOHPOJlvdwIXJ+/n1M1tKHY2IyLClBC6DUzWC5IL3cHbyzzyy8plSRyMiMmwpgcvgve5ykgTM33IrW3Z3lDoaEZFhSQlcBm/MUeyb8TYuSt7HPU8+X+poRESGJSVwOSgj3/z3jLY22pffXOpQRESGJSVwOTjT38jWujmc3nob23ZrNLqISLEpgcvBMcNf/2FmJTbz1B9uL3U0IiLDjhK4HLSJb3gPO2wMTc/cWOpQRESGHSVwOXipNGumvpv5XSvY8ZeVpY5GRGRYUQKXQzLxtA/T4VXsvO9rpQ5FRGRYUQKXQ3L0kdO4N3060zf9AvZtL3U4IiLDhhK4HBIzo2XepaTpZt9DN5Q6HBGRYUMJXA7Z61/3Rn6fnU/isRsh01nqcEREhgUlcDlkcyaO4q66v2FE1w545rZShyMiMiwogcshMzOaFixlXTCZzJ/+E9xLHZKIyJCnBC4FcdYJR3BT9kxSLavghQdLHY6IyJCnBC4FMXdSPY83vJU9iQZ4+LpShyMiMuQVNYGb2VIzW2tmzWZ2dT/b/8HMVpvZU2Z2n5lNL2Z8cvDMjLeccCT/3X06vu5u2PFcqUMSERnSipbAzSwJXAucCcwFlpnZ3D67PQEsdvcTgJ8CXypWfHLozj5+Et/rfguBJeGR60sdjojIkFbMK/AlQLO7P+/uXcAtwLnxHdz9fndvj1YfAaYUMT45RPOOqKdmzBE8VHsaPPEDaNtW6pBERIasYibwycDG2PqmqO5ALgN+3d8GM7vczJab2fKWlpYChiiHwsw46/hJXLNrKZ7thD9pelURkcOlLAexmdnFwGLgy/1td/cb3H2xuy9uamoqbnDyqs4+fhLrg0m8OPnt8NiNsHdLqUMSERmSipnANwNTY+tTorpezOwtwCeBc9xd03pVmOMnNzBl9AiuD94B2W7441dLHZKIyJBUzAT+GDDLzGaYWRq4ELgzvoOZLQS+RZi8dQO1ApkZZ58widteqGb/vHfB8ptgz0ulDktEZMgpWgJ39wxwBfAbYA3wE3dfZWbXmNk50W5fBkYCt5rZSjO78wCHkzL27sVTyQTOT+uWgWfhwa+UOiQRkSHHvMKnvVy8eLEvX7681GFIH+/+1sO8vLuDP8z5GfbkzXDlCmic+tp/KCIivZjZ4+6+uG99WQ5ik8r3ntdNY8POdh6fdlk4N/qD/17qkEREhhQlcDks3jZvIo21VXxndRYWvS/8XfiuF0sdlojIkKEELodFTVWS8xdN4bertrDzxCvBDB78t1KHJSIyZCiBy2GzbMlUurPOresCOPESeOKHsPP5UoclIjIkKIHLYTNz/ChOOnI0tzy2ET/5o5Csggd0FS4iUghK4HJYLVsyjb9s38cjLdWw+DJ48mY9qUxEpACUwOWwOuv4SdTXpLj50Q1w8kchWQ1/+GKpwxIRqXhK4HJY1VQleceiKdz9zBZ2WiMs+QA8fSu0rCt1aCIiFU0JXA67C5dMpSsbcPuKTfCmj0JqhK7CRUQOkRK4HHZzJtazaFojNz+6Aa8dC6+7HJ65DV58qNShiYhULCVwKYplS6bxXMs+HnthF/zVP8KYGXDbB2H/rlKHJiJSkZTApSjOPmESo6qjwWzVo+D8G6FtC/ziqnCqVRERGRQlcCmK2nSK8xZO5ldPv0xrexdMPhFO/xSs/jms+O9ShyciUnGUwKVoli2ZRlcm4PYVm8OKN14FM06Bu6/WqHQRkUFSApeimXtEPfOnNnLLYxtwd0gk4G9ugFQN3HYpZDpLHaKISMVQApeiWnbSVNZtbWPFhmjwWv0kOO862PI03PvZ0gYnIlJBlMClqP7H/COoSyf50Z835itnnwknfRAeuRbW31u64EREKogSuBRVXXWKcxdO5pdPvUTL3liX+Rn/CuPnwh0fgrZtpQtQRKRCKIFL0V128gwCd/7vr9fkK6tGwPnfhs69cMeHIQhKF6CISAVQApeiO7ppJJefchS3r9jMI8/vyG+YMBfO+Bw03wt/vr50AYqIVAAlcCmJK06bxZTRI/jUHc/QlYldbZ/0AZh9FtzzGXj+9yWLT0Sk3CmBS0mMSCf57DnzaN7Wxrf/+Jf8BjM45z9h7Ez4wQWw8ubSBSkiUsaUwKVk/vrYCZwxdwJfv289m3a15zfUjYVL74bpbwgHtf3+i5puVUSkDyVwKanPnDMPgM/+YnXvDSMa4aLbYP4y+P0X4OdXQLa7BBGKiJQnJXApqcmNI7jqLbO4Z/VW7l29tffGVBrOux7e/HFY+QP44TuhY09pAhURKTNK4FJyl508g1njR/KZO1exvyvbe6MZnPYJOPdaeOFB+M6ZsHtzaQIVESkjRU3gZrbUzNaaWbOZXd3P9lPMbIWZZczsgmLGJqVTlUzwufOOY3Prfr7xu/X977TwYrjop7DrRbjxLeHUqyIiw1jREriZJYFrgTOBucAyM5vbZ7cNwN8CPypWXFIeXnfUWM5fNIX/evB5mrft7X+no0+Dy34TXpXfdCas+D5kM8UNVESkTBTzCnwJ0Ozuz7t7F3ALcG58B3d/wd2fAjQN1zD0ibPmUJtO8ak7ngmfVtafCfPgA/fC+GPhzivgutfB0z/VzG0iMuwUM4FPBmJPsGBTVDdoZna5mS03s+UtLS0FCU5Kb+zIaj62dDaPPL+TO1a+yn3u+iPgst/ChT+CZDXcdhl8802w5hf6uZmIDBsVOYjN3W9w98XuvripqanU4UgBLTtpGvOnNvL5X61h576uA+9oBnPOhg/9ES64KfyJ2Y8vhhtOhfX3KJGLyJBXzAS+GZgaW58S1Yn0SCSMz593HHs6MrzrWw/zUuv+1/oDOO58+J+PwHnfhP274IcXwLfPgNU/h8624gQuIlJkxUzgjwGzzGyGmaWBC4E7i/j+UiGOm9zAf1+6hK27Ozj/+ocOPKgtLpmCBcvgysfh7V+FPZvhJ++DL82A778DHv0vaN342scREakQdsDBQofjzczOAr4KJIGb3P3zZnYNsNzd7zSzk4CfAaOBDmCLu897tWMuXrzYly9ffrhDlxJY/dIe3v+dR+nOBnz7/Sdx4vTRA//jbAY2PgJrfx2+dj4X1k84HmYvhWPOhCMWhlfwIiJlzMwed/fFr6gvZgI/HJTAh7YNO9p5301/ZsueDq6/6EROmzP+4A60fX2YyNfdDRseBg+guiEczT5hLoyPXhPmwohBNBRERA4zJXCpWC17O7nku4+y5uW9fPmCE3jHoimHdsD2neEzxzc8AttWw9bV0Lk7v33UEWFiH38sNE6HxqnQOA0apkJN/aG9t4jIICmBS0Xb29HN333/cR56bgefPOtYPnjKUYU7uDvseSlK5qtg2xrYtiq8as909N63pjFK6NOhYUq4XlMP1aOguj4q18fKo6CqNhw1LyJyEJTApeJ1ZrL8w4+f5FdPv8zfnXIUV585BzucidEd9rVA64b8a/fGWHkzdA1ggJ0l8wm+elT+VVMP6TqoqoN0bZjo03W9l9WjoKYhfDpbTQOkR+m+vcgwc6AEnipFMCIHozqV5OvLFjKmLs23HnieR1/Yyd+fPotTZzcdnkRuBiPHh68pr/juhIIsdO6Fzj3hsmNPrLw72hbbnivvawkH1nW1Q3c7dO0Dz/b/Hr1iSkRX91FST48KR+AnqiCZjpWrIJEK61LV4SsZLVM14ZPeUjVhnQfheweZ8PN4NlwGUV0ynW9gVNW+spyszr9f/JWsUs+DyGGkBC4VJZkwrjl3HvOOqOcbv2vmku8+xrwj6rny9JmcMXciiUSRE0YiGSbSEY2Hdhx3yHaFiby7PUrs+/INgY7dsL81X+5ozTcQutoh6A5H3gfd4XHi5UwXZDvDcrFZIkzw1SMhPTJcVtfny+mRYW+DRb0KZlHZouQfrecaBMl0tIzKPY2VZOzvEv28cseJ9rNk7/VUddgoqmmM4lHDQ8qfutClYnVlAu54YjPX/b6ZF3a0c8yEkXzktJm8/YQjSBY7kVeCIAgTeSZ65co9iSwZLhOpWDkZznKXa1j0NC72hw2MrvbwOEEmajREDYfc1Xy2OxxH0NUWTqrTs9wbLjv3hsd0Bzy2DPqUizjXfSIVJfMoodc0hLcy+jYGejUWYuX4uYzXuec/R+7zeZD/jImqfno66qBqRFhOjcj3pqRq+vSqVId/nzuH8eP2nEvC4+oWTMXRPXAZsjLZgF89/TLX3t/Muq1tzBhXx4dPPZrzFkwmndI/VkNCEEQ9ClGvQpCJehq6wrqeRBVLjD3lbGwZ5G8TeBAe1wPI7O+np6M1v97V1n/S9QCc/G2Ifo8f3Y7oN/nHehqCzMBvpRw0yw+wrGnoM/ByVPjema6w0ZWNlvEGXyIVNTBGRK/a3stUTVSuCRscuf1y9ZaIGn65V1vvcqYzdozob9N1vd8j3uuSiG4ZJVLR7aNUvjeHqBGfO7+5ck/OizcY40v69Nb07cWpCuMoYi+NErgMeUHg/Hb1Fr7xu2ZWvbSHunSSJTPG8Majx/HGmWM5dmJ98bvYRQbDPWyQdO8LezlyYyRyr9ztkExn7+Sa6QgbAD0NA/KJJ1fnQZgkc+M0epa782M0LJkfIxG/us+NpQiyUe9L1AuT2R+tR3WH0lOSG5OR6Qg/YzlLVPUeXBrvrRnRCPPfA03HFOztNIhNhrxEwlh63CTeNm8iD6zfzj2rt/DQczu4f+0aAEbXVvGGo8eGCf3oscwYV3d4R7GLDJZZNMAwXXkTCuXGcXTvD5Nwdzt0d8SSfEd4hZ+ui14j8+WquvAKOqdXQ6G9d2Mm6M7fngkyvV/ZbnpdSfdX7ntF3mu8RVTfX49OEPWqZLvyDZ/4uJTWDfm6GacUNIEfiBK4DDlmxpuPaeLNx4RPqnt5934efm4Hf2rewUPPbeeup7cAMLYuzawJI5k5fiSzxo9i1viRzJwwkqaR1UrsIoNllr9iP1SJZDTgceShH6vYijk9ubrQZThxd17Y0c6fmrfz1KZW1m9ro3lbG3s7Mj371NekmDVhFEeNq2NiQw3jR1Uzvj5cTqivoWlUNVVJ3VsXkeJQF7oI4dX5jHF1zBhXB0wHwqS+bW8n67e20bxtb09S/8O6Fra3dRL008YdW5emaVQ1Y+rSjK5N01hb1Ws5uq6Kxto0dekUtekkI9JJatNJalJJ3YcXkYJQApdhz8yYUF/DhPoaTp41rte2bODs2NfJtj2dbN3Twba9+eW2PZ20tnexZsseWtu7aW3v6jfZ91VTlaA2nWJEVbInuefKtelUT7LP1adTCapTSapTifBVFZbTufVoW01VgnQySXVVvj6dSugndSJDlBK4yKtIJozxo2oYP6qG4yY3vOq+QeDs7ciwq72LXe1dtLZ3s68rQ3tXlv1dWfZ3Z6Nyf3VZWto6ae9qp6MrS3tU35U59N8/JwyqkgmqkglSSQvLCaMqleipT0f16agunUqQTiaoShqp3DKR//tUIqqPluH+lj9ez3Hyf5dKhI2JVMLCZTJXzh0v2jcqVyXz+2tMgsgrKYGLFEgiYTTUVtFQW8WR1BXkmEHgdGUDOrsDOrPZcJkJ6MoEdGaydHQH0fZsrD7c1pkJ6OjOksk63UFAd8bJBAHd2YDurEfLsNyVCXrW93Vm6Iq2d2UCMtmA7sDJZIOeY2WyTmYg3Q0FkkrkGgf5RkJVyqhKJPKJPhk2DKoSvdfzDYaocZDINUzi5XxjJJmwngZLbluuUZGwfEMkfuxkItoW7ZPbliv3bbgkE/njVEXryWhfNVZkoJTARcpYImHUJJLUVCWBqlKH04t7mMQz2bCRkUv43T3lqBGQDcgGTjYI989GDYDceibWIMjEGgq5cq7xEG94ZLL5Y3dHx8/k3iM6dmcmSzZwuqP17iDo2d6dDcgE+WOFDZvyGNCbMHo1EuKvXCMjlQwbKqlejY9X9nLEezESfZfxhkasLqwP/9/LNSriDZGwwZHvlelp8PQ0kHrHUpXs3VCpijWkUsne+8rgKIGLyEExs+iKGEaQLHU4h8w93qjINRryDYzctsDzjYSsxxok7gQBZIKAwJ1sANkgIBur69VwiRoR8cZHvKGT2z/wMIZcY+SVPSL5cnsm07sx07MMyGZz8cbeI8jXZQIv5i+gXsGM3r0WFjYs8g0ISFrYQxHWETY44mUzEolY2ejp1UjGtsVv3SSTsd6ReH2ssRJfTyQMwzALG1u5cvgZDAPeOncCU8fUHvZzpgQuIkL4j294RVjqSErH3Qk8HLwZeL6REsQaF92Bk+11K6V3j0lun1wDpfc++UZLrjHSa58g9l4eb2DQUw48bGjk4ouXc9uy0ecI4p8jcLqyvRtKPY2bWGOnO2o0xXt2MtlgQANUc2Y01SmBi4hI8YRXqqg7ux/x3gp3cPKNByc2nTpObbo4qVUJXERE5DUkEkaC8JZRudB0UiIiIhVICVxERKQCKYGLiIhUICVwERGRCqQELiIiUoGKmsDNbKmZrTWzZjO7up/t1Wb242j7n83syGLGJyIiUimKlsDNLAlcC5wJzAWWmdncPrtdBuxy95nAfwBfLFZ8IiIilaSYV+BLgGZ3f97du4BbgHP77HMu8L2o/FPgr00z+4uIiLxCMRP4ZGBjbH1TVNfvPu6eAXYDY/seyMwuN7PlZra8paXlMIUrIiJSvipyJjZ3vwG4AcDMWszsxQIefhywvYDHG650HgtD57EwdB4LQ+exMAZ7Hqf3V1nMBL4ZmBpbnxLV9bfPJjNLAQ3Ajlc7qLs3FTJIM1vu7osLeczhSOexMHQeC0PnsTB0HgujUOexmF3ojwGzzGyGmaWBC4E7++xzJ/D+qHwB8Dv3Uj7gTkREpDwV7Qrc3TNmdgXwGyAJ3OTuq8zsGmC5u98JfBv4vpk1AzsJk7yIiIj0UdR74O5+F3BXn7pPx8odwDuLGVM/bijx+w8VOo+FofNYGDqPhaHzWBgFOY+mHmoREZHKo6lURUREKpASuIiISAVSAo95rbnapX9mdpOZbTOzZ2J1Y8zsHjNbHy1HlzLGcmdmU83sfjNbbWarzOyqqF7ncRDMrMbMHjWzJ6Pz+Nmofkb0fIXm6HkL6VLHWgnMLGlmT5jZL6N1ncdBMrMXzOxpM1tpZsujuoJ8r5XAIwOcq136911gaZ+6q4H73H0WcF+0LgeWAf7R3ecCrwc+Ev3/p/M4OJ3A6e4+H1gALDWz1xM+V+E/oucs7CJ87oK8tquANbF1nceDc5q7L4j99rsg32sl8LyBzNUu/XD3Bwh/9hcXn9f+e8B5RQ2qwrj7y+6+IirvJfxHczI6j4PiobZotSp6OXA64fMVQOdxQMxsCnA2cGO0bug8FkpBvtdK4HkDmatdBm6Cu78clbcAE0oZTCWJHqO7EPgzOo+DFnX7rgS2AfcAzwGt0fMVQN/tgfoq8DEgiNbHovN4MBz4rZk9bmaXR3UF+V5X5FzoUlnc3c1Mv1ccADMbCdwGfNTd98QfxqfzODDungUWmFkj8DNgTolDqjhm9nZgm7s/bmanljqeCneyu282s/HAPWb2bHzjoXyvdQWeN5C52mXgtprZJIBoua3E8ZQ9M6siTN4/dPfbo2qdx4Pk7q3A/cAbgMbo+Qqg7/ZAvAk4x8xeILydeDrwNXQeB83dN0fLbYQNyiUU6HutBJ43kLnaZeDi89q/H/h5CWMpe9H9xW8Da9z9K7FNOo+DYGZN0ZU3ZjYCeCvheIL7CZ+vADqPr8nd/9ndp7j7kYT/Fv7O3S9C53FQzKzOzEblysAZwDMU6HutmdhizOwswvs+ubnaP1/ikCqCmd0MnEr4iLytwGeAO4CfANOAF4F3uXvfgW4SMbOTgQeBp8nfc/wE4X1wnccBMrMTCAcFJQkvUH7i7teY2VGEV5JjgCeAi929s3SRVo6oC/2f3P3tOo+DE52vn0WrKeBH7v55MxtLAb7XSuAiIiIVSF3oIiIiFUgJXEREpAIpgYuIiFQgJXAREZEKpAQuIiJSgZTAReSwM7NTzcyj+bVFpACUwEVERCqQEriIiEgFUgIXGQbM7Eoze9bMOsxsvZl9MjentZm9YGafN7MbzWyPmW03sy+YWSL296PM7Ftm1mJmnWa23MzO6PMe483sO2a2NXqftWZ2aZ9QjjWzB8ys3cxWm9mZRfj4IkOSnkYmMsSZ2b8AlwAfBVYCxwLfBGqA/xPtdiXhNMInET5s4ZuE0+J+Ldp+U7TtYmAD8CHgl2Z2grs/G807/gdgP3AR8Dwwk3DKzbh/Az5O+IjPTwA/NrPp7r6rsJ9aZOjTVKoiQ5iZ1QLbgXe4+92x+vcBX3f3xuiJUxvd/a9i278AvNfdp5rZTGA9cLa73xXbZwWw0t0vNbPLgGuBme6+qZ84TiV8EMb5uSetmdkEwmchL3X33xT6s4sMdboCFxna5gEjgNv6PHM4CdSYWVO0/nCfv/sT8M9mVg/Mjeoe6LPPA4SP6gQ4EVjdX/LuY2Wu4O5bzSwLTBjQJxGRXpTARYa23H3sdwLr+tle7CebdfVTp7E4Ige9WVxiAAABHklEQVRBXxyRoW0V0AEc5e7N/byy0X6v7/N3bwQ2u/ue6BgAp/TZ5xTCZxsDPA7M1e+8RYpHCVxkCHP3NuALwBfM7CNmNtvM5pnZhWb2xdiuC8zsX8zsGDN7D3AV8O/RMZ4DbgWuM7O3mdkcM/sacBzw5ejvbyZ8rvGdZvYWM5thZn9tZu8u1mcVGW7UhS4yxLn7v5rZy8AVhEl5P2F3+ndju30DmA4sB7qB/yQ/Ah3gA4TJ+gdAPfA08HZ3fzZ6j3YzezPwJeAWYCTwAvD/DtfnEhnuNApdZJiLRqHf6O6fK3UsIjJw6kIXERGpQErgIiIiFUhd6CIiIhVIV+AiIiIVSAlcRESkAimBi4iIVCAlcBERkQqkBC4iIlKB/j9F7tAN2iVWHAAAAABJRU5ErkJggg==\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"utils.plot_history(wn18_history)"
]
},
{
"cell_type": "markdown",
"id": "18",
"metadata": {},
"source": [
"### Evaluate the model\n",
"\n",
"We've now trained a model, so we can apply the evaluation procedure from the paper to it. This is done by taking each test edge `E = (s, r, o)`, and scoring it against all mutations `(s, r, n)` and `(n, r, o)` for every node `n` in the graph, that is, doing a prediction for every one of these edges similar to `E`. The \"raw\" rank is the number of mutated edges that have a higher predicted score than the true `E`."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "19",
"metadata": {},
"outputs": [],
"source": [
"wn18_raw_ranks, wn18_filtered_ranks = wn18_complex.rank_edges_against_all_nodes(\n",
" wn18_gen.flow(wn18_test), wn18_graph\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "20",
"metadata": {},
"outputs": [],
"source": [
"# helper function to compute metrics from a dictionary of name -> array of ranks\n",
"def results_as_dataframe(name_to_results):\n",
" return pd.DataFrame(\n",
" name_to_results.values(),\n",
" columns=[\"mrr\", \"hits at 1\", \"hits at 3\", \"hits at 10\"],\n",
" index=name_to_results.keys(),\n",
" )\n",
"\n",
"\n",
"def summarise(name_to_ranks):\n",
" return results_as_dataframe(\n",
" {\n",
" name: (\n",
" np.mean(1 / ranks),\n",
" np.mean(ranks <= 1),\n",
" np.mean(ranks < 3),\n",
" np.mean(ranks <= 10),\n",
" )\n",
" for name, ranks in name_to_ranks.items()\n",
" }\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "21",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
mrr
\n",
"
hits at 1
\n",
"
hits at 3
\n",
"
hits at 10
\n",
"
\n",
" \n",
" \n",
"
\n",
"
raw
\n",
"
0.598731
\n",
"
0.4814
\n",
"
0.6065
\n",
"
0.8192
\n",
"
\n",
"
\n",
"
filtered
\n",
"
0.940128
\n",
"
0.9299
\n",
"
0.9451
\n",
"
0.9546
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" mrr hits at 1 hits at 3 hits at 10\n",
"raw 0.598731 0.4814 0.6065 0.8192\n",
"filtered 0.940128 0.9299 0.9451 0.9546"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"summarise({\"raw\": wn18_raw_ranks, \"filtered\": wn18_filtered_ranks})"
]
},
{
"cell_type": "markdown",
"id": "22",
"metadata": {},
"source": [
"For comparison, Table 2 in the paper gives the following results for WN18 (`NaN` denotes values the paper does not include). All of the numbers are similar:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "23",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"