{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e13159d6",
   "metadata": {},
   "source": [
    "#### Инициализация Keras\n",
    "\n",
    "torch был заменен на jax, так как с torch рекуррентные сети не работали"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "507915ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.9.2\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n",
    "import keras\n",
    "\n",
    "print(keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e989a5",
   "metadata": {},
   "source": [
    "#### Загрузка данных для классификации с помощью глубоких сетей\n",
    "\n",
    "В качестве набора данных используется набор отзывов к фильмам с сайта IMDB.\n",
    "\n",
    "Набор включает 50 000 отзывов, половина из которых находится в обучающем наборе данных (x_train), а половина - в тестовом (x_valid). \n",
    "\n",
    "Метки (y_train и y_valid) имеют бинарный характер и назначены в соответствии с этими 10-балльными оценками:\n",
    "- отзывы с четырьмя звездами или меньше считаются отрицательным (y = 0);\n",
    "- отзывы с семью звездами или больше считаются положительными (y = 1);\n",
    "- умеренные отзывы — с пятью или шестью звездами — не включались в набор данных, что упрощает задачу бинарной классификации.\n",
    "\n",
    "Данные уже предобработаны для простоты работы с ними.\n",
    "\n",
    "unique_words - в векторное пространство включается только слова, которые встречаются в корпусе не менее 10 000 раз.\n",
    "\n",
    "max_length - максимальная длина отзыва (если больше, то обрезается, если меньше, то дополняется \"пустыми\" словами)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e0043e5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.api.datasets import imdb\n",
    "import os\n",
    "\n",
    "unique_words = 10000\n",
    "max_length = 100\n",
    "\n",
    "output_dir = \"tmp\"\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "\n",
    "(X_train, y_train), (X_valid, y_valid) = imdb.load_data(num_words=unique_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86b62f23",
   "metadata": {},
   "source": [
    "#### Приведение отзывов к длине max_length (100)\n",
    "\n",
    "padding и truncating - дополнение и обрезка отзывов начинается с начала (учитывается специфика затухания градиента в рекуррентных сетях)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "131e125a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.api.preprocessing.sequence import pad_sequences\n",
    "\n",
    "X_train = pad_sequences(X_train, maxlen=max_length, padding=\"pre\", truncating=\"pre\", value=0)\n",
    "X_valid = pad_sequences(X_valid, maxlen=max_length, padding=\"pre\", truncating=\"pre\", value=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a78bde02",
   "metadata": {},
   "source": [
    "#### Формирование архитектуры глубокой рекуррентной сети\n",
    "\n",
    "\n",
    "Первый слой (Embedding) выполняет векторизацию"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1e3fb0ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"sequential\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"sequential\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ embedding (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Embedding</span>)           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)        │       <span style=\"color: #00af00; text-decoration-color: #00af00\">640,000</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ spatial_dropout1d               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)        │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">SpatialDropout1D</span>)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ simple_rnn (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">SimpleRNN</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">82,176</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                   │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>)              │           <span style=\"color: #00af00; text-decoration-color: #00af00\">257</span> │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ embedding (\u001b[38;5;33mEmbedding\u001b[0m)           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m, \u001b[38;5;34m64\u001b[0m)        │       \u001b[38;5;34m640,000\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ spatial_dropout1d               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m, \u001b[38;5;34m64\u001b[0m)        │             \u001b[38;5;34m0\u001b[0m │\n",
       "│ (\u001b[38;5;33mSpatialDropout1D\u001b[0m)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ simple_rnn (\u001b[38;5;33mSimpleRNN\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m)            │        \u001b[38;5;34m82,176\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense (\u001b[38;5;33mDense\u001b[0m)                   │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m1\u001b[0m)              │           \u001b[38;5;34m257\u001b[0m │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">722,433</span> (2.76 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m722,433\u001b[0m (2.76 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">722,433</span> (2.76 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m722,433\u001b[0m (2.76 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from keras.api.models import Sequential\n",
    "from keras.api.layers import InputLayer, Embedding, SpatialDropout1D, SimpleRNN, Dense\n",
    "\n",
    "rnn_model = Sequential()\n",
    "rnn_model.add(InputLayer(shape=(max_length,), dtype=\"float32\"))\n",
    "rnn_model.add(Embedding(unique_words, 64))\n",
    "rnn_model.add(SpatialDropout1D(0.2))\n",
    "rnn_model.add(SimpleRNN(256, dropout=0.2))\n",
    "rnn_model.add(Dense(1, activation=\"sigmoid\"))\n",
    "\n",
    "rnn_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23900b11",
   "metadata": {},
   "source": [
    "#### Обучение модели\n",
    "\n",
    "Веса модели сохраняются в каталог tmp после каждой эпохи обучения с помощью callback-параметра\n",
    "\n",
    "В дальнейшем веса можно загрузить"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "11236198",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 68ms/step - accuracy: 0.5207 - loss: 0.6994 - val_accuracy: 0.5872 - val_loss: 0.6700\n",
      "Epoch 2/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 64ms/step - accuracy: 0.6188 - loss: 0.6423 - val_accuracy: 0.6368 - val_loss: 0.6183\n",
      "Epoch 3/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 64ms/step - accuracy: 0.7102 - loss: 0.5539 - val_accuracy: 0.6463 - val_loss: 0.6441\n",
      "Epoch 4/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 65ms/step - accuracy: 0.7746 - loss: 0.4737 - val_accuracy: 0.7338 - val_loss: 0.5681\n",
      "Epoch 5/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 65ms/step - accuracy: 0.8127 - loss: 0.4065 - val_accuracy: 0.6766 - val_loss: 0.6422\n",
      "Epoch 6/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 67ms/step - accuracy: 0.8613 - loss: 0.3246 - val_accuracy: 0.7152 - val_loss: 0.6385\n",
      "Epoch 7/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 66ms/step - accuracy: 0.8923 - loss: 0.2667 - val_accuracy: 0.7202 - val_loss: 0.6684\n",
      "Epoch 8/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 67ms/step - accuracy: 0.9032 - loss: 0.2335 - val_accuracy: 0.7296 - val_loss: 0.6990\n",
      "Epoch 9/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 70ms/step - accuracy: 0.9118 - loss: 0.2143 - val_accuracy: 0.6944 - val_loss: 0.7852\n",
      "Epoch 10/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 70ms/step - accuracy: 0.9205 - loss: 0.2022 - val_accuracy: 0.7359 - val_loss: 0.7074\n",
      "Epoch 11/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 69ms/step - accuracy: 0.9418 - loss: 0.1523 - val_accuracy: 0.7127 - val_loss: 0.8376\n",
      "Epoch 12/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 68ms/step - accuracy: 0.9440 - loss: 0.1462 - val_accuracy: 0.7288 - val_loss: 0.8534\n",
      "Epoch 13/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 69ms/step - accuracy: 0.9344 - loss: 0.1649 - val_accuracy: 0.7157 - val_loss: 0.8279\n",
      "Epoch 14/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 70ms/step - accuracy: 0.9201 - loss: 0.1998 - val_accuracy: 0.6386 - val_loss: 1.1343\n",
      "Epoch 15/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 68ms/step - accuracy: 0.9301 - loss: 0.1774 - val_accuracy: 0.7041 - val_loss: 0.9636\n",
      "Epoch 16/16\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 69ms/step - accuracy: 0.9616 - loss: 0.1055 - val_accuracy: 0.6747 - val_loss: 1.1050\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x3448ff2c0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.api.callbacks import ModelCheckpoint\n",
    "\n",
    "rnn_model.compile(\n",
    "    loss=\"binary_crossentropy\",\n",
    "    optimizer=\"adam\",\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n",
    "\n",
    "rnn_model.fit(\n",
    "    X_train,\n",
    "    y_train,\n",
    "    batch_size=128,\n",
    "    epochs=16,\n",
    "    validation_data=(X_valid, y_valid),\n",
    "    callbacks=[ModelCheckpoint(filepath=output_dir + \"/rnn_weights.{epoch:02d}.keras\")],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccf878d3",
   "metadata": {},
   "source": [
    "#### Загрузка лучшей модели и оценка ее качества\n",
    "\n",
    "Качество модели - 73.6 %."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "94987771",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m782/782\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m8s\u001b[0m 10ms/step - accuracy: 0.7307 - loss: 0.7206\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.7074107527732849, 0.7359200119972229]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnn_model.load_weights(output_dir + \"/rnn_weights.10.keras\")\n",
    "rnn_model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e05b98c0",
   "metadata": {},
   "source": [
    "#### Визуализация распределения вероятностей результатов модели на валидационной выборке"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8965a612",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m782/782\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m8s\u001b[0m 10ms/step\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 16456 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.hist(rnn_model.predict(X_valid))\n",
    "_ = plt.axvline(x=0.5, color=\"orange\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv (3.12.10)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
