{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dc64c2b6",
   "metadata": {},
   "source": [
    "#### Инициализация Keras\n",
    "\n",
    "torch был заменен на jax, так как с torch рекуррентные сети не работали"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "507915ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.9.2\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n",
    "import keras\n",
    "\n",
    "print(keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dec3146",
   "metadata": {},
   "source": [
    "#### Загрузка данных для классификации с помощью глубоких сетей\n",
    "\n",
    "В качестве набора данных используется набор отзывов к фильмам с сайта IMDB.\n",
    "\n",
    "Набор включает 50 000 отзывов, половина из которых находится в обучающем наборе данных (x_train), а половина - в тестовом (x_valid). \n",
    "\n",
    "Метки (y_train и y_valid) имеют бинарный характер и назначены в соответствии с этими 10-балльными оценками:\n",
    "- отзывы с четырьмя звездами или меньше считаются отрицательным (y = 0);\n",
    "- отзывы с семью звездами или больше считаются положительными (y = 1);\n",
    "- умеренные отзывы — с пятью или шестью звездами — не включались в набор данных, что упрощает задачу бинарной классификации.\n",
    "\n",
    "Данные уже предобработаны для простоты работы с ними.\n",
    "\n",
    "unique_words - в векторное пространство включается только слова, которые встречаются в корпусе не менее 10 000 раз.\n",
    "\n",
    "max_length - максимальная длина отзыва (если больше, то обрезается, если меньше, то дополняется \"пустыми\" словами)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e0043e5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.api.datasets import imdb\n",
    "import os\n",
    "\n",
    "unique_words = 10000\n",
    "max_length = 100\n",
    "\n",
    "output_dir = \"tmp\"\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "\n",
    "(X_train, y_train), (X_valid, y_valid) = imdb.load_data(num_words=unique_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bfd826c",
   "metadata": {},
   "source": [
    "#### Приведение отзывов к длине max_length (100)\n",
    "\n",
    "padding и truncating - дополнение и обрезка отзывов начинается с начала (учитывается специфика затухания градиента в рекуррентных сетях)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "131e125a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.api.preprocessing.sequence import pad_sequences\n",
    "\n",
    "X_train = pad_sequences(X_train, maxlen=max_length, padding=\"pre\", truncating=\"pre\", value=0)\n",
    "X_valid = pad_sequences(X_valid, maxlen=max_length, padding=\"pre\", truncating=\"pre\", value=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c337946",
   "metadata": {},
   "source": [
    "#### Формирование архитектуры глубокой многослойной рекуррентной двунаправленной LSTM сети\n",
    "\n",
    "Параметр return_sequences=True первого рекуррентного слоя позволяет представить вывод этого слоя в формате, подходящем для передачи во второй рекуррентный слой\n",
    "\n",
    "Первый слой (Embedding) выполняет векторизацию"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1e3fb0ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"sequential\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"sequential\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ embedding (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Embedding</span>)           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)        │       <span style=\"color: #00af00; text-decoration-color: #00af00\">640,000</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ spatial_dropout1d               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)        │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">SpatialDropout1D</span>)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ bidirectional (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Bidirectional</span>)   │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">100</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">128</span>)       │        <span style=\"color: #00af00; text-decoration-color: #00af00\">66,048</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ bidirectional_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Bidirectional</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">128</span>)            │        <span style=\"color: #00af00; text-decoration-color: #00af00\">98,816</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                   │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>)              │           <span style=\"color: #00af00; text-decoration-color: #00af00\">129</span> │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ embedding (\u001b[38;5;33mEmbedding\u001b[0m)           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m, \u001b[38;5;34m64\u001b[0m)        │       \u001b[38;5;34m640,000\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ spatial_dropout1d               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m, \u001b[38;5;34m64\u001b[0m)        │             \u001b[38;5;34m0\u001b[0m │\n",
       "│ (\u001b[38;5;33mSpatialDropout1D\u001b[0m)              │                        │               │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ bidirectional (\u001b[38;5;33mBidirectional\u001b[0m)   │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m100\u001b[0m, \u001b[38;5;34m128\u001b[0m)       │        \u001b[38;5;34m66,048\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ bidirectional_1 (\u001b[38;5;33mBidirectional\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m128\u001b[0m)            │        \u001b[38;5;34m98,816\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense (\u001b[38;5;33mDense\u001b[0m)                   │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m1\u001b[0m)              │           \u001b[38;5;34m129\u001b[0m │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">804,993</span> (3.07 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m804,993\u001b[0m (3.07 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">804,993</span> (3.07 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m804,993\u001b[0m (3.07 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from keras.api.models import Sequential\n",
    "from keras.api.layers import InputLayer, Embedding, SpatialDropout1D, LSTM, Bidirectional, Dense\n",
    "\n",
    "mlstm_model = Sequential()\n",
    "mlstm_model.add(InputLayer(shape=(max_length,), dtype=\"float32\"))\n",
    "mlstm_model.add(Embedding(unique_words, 64))\n",
    "mlstm_model.add(SpatialDropout1D(0.2))\n",
    "mlstm_model.add(Bidirectional(LSTM(64, dropout=0.2, return_sequences=True)))\n",
    "mlstm_model.add(Bidirectional(LSTM(64, dropout=0.2)))\n",
    "mlstm_model.add(Dense(1, activation=\"sigmoid\"))\n",
    "\n",
    "mlstm_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c27996b5",
   "metadata": {},
   "source": [
    "#### Обучение модели\n",
    "\n",
    "Веса модели сохраняются в каталог tmp после каждой эпохи обучения с помощью callback-параметра\n",
    "\n",
    "В дальнейшем веса можно загрузить"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "11236198",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m65s\u001b[0m 329ms/step - accuracy: 0.6714 - loss: 0.5539 - val_accuracy: 0.8469 - val_loss: 0.3582\n",
      "Epoch 2/4\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m79s\u001b[0m 403ms/step - accuracy: 0.8938 - loss: 0.2680 - val_accuracy: 0.8510 - val_loss: 0.3457\n",
      "Epoch 3/4\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m85s\u001b[0m 437ms/step - accuracy: 0.9221 - loss: 0.2047 - val_accuracy: 0.8436 - val_loss: 0.3869\n",
      "Epoch 4/4\n",
      "\u001b[1m196/196\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m97s\u001b[0m 496ms/step - accuracy: 0.9459 - loss: 0.1525 - val_accuracy: 0.8372 - val_loss: 0.4265\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x16e15e660>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.api.callbacks import ModelCheckpoint\n",
    "\n",
    "mlstm_model.compile(\n",
    "    loss=\"binary_crossentropy\",\n",
    "    optimizer=\"adam\",\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n",
    "\n",
    "mlstm_model.fit(\n",
    "    X_train,\n",
    "    y_train,\n",
    "    batch_size=128,\n",
    "    epochs=4,\n",
    "    validation_data=(X_valid, y_valid),\n",
    "    callbacks=[ModelCheckpoint(filepath=output_dir + \"/mlstm_weights.{epoch:02d}.keras\")],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b170b40c",
   "metadata": {},
   "source": [
    "#### Загрузка лучшей модели и оценка ее качества\n",
    "\n",
    "Качество модели - 85.1 %."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "94987771",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m782/782\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m36s\u001b[0m 46ms/step - accuracy: 0.8513 - loss: 0.3512\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.3456658124923706, 0.8509600162506104]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlstm_model.load_weights(output_dir + \"/mlstm_weights.02.keras\")\n",
    "mlstm_model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f4ded2c",
   "metadata": {},
   "source": [
    "#### Визуализация распределения вероятностей результатов модели на валидационной выборке"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8965a612",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m782/782\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m35s\u001b[0m 45ms/step\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 16484 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.hist(mlstm_model.predict(X_valid))\n",
    "_ = plt.axvline(x=0.5, color=\"orange\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv (3.12.10)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
