example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting random seed to avoid randomness\n",
    "from sniffer.training import seed_everything\n",
    "seed_everything(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example Notebook\n",
    "\n",
    "This notebook provides a few examples of the code that we ran for the CipherSniffer paper.\n",
    "\n",
    "In this notebook, we use the `Cipherdata_sample` (a very small subset of the Cipherdata). Its important to note that the results in the paper are based on models trained on the full dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ciphers\n",
    "\n",
    "These are the 5 ciphers that we applied to create the CipherData."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sniffer.ciphers import substitution, transposition, reverse, shift, wordflip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"canada wins the world cup\"\n",
    "\n",
    "print(\"Substitution: \", substitution(text))\n",
    "print(\"Transposition: \", transposition(text))\n",
    "print(\"Shift: \", shift(text))\n",
    "print(\"Reverse: \", reverse(text))\n",
    "print(\"Wordflip: \", wordflip(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom GloVe\n",
    "\n",
    "To train GloVe embeddings on your own corpus, navigate to the [official GloVe Repository](https://github.com/stanfordnlp/GloVe) and replace the `demo.sh` file with `modified_demo.sh` found in this repository. You will also have to create a copy of the `embedding.txt` file with all the text on a single line. This can be done in the terminal with the following command `tr '\\n' ' ' < input.txt > output.txt`.\n",
    "\n",
    "# Tokenizers\n",
    "\n",
    "In this section, we show how word-level and subword-level tokenizers are trained. This example skips over a small detail where we remove the labels from the txt file. This step can be done with the following command `sed 's/^..//' input.txt > output.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sniffer.tokenizers import bpe_train, wordpiece_train, tokenizer_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "infpath = \"data/cipherdata_sample/embedding.txt\"\n",
    "outfpath = \"./test\"\n",
    "\n",
    "bpe_train(infpath, outfpath)\n",
    "wordpiece_train(infpath, outfpath)\n",
    "tokenizer_train(infpath, outfpath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GRU + BPE Model Training Example\n",
    "\n",
    "In this example, we show how to train a GRU model with a BPE tokenizer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sniffer.training import load_data, ohe_labels, keras_train, evaluate\n",
    "from sniffer.tokenizers import subword_level\n",
    "from sniffer.models import GRU_model_trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Config\n",
    "N_LABELS = 6\n",
    "MAX_SEQUENCE_LENGTH = 158 # 76 -> normal, 158 -> subword, 443 -> character level\n",
    "cipher_data = \"data/cipherdata_sample\"\n",
    "tokenizer_file = \"data/tokenizers/BPE_trained.json\"\n",
    "\n",
    "# Loading Data\n",
    "train, valid, test = load_data(cipher_data)\n",
    "\n",
    "# Tokenizing Data\n",
    "x_train, x_valid, x_test, vocab_size = subword_level(train, valid, test, tokenizer_file, MAX_SEQUENCE_LENGTH)\n",
    "\n",
    "# Encoding Labels\n",
    "y_train, y_valid, y_test = ohe_labels(train, valid, test, N_LABELS)\n",
    "\n",
    "# Defining Models\n",
    "model = GRU_model_trainable(MAX_SEQUENCE_LENGTH, vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training Model\n",
    "keras_train(model, x_train, y_train, x_valid, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluating model\n",
    "evaluate(model, x_train, x_valid, x_test, y_train, y_valid, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT Training Example\n",
    "\n",
    "In this section, we show how the BERT model was trained. Note that the computational resources needed to train this model are much more than the GRU and LSTM architectures. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sniffer.training import load_data\n",
    "from sniffer.bert import BertClassifier, bert_evaluate, bert_train\n",
    "from transformers import BertTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data file path\n",
    "cipher_data = \"data/cipherdata_sample\"\n",
    "\n",
    "# Loading Data\n",
    "train, valid, test = load_data(cipher_data)\n",
    "\n",
    "# Loading Model and tokenizer\n",
    "model = BertClassifier()\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-cased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training Model\n",
    "bert_train(model, train, valid, tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluating Model\n",
    "bert_evaluate(model, test, tokenizer)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.3 ('venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "6549ab8b689ab7a083d6ad2eb1a3b39e2fe3c4142e49e5864ce987f02abf3471"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}