Lesson_03_version_control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Correctness, replicability (and reproducibility), auditing\n",
    "\n",
    "Doing research today means writing code.\n",
    "\n",
    "We usually learn to write code but not to take care of it in the long term\n",
    "\n",
    "a common saying in the programming world is:\n",
    "\n",
    "> write your code as if the next person having to work on it is a psychopath that knows where you live\n",
    "\n",
    "Considering the number of projects that the average scientist has to follow nowadays, the cited psychopath might be yourself six months in the future!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* CORRECTNESS: to be sure that your code does exactly what you think it does; that any modification you perform does not introduce errors, and that is they do, you would be able to identify them and rollback from them\n",
    "* REPLICABILITY - 1: you want to be able to repeat an analysis and obtain the same results, even after several months. This means keeping track of what are the prerequisites of the analysis, how to use it, on which data and which parameters\n",
    "* REPLICABILITY - 2: Allow to someone else to do the same, ideally without you having to be physicall present in the same room with them to explain step-by-step all the process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* REPRODUCIBILITY: allow others to reuse the same analysis (and in general your ideas) on other data and use cases compared to the one you worked on\n",
    "* AUDITING: keep the history of your project, knowing what have been done, when and why. This is necessary both to maintain the knowledge you obtain and to allow some external reviewer to verify what you did"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What is necessary to have \"healthy\" code?\n",
    "\n",
    "* version control\n",
    "* documentation\n",
    "* test procedures\n",
    "* procedure automation\n",
    "* well designed analysis pipeline\n",
    "\n",
    "In this course our plan is to teach you, alonside how to write code, also how to manage your projects to avoid those moments of sheer terror, desperation and discomfort that you could feel otherwise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you think I'm joking, imagine the following situations:\n",
    "\n",
    "* Your simulation run for 36 hours, and it fails at the last step. You don't know how to recover, you have to start from scratch, without understanding where the error was"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* while you edit your thesis you accidentally delete a paragraph (or an entire section) and you don't notice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* they ask you to modify a plot for an article, but to modify it you have to run again the previous 36 hours simulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* you have to focus on the exams for 6-7 months and, when it's time to go back to your project, you can't remember what you did already tried and what was still on the todo list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* your program requires and very precise series of steps to be executed in the correct order. The cat hides the notes where you wrote them and put it under the couch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* you kept a proper documentation for your project, but with the last backup you forgot to copy the last version and now your documentation and your code are not in sync"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* you modify your code and, while you are presenting it to your professor, you realize that there was a wrong step and the results are absurds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* you inherit someone else's code, and have no idea on why a specific line of code is there, but you cannot change it because you can't understand if it is relevant for the simulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "These are just some examples, really happened in real life, and are sources of stress that can be easily avoided\n",
    "\n",
    "What we are going to learn do not negate complete and utter misfortune, but can limit the negative effects of the disasters\n",
    "\n",
    "And their are easy enough to be used without erculean efforts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main concept behind this examples and instruments is the correct management of metadata (data about data)\n",
    "\n",
    "Consider a function in your code: variable names are absolutely arbitrary, and you can replace them with anything else without changing the behavior of the function.\n",
    "But if the names are confusing, the code will be hard to understand, hard to modify and you would shortly forget what it is supposed to be doing.\n",
    "\n",
    "And there is so much more, alonside naming conventions and comments that one can keeps track of, if one is aware of it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Distributed Version control and Code Source Management"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "I'm sure all of you, when working on an important file, have various versions, called\n",
    "\n",
    "* doc_v1\n",
    "* doc_v2\n",
    "* doc_final\n",
    "* doc_final_for_real\n",
    "* etc...\n",
    "\n",
    "This is already a very rough for of version control.\n",
    "\n",
    "Thanks to the computer we can do better than this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can:\n",
    "\n",
    "* keep track of everything that happens\n",
    "* roll back to a previous version at any moment\n",
    "* visualize the difference between two versions of it\n",
    "* keep track of \"parallel world versions\" of our work\n",
    "\n",
    "and so on.\n",
    "\n",
    "The programs that allow one to do so are called Version Control Systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "These work in tandem with Source Code Management tools, that also allow to keep track of:\n",
    "\n",
    "* bugs\n",
    "* documentation\n",
    "* feature requests\n",
    "* etc..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "cit. \n",
    "\n",
    "> If it worth doing, it is worth doing under version control"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can use version control for any text-based file that you want, not just code.\n",
    "\n",
    "For example you can also keep (and I suggest you do) your thesis and your research articles under VC.\n",
    "\n",
    "One can still use version control for non text-based files, but it loses a great deal of its power.\n",
    "\n",
    "Any time you stray away from text based files you lose the opportunity to use the full power of VC (and many other tools), and I personally think that this is a huge argument toward text-based solutions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this course we will use **git** as a control version system, and the (free) website **GitHub** (https://github.com/) to do the general code management and as a central repository of our work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### full disclosure\n",
    "\n",
    "I'm not a big fan of **git**.\n",
    "\n",
    "I personally prefer a different system called **fossil** (https://fossil-scm.org).\n",
    "\n",
    "I will teach you git (and GitHub) because they are the *de-facto* industry standards, and what you will be expected to know if you collaborate with other groups.\n",
    "\n",
    "If any of you is curious about **fossil** and the diffences between it and git, I'll be happy to discuss them out of the lessons hours"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The idea behind git\n",
    "\n",
    "If you were to design a simple version control system, how would you do it?\n",
    "\n",
    "for a better written version, see http://tom.preston-werner.com/2009/05/19/the-git-parable.html\n",
    "\n",
    "### version 1\n",
    "\n",
    "> everytime you modify a file, save it as a new version with the date of edit attached to it\n",
    "\n",
    "It is basically what you are already doing, but using date instead of numbers is more explicit.\n",
    "But if you have to coordinate various files it's still a mess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### version 2\n",
    "> save a snapshot of the whole directory everytime you modify something, with the date in the name\n",
    "\n",
    "good for keeping various file together, but a huge waste of file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### version 3\n",
    "\n",
    "> go back to version 1, but keep a document (the manifest) with written what is the current state of the directory\n",
    "\n",
    "We are already 90% of the road toward a real version control system, but how would I know the state in previous moments in time?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### version 4\n",
    "\n",
    "> keep several manifests, each one referencing the one before it and what changed between them\n",
    "\n",
    "99% of the way now... how do you manage collaboration and such?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### version 5\n",
    "\n",
    "> Identify each file and manifest with a unique hash, store the various versions of the various files in a hidden subdirectory together with the manifests, build the directory based on those, exchange the blobs (files) and manifests with other people\n",
    "\n",
    "Now you have basically a simple implementation of git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ideas behind git\n",
    "\n",
    "* you have to choose which files to follow, and which files to ignore\n",
    "* everytime you change one or more files, you have to **commit** them to the VC, to let it know to create a new snapshot\n",
    "* once in a while, you have to sync with the remote server (GitHub)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using Git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**git** is based on subcommands.\n",
    "\n",
    "This means that all the operations you need to do are going to be called as:\n",
    "\n",
    "    git <some_command> command options\n",
    "    \n",
    "some commands will be:\n",
    "\n",
    "* commit\n",
    "* push/pull\n",
    "* diff\n",
    "* history\n",
    "* etc..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Setting up git    \n",
    "\n",
    "First thing, tell git who you are.\n",
    "\n",
    "This info are required, and will be attached to all the commits you do\n",
    "    \n",
    "    git config --global user.name \"Enrico Giampieri\"\n",
    "    git config --global user.email \"enrico.giampieri@unibo.it\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### creating the repository\n",
    "\n",
    "I need to create an empty folder to host my project\n",
    "\n",
    "    mkdir myproject\n",
    "    cd myproject\n",
    "    \n",
    "and now I can initialize the git repository\n",
    "    \n",
    "    git init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### obtain the current situation of the repository\n",
    "\n",
    "tells you which files are new, which are modified, and which have been deleted\n",
    "\n",
    "    git status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Adding a file to the one you follow\n",
    "    \n",
    "    git add myfile.txt\n",
    "    git status\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Commit the snapshot to the history of the repository\n",
    "    \n",
    "commit all the changes\n",
    "    \n",
    "    git commit -a -m \"added the first file\"\n",
    "    \n",
    "you can also add one or more modified files again to tell git that you want to keep trackof only those changes, and then do\n",
    "\n",
    "    git commit -m \"my commit only of the staged modifications\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### show the history of the current branch\n",
    "\n",
    "for each commit it shows the message and the SHA (hash) of the commit\n",
    "\n",
    "    git log\n",
    "    \n",
    "all the commits that are related to a specific file\n",
    "\n",
    "    git log myfile.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### show the edits\n",
    "\n",
    "after you edit your file, you chan check what has been changed for each file\n",
    "\n",
    "    git diff myfile.txt\n",
    "    \n",
    "or for the whole directory\n",
    "\n",
    "    git diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### edit on the history\n",
    "\n",
    "differences between the file and the last commit (**HEAD**)\n",
    "\n",
    "    git diff HEAD myfile.txt\n",
    "    \n",
    "differences between the file and the commit before the HEAD (or the one even before that)\n",
    "\n",
    "\n",
    "    git diff HEAD~1 myfile.txt\n",
    "    git diff HEAD~2 myfile.txt\n",
    "    \n",
    "difference with a specific commit from the history\n",
    "\n",
    "    git diff <commit SHA1> myfile.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### revert back history\n",
    "\n",
    "replace the current version of the file with a previous one from an older commit\n",
    "\n",
    "    git checkout <commit SHA1> myfile.txt\n",
    "    \n",
    "    \n",
    "You could revert the whole repository if you forget the `myfile.txt`, so please the careful!!\n",
    "\n",
    "if you want to go back to the latest commit you did, you can write:\n",
    "\n",
    "    git checkout master\n",
    "    \n",
    "beware that in both cases you will lose all the edits you have done and not commited!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### removing and renaming a file\n",
    "\n",
    "    git rm <oldfile>\n",
    "    git mv <oldfile> <newfile>\n",
    "    \n",
    "watch out because git will do the modification and keep track of it.\n",
    "\n",
    "you can always revert the changes by doing\n",
    "\n",
    "    git checkout HEAD <oldfile>\n",
    "    \n",
    "git will explain precisely what to do to revert your changes, just remember to always check the status of the repo before committing!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### sync with the main server - setting up the remote\n",
    "\n",
    "The big advantage of distributed version control systems such as git is that you can store your repository on a remote server that multiple people can collaborate on.\n",
    "\n",
    "first thing you have to set the remote server location (such as github), with one of two version depending if it is a public location or behind an ssh login.\n",
    "\n",
    "    git remote add <origin> ssh://login@IP/path/to/repository\n",
    "\n",
    "    git remote add <origin> http://IP/path/to/repository\n",
    "    \n",
    "you can have multiple remotes, and each one will have to be synced independently. **origin** is the most common name.\n",
    "\n",
    "you can also duplicate an existing repository in a clean folder with \n",
    "\n",
    "    git clone http://IP/path/to/repository\n",
    "    \n",
    "this will automatically also set up the remote for you\n",
    "\n",
    "to see the list of currente remotes, you can use:\n",
    "\n",
    "    git remote -v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### note - updating the origin\n",
    "\n",
    "If you want to change the location for a remote, such as origin, of your repository, you have to update it, don't try to use again `remote add`\n",
    "\n",
    "    git remote set-url origin <link to repo>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### sync with the main server - push and pull\n",
    "\n",
    "once you have the remote set up, everytime you want to modify the project, you should\n",
    "\n",
    "    git pull origin master\n",
    "    \n",
    "to get the latest updates from the other authors.\n",
    "After you do your commits, you can\n",
    "\n",
    "    git push origin master\n",
    "    \n",
    "to share it with them.\n",
    "\n",
    "**origin** is the name of the remote to which you are syncing, if you use a different name, work accordingly\n",
    "\n",
    "**master** is the name of the main branch of commits (the possible parallel realities you can work on).\n",
    "If you work on different branches, change them accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### sync with the main server - merging conflicts\n",
    "\n",
    "If you and another author modify the same file, git will try to be smart and merge by itself the edits, as long as they involve different part of the file.\n",
    "\n",
    "If you are editing the same positions, it will complain, **stop you from committing** and ask you to **solve it by hand**, giving you a modified version of the file that highlight the stuff that you need to merge by hand.\n",
    "\n",
    "To limit this, try to keep each commit nice and well focused, so that you don't risk modifying random stuff in a file and risk a merge conflict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](https://github.com/louim/in-case-of-fire/blob/master/in_case_of_fire.png?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ignoring stuff\n",
    "\n",
    "sometimes you will have some files you don't want to be notified about, for example chaches, temporary files, etc...\n",
    "\n",
    "For these files, you can silence all the notifications by creating a `.gitignore` file, that is a list of filters that will be used to not show the files.\n",
    "\n",
    "for example, a `.gitignore` file could be:\n",
    "\n",
    "    *.temp\n",
    "    *.cache\n",
    "    \n",
    "you have to leep the .gitignore under version control like any other file!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Alternative histories - branching\n",
    "\n",
    "Branching is the process by which you can create alternative realities in your repository to experiment without messing up other people's work.\n",
    "\n",
    "it can lead to some real mess, so use them sparingly..."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}