Providing all the past releases via GitHub Pages

[peterjc]

PyPI has our (recent?) *.tar.gz releases, and in future we'd like to do wheels there too.

However, can we and should we simply host all the old *.tar.gz, *.zip, *.exe and recently *.msi files via GitHub pages?

[peterjc]

I've created https://github.com/biopython/DIST which will handle the DIST/ folder as a separate repository; this seems more elegant than forcing everything under https://github.com/biopython/biopython.github.io

(I will now rewrite the history here to remove the DIST/ files)

[peterjc]

I've uploaded the most recent Windows installers to https://github.com/biopython/DIST - taking the time to double check their MD5 checksums and note this in the commit comments. The good news is the download via GitHub pages seems to work fine:

e.g. https://biopython.github.io/DIST/biopython-1.66.win32-py3.5.msi

[vincentdavis]

@peterjc I can try working on migrating the remaining installers.

[MarkusPiotrowski]

I'm actually working on the Download page. @peterjc can you go on uploading more releases, maybe starting with the more recent ones?

[peterjc]

I've asked on the mailing list for help confirming checksums before posting the remaining releases:

• http://mailman.open-bio.org/pipermail/biopython/2016-April/015924.html
• http://mailman.open-bio.org/pipermail/biopython-dev/2016-April/021376.html

[peterjc]

Progress: I think all the *.tar.gz files are back, thanks to community validation of checksums and the Gentoo packaging manifest records.

[MarkusPiotrowski]

Can we use the validated *.tar.gz files to generate the *.zip files? Is this a simple conversion or is the content different?

[peterjc]

Good idea. I always assumed the *.zip files might be built with Windows line endings, but no - probably as these were always produced together on the same (Linux) machine, their contents seem to be the same.

Based on a single test:

$ diff -r biopython-1.61-tar-gz/ biopython-1.61-zip/ && echo "Returned $?"
Returned 0

OK, that ought to let me verify the contents of all the *.zip files against the previously validated *.tar.gz files.

[peterjc]

Other than having biopython-1.21/Tests/*.pyc in biopython-1.21.zip all the zip file contents seem to match the tar-balls :)

biopython/DIST@1a6013a

If anyone is interested, the quick-and-dirty comparison script I used was:

import os
import sys

def untar(tar, folder):
    assert os.path.isfile(tar)
    if os.path.isdir(folder):
        print("Assuming %s already untarred to %s" % (tar, folder))
        return
    os.mkdir(folder)
    cmd = 'tar -zxvf "%s" -C "%s"' % (tar, folder)
    print(cmd)
    rc = os.system(cmd)
    if rc:
        sys.exit("Error %i from %s" % (rc, cmd))

def unzip(zip, folder):
    assert os.path.isfile(zip), zip
    if os.path.isdir(folder):
        print("Assuming %s already unzipped to %s" % (zip, folder))
        return
    os.mkdir(folder)
    cmd = 'unzip "%s" -d "%s"' % (zip, folder)
    print(cmd)
    rc = os.system(cmd)
    if rc:
        sys.exit("Error %i from %s" % (rc, cmd))

def tar_vs_zip(tar, zip, stem):

    print("%s vs %s" % (tar, zip))

    assert tar == stem + ".tar.gz"
    tar_folder = "/tmp/%s-tar-gz" % stem
    untar(tar, tar_folder)

    assert zip == stem + ".zip"
    zip_folder = "/tmp/%s-zip" % stem
    unzip(zip, zip_folder)

    cmd = 'diff -r "%s" "%s"' % (tar_folder, zip_folder)
    rc = os.system(cmd)
    if rc:
        sys.exit("Error %i from %s" % (rc, cmd))

    cmd = 'git add "%s" "%s"' % (tar, zip)
    rc = os.system(cmd)
    if rc:
        sys.exit("Error %i from %s" % (rc, cmd))
    md5.extend([tar, zip])


md5 = []
for tar in os.listdir("."):
    if not tar.endswith(".tar.gz"):
        continue
    stem = tar[:-7]
    zip = stem + ".zip"
    if not os.path.isfile(zip):
        continue
    tar_vs_zip(tar, zip, stem)

cmd = "md5 %s" % " ".join(md5)
os.system(cmd)

cmd = "shasum -a 256 %s" % " ".join(md5)
os.system(cmd)

Plus use of rm /tmp/biopython-1.21-zip/biopython-1.21/Tests/*.pyc to get the diff to pass.

[peterjc]

Another contributor pointed out that archive.org actually does record binary files, and I was able to download almost all of the past releases from them and compare them to the files from the old web server - and everything matched: biopython/DIST@d5f7b01

We're still missing verification for just a handful of files:

• biopython-1.30.win32-py2.2.exe
• biopython-1.30.win32-py2.3.exe
• biopython-1.30.win32-py2.4.exe
• biopython-1.62.win32-py2.5.exe
• biopython-1.62.win32-py2.6.exe
• biopython-1.62.win32-py2.7.exe
• biopython-1.62.win32-py3.3.exe
• biopython-1.63b.win32-py2.6.exe
• biopython-1.63b.win32-py2.7.exe
• biopython-1.63b.win32-py3.3.exe
• biopython-Mac.sit

[peterjc]

Just FYI, GitHub is allowing us to host the installers, but does grumble about it:

The page build completed successfully, but returned the following warning:

It looks like you're using GitHub Pages to distribute binary files. We strongly suggest that you use releases to ship projects on GitHub. Releases are GitHub's way of packaging and providing software to your users. You can think of it as a replacement to using downloads to provide software. We found the following file(s) which may be a good candidate for releases: biopython-1.65.win32-py2.6.exe, ... For more information, see https://help.github.com/articles/about-releases/.

For information on troubleshooting Jekyll see:

https://help.github.com/articles/troubleshooting-jekyll-builds

If you have any questions you can contact us by replying to this email.

I'm expecting in future we won't need to host new releases on http://biopython.org/DIST/ and can use PyPI instead.

[peterjc]

Biopython 1.62 and 1.63b are not currently linked to from the old releases section downloads page. They were publicly announced via http://lists.open-bio.org/pipermail/biopython/2013-August/014917.html and http://lists.open-bio.org/pipermail/biopython/2013-November/014995.html so I think it was just an oversight in bfbdb5a and bfbdb5a