Web Historian Standalone

This repository contains scripts to convert the Web Historian browser extension to a standalone executable.

Background

Online and mobile news consumption leaves digital traces that are used to personalize news supply, possibly creating filter bubbles where people are exposed to a low diversity of issues and perspectives that match their preferences. The JEDS Filter Bubble project aims to understand the filter bubble effect by performing deep semantic analyses on mobile news consumption traces. This project is a collaboration between the VU, the UvA and NLeSC, lead by Wouter van Atteveldt.

Part of this project makes use of the Web Historian browser extension, developed by Ericka Menchen-Trevino. In some cases, it is beneficial to run this extension as a standalone executable. For example, Safari does not support the WebExtensions API. This project allows Web Historian to run as a MacOS app for the Safari browser. Another benefit of this method is that it gives access to additional entries in the browser history database. For example, in Chrome, the device origin of a URL is not accessible via the WebExtensions API, but it is via a direct query on the database. This project uses Python to orchestrate the code, SQL to query the browser database and a JavaScript web extension shim, which implements some of the WebExtensions API calls, to allow the extension to run as a local web page.

Installation

Prerequisites

• Clone the repository and its submodules using git clone --recurse-submodules.
• Install pyinstaller using pip install pyinstaller.

Build the app

• Configure your edition of Web Historian in the submodule directory. Currently, the only supported edition is the Community edition. Standalone versions of Web Historian Habits or Web Historian Education are possible in principle, but currently not implemented.
• Run pyinstaller Build[OS][Browser][WebHistorianEdition].spec where [OS] is either Windows or Mac, [Browser] is only Chrome for Windows and either Chrome or Safari for Mac and, finally, the choice for [WebHistorianEdition] is only Community. These choices may be extended in the future.
• When building the app on Mac, also run Disk Utility and choose File > New Image > Image from Folder (see here). Then select the directory with the Web Historian app. This will generate a .dmg file which allows the app to be distributed more easily.

Additional commands

• Run git submodule update --init --recursive to initiate and update all submodules if you forgot during the clone.
• Run git submodule foreach git pull origin master to pull the latest commit for each submodule.

Repository content

Web-Extension-Shim

This directory contains the Web Extension Shim submodule which contains a JavaScript file that implements some of the WebExtensions API calls.

Web-Historian-Community

This directory contains the Web Historian - Community Edition submodule which contains all the source code of the Web Historian Community Edition browser extension.

Web-Historian-Education

This directory contains the Web Historian - Education Edition submodule which contains all the source code of the Web Historian Education Edition browser extension.

Web-Historian-Habits

This directory contains the Web Historian - Habits Edition submodule which contains all the source code of the Web Historian Habits Edition browser extension.

Build[OS][Browser][WebHistorianEdition].spec

These files contain the instructions for pyinstaller to build the application.

FullDiskAccess.gif

This GIF shows how to add the app to the Full Disk Access list and grant it permission to access browser history data on MacOS.

FullDiskAccess.html

This webpage explains how to add the app to the Full Disk Access list and grant it permission to access their browser history database on MacOS.

GenerateStandalone.py

This Python script copies the relevant files to a temporary location on the user's computer. It also extracts the browsing history from their browser and converts this to JSON files. For Mac users, if the script cannot get access to the browsing history, the FullDiskAccess.html page is launched in the user's default browser. If everything goes well, the default browser is launched with a standalone version of Web Historian including the browsing history.

icon-128.[icns/ico]

This icon (.icns for Mac, .ico for Windows) contains the logo of the Web Historian browser extension which is used for the app.

LoadJson.js

When running a web page with JavaScript locally, many browsers disable the ability to load JSON data files. The jQuery get function, which is used in Web Historian, would give an error. This file overrides the get function and bypasses the loading of JSON files by defining the data as variables and passing these on directly.

Standalone[OS][Browser].py

These Python scripts contain the configurations to create standalone versions of Web Historian for various OS's and various browsers. It imports the GenerateStandalone.py script and subsequently runs the generateStandalone function.