Skip to content

Makefiles that will download and setup a local wikipedia instance.

Notifications You must be signed in to change notification settings

infolab-csail/wikipedia-mirror

Repository files navigation

Usage

See the examples and work it out ;)

Building everything

Basically all you want to do is load some sql files into an sql database and let the dependency tree find it’s way into doing that. So if all goes well you should be able to have everything set up with:

make sql-load-parts ROOT_DIR=/new/path/of/proj/root

Make sure the media on which ROOT_DIR is mounted has more than 150G and is preferably not a network fs or it will be slow as hell.

Troubleshooting

Wikipedia is known to put some strangeness into utf that mwdumper does not handle well. If the make process fails you can fix the file that failed with

make fix-<file-that-failed>

which is an alias to

make <filename that failed without .bz2 extension>.fix.xml

This will extract the xml and replace all invalid characters with spaces. Then rerun sql-load-ports

Note: you should still define ROOT_DIR when running make in this way if you are building outside of the project dir.

Installing Bitnami MediaWiki

Downloads and installs bitnami

make bmw-install

To have the downloaded stuff and the install dir in another path use

make bmw-install ROOT_DIR=/new/path/of/proj/root

NOTE: ROOT_DIR is not persistent. The default is the current dir and to run other commands based on an installed env keep redefining ROOT_DIR.

Uninstalling Bitnami MediaWiki

make bmw-uninstall

Running Bitnami

This will run apache and sql etc. To do any db operation be sure that it is running.

make bmw-run

This kills (a bit blindly) anything that resembles bitnami if it fails to halt gracefully.

make bmw-kill

Mysql

A mysql shell interface to the database.

make mysql-shell

About

Makefiles that will download and setup a local wikipedia instance.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published