Skip to content
forked from ku1ik/bitpocket

Bitpocket two way sync with rsync (fork maintained by Faster IT)

License

Notifications You must be signed in to change notification settings

fasterit/bitpocket

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 

Repository files navigation

bitpocket

Originally cloned from git://github.com/sickill/bitpocket.git
Hosted at https://git.faster-it.com/bitpocket
Mirrored at https://github.com/fasterit/bitpocket

About

This is a fork of @sickill/bitpocket we used internally since 2014. While there are quirks we have found this tool to be very useful for controlled multi-device syncing for tech-savy users.

We use it cross-platform for Linux, Windows/cygwin (bitpocket) and Mac OSX with homebrew (bitpocket_mac).

If you are new to bitpocket:

bitpocket is a small but smart script that does 2-way directory synchronization. It uses rsync to do efficient data transfer and tracks local file creation/removal to avoid known rsync problem when doing 2-way syncing with deletion.

bitpocket can use any server which you have ssh access to for its central storage.

Installation

Clone repository and symlink bitpocket (or the bitpocket_mac) bin to somewhere within your $PATH:

$ git clone git://github.com/fasterit/bitpocket.git
$ ln -s `pwd`/bitpocket/bitpocket ~/bin/bitpocket

Or download script and place it in a directory in your $PATH:

$ curl -sL https://raw.github.com/fasterit/bitpocket/master/bitpocket > ~/bin/bitpocket
$ chmod +x ~/bin/bitpocket

For MacOSX you need homebrew and have coreutils and gnu-sed installed ... eh ... "brewed".
The BSD versions of coreutils and sed that MacOSX ships are just way too old to be useful.

Setting up master

Create an empty directory on some host that will be the master copy of your files:

$ ssh user@example.org
$ mkdir ~/BitPocketMaster

Setting up slaves

On each machine you want to synchronize initialize an empty directory as your bitpocket:

$ mkdir ~/BitPocket
$ cd ~/BitPocket
$ bitpocket init user@example.org ~/BitPocketMaster

Usage

After installation, you use the bitpocket command for synchronization and other tasks. Running bitpocket help will display the following message.

usage:  bitpocket [sync | push | pull | pack | log | cron | list | help]
        bitpocket init {<REMOTE_HOST> | ""} <REMOTE_PATH>

sync    Run the sync process. If no command is specified, sync is run by default.
push    Only push new files to the server.
pull    Only pull new files from the server.
init    Initialize a new bitpocket folder. Requires remote host and path params.
pack    Pack any existing (automatic) backups into a git repository.
cron    Run sync optimized for cron, logging output to file instead of stdout.
log     Display the log generated by the cron command
list    List all files in the sync set (honoring include/exclude/filter config).
help    Show this message.

Note: All commands (apart from help), must be run in the root of a
      new or existing bitpocket directory structure.

Manual sync (bitpocket sync)

To synchronize your local slave with master just run bitpocket sync inside your bitpocket directory:

$ cd ~/BitPocket
$ bitpocket sync

Ensure that you run bitpocket at least once immediately after creating a new slave and before adding new files to the slave directory. If there are files in the master they will be pulled into the slave. You may then move files into your slave directory and they will be detected as added.

To improve the safetely of daily operations the Faster IT version of bitpocket has patches added which allow to only push or pull.

So

$ cd ~/BitPocket
$ bitpocket pull

will only pull from the server and leave files there unclobbered. Similarly bitpocket push will sync new files to the server but not risk local files when you are in a hurry.

Maintaining backups (bitpocket pack)

bitpocket does not include a full-fledged versioning system at the moment, but it does automatically create a local backup of any files before overwriting or deleting with changes from the BitPocketMaster. These backups are placed into a timestamped directory (one directory per sync). The path to this directory is:

.bitpocket/backups/YYYY-MM-DD.hhmmss

As these files accumulate, you may want to remove them, but you can also run bitpocket pack to combine all the backup files into a git repository contained within the .bitpocket directory:

$ cd ~/BitPocket
$ bitpocket pack

This requires an installation of git, but allows all the space saving advantages of git when making repeated changes to the same files.

There is a discussion about potential directions for versioning direction in the issue 15 of sickill's original bitpocket git: github.com/sickill/bitpocket/issues/15
(tl;dr version: The discussion wasn't conclusive.)

Redirecting output to log file (bitpocket cron)

If bitpocket is run with the cron parameter ( bitpocket cron ), it will perform a sync, but instead of showing the progress on stdout, it will redirect all output to a log file:

$ cd ~/BitPocket
$ bitpocket cron

As the name of this parameter implies, this is mainly useful when running bitpocket through the cron command. (See "Automatic sync with cron" for more information about how to configure this).

Displaying logs (bitpocket log)

When running bitpocket in cron with bitpocket cron it will append its output to .bitpocket/log file. You can review the tail end of an existing log file, or watch live log as it is generated, with following command:

$ cd ~/BitPocket
$ bitpocket log

Displaying list of files to be synchronized (bitpocket list)

You may want to know which files will be synchronized before actually performing the syncronization. You can verify which files are in the synchronization set by running bitpocket list:

$ cd ~/BitPocket
$ bitpocket list

Note that this does not list changed files, it only lists all the local files that bitpocket will look at in determining which files to sync. Also, note that if there are new files in the master that will be added on a sync, they will not be included here. This command is only intended to verify which files are in the synchronization set. (See "Configuring file exclusion and inclusion" for information about how to control which files are in the synchronization set).

Configuration

Automatic sync with cron

Add following line to your crontab to synchronize every 5 minutes:

*/5 * * * * cd ~/BitPocket && nice ~/bin/bitpocket cron

Note that cron usually has very limited environment and your ssh keys with passphrases won't work in cron jobs as ssh-agents/keyrings don't work there. Thus it's preferable to generate passphrase-less ssh key for bitpocket authentication:

$ cd ~/BitPocket
$ ssh-keygen -t rsa -C bitpocket-`hostname` -N '' -f .bitpocket/id_rsa
$ ssh-copy-id -i .bitpocket/id_rsa user@example.org

and uncomment line with RSYNC_SSH in .bitpocket/config file.

Custom rsync options

You can pass additional switches to rsync by setting RSYNC_OPTS in .bitpocket/config file. Generated config file includes (commented out) example setting for dereferencing symlinks:

# RSYNC_OPTS="-L"

Just uncomment it and change at will.

Configuring file exclusion and inclusion

If you want some files to be ignored by bitpocket you can create .bitpocket/exclude file and list the paths there:

*.avi
jola
/misio.txt

*.avi and jola will be matched anywhere in path, misio.txt will be matched at bitpocket root dir ( ~/BitPocket/misio.txt ).

This exclude file is passed to rsync as --exclude-from argument, check man rsync for INCLUDE/EXCLUDE PATTERN RULES.

You can set up even more advanced exclusion/inclusion rules. In all, there there are three files that you can create to change this configuration:

.bitpocket/exclude
.bitpocket/include
.bitpocket/filter

Be aware that all the quirks from rsync exclusion/inclusion rules carry over into bitpocket. If you decide that you need such advanced configuration, make sure that you understand those rules very well, and consider double checking them before syncing by running bitpocket list.

Slow sync callbacks

When syncing takes more than 10 seconds (SLOW_SYNC_TIME setting) bitpocket can fire off user provided command in background. This can be usefull to notify user about long sync happening, preventing him from turning off the machine during sync etc.

There are 3 settings that can be enabled in .bitpocket/config file:

# SLOW_SYNC_TIME=10
# SLOW_SYNC_START_CMD="notify-send 'BitPocket sync in progress...'"
# SLOW_SYNC_STOP_CMD="notify-send 'BitPocket sync finished'"

Just uncomment them and change at will.

You can show tray icon during long sync with traytor and following settings:

SLOW_SYNC_START_CMD='~/bin/traytor -t "BitPocket syncing..." -c "xdg-open ." .bitpocket/icons & echo $! >.bitpocket/traytor.pid'
SLOW_SYNC_STOP_CMD='kill `cat .bitpocket/traytor.pid`'

Failsafes

You can add a remote mount point check to ensure a remote path is available prior to the sync running. If the folder is not mounted on the remote server, then the sync will be aborted. The path must be an absolute path which is expected to be a mountpoint on the remote server. Add or change this lines in your .bitpocket/config file:

# REMOTE_MOUNTPOINT=/

Authors

About

Bitpocket two way sync with rsync (fork maintained by Faster IT)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 100.0%