- Summary
- Development VM
- Install Development VM
- Troubleshooting
- TO VMS Developers
This git project contains vagrant virtualbox's Vagrantfile to help you work in development with Ubuntu Server + salt-stack + docker. It also contains some Docker images.
This schema should help you visualize interactions between the VM and the development host
For vagrant images, we provide on specific branch those boxes:
- master: Ubuntu saucy / 64 bits Ubuntu
- vagrant-ubuntu-1304-raring64: Ubuntu raring / 64 bits
- vagrant-ubuntu-lts-precise64: Ubuntu raring / 64 bits
- vagrant-debian-7-wheezy64: Vagrant box for Debian wheezy 7.2 / 64 bits
For docker, we use a docker subfolder with the appropriate stuff to build the base docker images insides.
WARNING You need to comment out all the /etc/apparmor.d/usr.bin.ntpd profile and do sudo invoke-rc.d apparmor reload
- makinacorpus/ubuntu: minimal ubuntu system
- makinacorpus/ubuntu_salt: ubuntu + salt master + salt minion
- makinacorpus/ubuntu_mastersalt: ubuntu + salt master + salt minion + mastersalt minion
- makinacorpus/ubuntu_deboostrap: ubuntu deboostrapped
- makinacorpus/debian: minimal debian system
- debian-7.2.0-amd64: base vagrant box for the official makinacorpus/vms debian based vagrant box
Following theses instructions you can install this git repository on a directory of your local host, then start a Virtualbox vm from this directory. this virtualbox VM handled by vagrant will then run the docker VMs. All files used in the VirtualBox VM and in the docker mounts will be editable from your host as this VM will ensure your current user will be member of the right group, shared with the VM, and that all important files used by the vm are shared from your development host via nfs
You need to have virtualbox, vagrant (with vagrant-vbguest plugin) and sshfs.
On macosx, sshfs is also known as MacFusion.
By default file transferts between host and guest is really, really slow. We have improved performances by some techniques:
- Increasing the MTU to 9000 (jumbo frames) on host and guest Ethernet nics
- Leaving most of files on the guest side, leaving up to you to access the files on the guest. We recommend and also integrate this access to be via sshfs. On previous versions tests were made with NFS, having project files stored on the host and shared in the guest. This was too slow for read-heavy services like salt and plone, for example, so we finally choose to share files from the guest to the host.
Install Oracle Virtualbox at at least the 4.3 version and more generally the most up to date virtualbox release. Check https://www.virtualbox.org/ for details.
Typically on Debian and Ubuntu:
wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add - if [[ -f /etc/lsb-release ]];then . /etc/lsb-release;distrib="$DISTRIB_CODENAME"; elif [[ -f /etc/os-release ]];then . /etc/os-release;distrib="$(echo $VERSION|sed -re "s/.*\((.*)\)/\1/g")";fi echo "deb http://download.virtualbox.org/virtualbox/debian $distrib contrib">/etc/apt/sources.list.d/vbox.list apt-get update apt-get install virtualbox-4.3
On MacOSX, Install http://download.virtualbox.org/virtualbox/4.3.6/VirtualBox-4.3.6-91406-OSX.dmg
You could make you a supersudoer without password to avoid sudo questions when lauching the VMs (not required):
# visudo # Allow members of group sudo to execute any command %sudo ALL=(ALL:ALL) NOPASSWD:ALL
For a Debian / Ubuntu deb-like host:
url="https://dl.bintray.com/mitchellh/vagrant/vagrant_1.4.3_x86_64.deb";wget "$url" sudo dpkg -i vagrant_1.3.5_x86_64.deb
For macosx, use https://dl.bintray.com/mitchellh/vagrant/Vagrant-1.4.3.dmg
IMPORTANT THE VBGUEST PLUGIN, to sync the guest addition packages from your host virtualbox version:
vagrant plugin uninstall vagrant-vbguest vagrant plugin install vagrant-vbguest
- Install your sshfs distribution package (surely sshfs).
- Relog into a new session or reboot
- Remove old unsupported sshfs:
- uninstall sshfs & osxfuse from brew if you did installed it
- uninstall sshfs from MacFusion if any
- uninstall sshfs from MacFuse if any
- Install osxfuse & sshfs from osxfuse
Take care with this part, it can prevent your system from booting.
On MacOSX, edit /etc/sysctl.conf
add or edit a line:
kern.aiomax=2048 kern.aioprocmax=512 kern.aiothreads=128Reload the settings:
sysctl -pOn linux, edit /etc/sysctl.conf
add or edit a line:
fs.aio-max-nr = 1048576 fs.file-max = 6815744Reload the settings:
sysctl -p
Now you can start the vm installation with vagrant. Note that this repository will be the base directory for your projects source code managment.
You will have to use ./manage.sh, a wrapper to vagrant in the spirit but do much more.
Take a base location on your home:
mkdir -p ~/makina/ cd ~/makina/
Get this project in the vms subdirectory of this base place note that you can alter the last name to choose another directory:
git clone https://github.com/makinacorpus/vms.git vms cd vms
Alternatively if you want the precise64 LTS ubuntu server use:
git clone https://github.com/makinacorpus/vms.git -b vagrant-ubuntu-lts-precise64 vms-precise cd vms-precise
Or for Debian (see that the last word is up to you, it's the destination directory):
git clone https://github.com/makinacorpus/vms.git -b vagrant-debian-7-wheezy64 vm-debian cd vm-debian
start the VM a first time, this will launch the base vm download from DNS, then VM creation and provisioning:
./manage.sh init
You will certainly need one or to reload to finish the provision steps (normally the first time, the script do it for you) but to do it on your own you could use:
./manage.sh reload
Now that vagrant as created a virtualbox image for you, you should always manipulate this virtualbox VM with ./manage.sh command and use directly vagrant at last resort.
Please note that when the vm is running, we will try to mount the VM root as
root user with sshfs in the ./VM folder.
To launch a Vagrant command always cd to the VM base directory:
cd ~/makina/vms
Initialising from scratch (low level base iOS mage) rather than from a preconfigured makina corpus image:
./manage.sh up
Starting the VM after creation is indeed the same command, but use the preconfigured VM under the hood if already initialized:
./manage.sh up
Stoping the VM can be done like that:
./manage.sh down # classical ./manage.sh suspend # faster on up, but requires disk space to store current state
Reloading the vm is:
./manage.sh reload # with sometimes tiemout problems on stop, redo-it.
To remove an outdated or broken VM:
./manage.sh destroy
You can tweak some settings via a special config file: vagrant_config.rb
- Read the Vagrantfile top section, containing VM cpu and memory settings and even more.
- From there, as explained, you should create a .vagrant_config.rb file, to alter what you need.
For exemple, you can clone the vms git repository on another place where you can manage another vagrant based virtualbox vm.
Take not that it will provision the base vm of the template and not the running VM. If you want a full clone, use export & import.
To create a new vm from an already existing one is damn easy
cd ~/makina/<VM-TEMPLATE> ./manage.sh clonevm /path/to/a/new/vm/directory
- lasting Slash are importants with rsync
cd ~/makina/ rsync -azv --exclude=VM --exclude="*.tar.bz2" <VM-template>/ <NEW-VM>/ cd <NEW-VM> ./manage reset && ./manage init ../<VM-TEMPLATE>/<devhost_master*tar.bz2> # the downloaded archive at init time
mkdir -p ~/makina/ cd ~/makina/ # get this project in the vms subdirectory of this base place git clone https://github.com/makinacorpus/vms.git vm2 cd vm2 or c
m ID and Subnet.
You must read at least once the Vagrantfile, it will be easier for you to know how to alter the vm settings. Such settings can go from MAX_CPU_USAGE_PERCENT,CPUS & MEMORY settings. to more useful: change this second v
You will indeed realise that there is a magic DEVHOST_NUM setting (take the last avalaible one as a default).
You can then this settings, along with the other settings in vagrant_config.rb .
By default this file is not yet created and will be created on first usage. But we can enforce it right before the first vagrant up:
cat > vagrant_config.rb << EOF module MyConfig DEVHOST_NUM="22" end EOF
This way the second vagrant VM is now using IP: 10.1.22.43 instead of 10.1.42.43 for the private network and the docker network on this host will be 172.31.22.0 and not 172.31.42.0. The box hostname will be devhost22.local instead of devhost42.local.
You can tell to the provision script to run system updates and reprovision salt entirely by setting the DEVHOST_AUTO_UPDATE setting to true.
We add the hosts presents in the VM to the /etc/hosts of the host at up & reload stages (you ll be asked for)
Read makina-states.nodetypes.vagrantvm if you want to know which hostnames are exported.
You can optionnaly sync those hosts with:
./manage.sh sync_hosts
We have made a wrapper similar to
vagrant ssh.but this one use the hostonly interface to improve transfer and shell reactivity.
We also configured the vm to accept the current host user to connect as root and vagrant users.
Thus, you can sonnect to the VM in ssh with either
rootor thevagrantuser (sudoer) is:./manage.sh ssh (default to vagrant)
or:
./manage.sh ssh -l root
To export in package.tar.bz2, to share this development host with someone:
./manage.sh export
To import from a package.tar.bz2 file, either:
- Give an url to the archive
- Give an absolute path to the archive
- place the archive in ./package.box.tar.bz2
Then issue:
./manage.sh import [ FILE_ARCHiVE | URL | ./package.box.tar.bz2 ]
Note that all the files mounted on the /vagrant vm directory are in fact stored on the base directory of this project.
Time to time, it can be useful to regain free space by deleting old imported devhost base boxes, list them:
vagrant box list
Look for lines beginning by devhost-. None of those boxes are linked to your running vms, you can safely remove them.
You can then delete them by using:
vagrant box remove <id>
- /mnt/parent_home: Host user Home folder
- /vagrant/share:
Current working directory/sharein the host (where ./manage.sh up has been done - /vagrant/packer:
Current working directory/packerin the host (where ./manage.sh up has been done - /vagrant/docker:
Current working directory/dockerin the host (where ./manage.sh up has been done - /vagrant/vagrant:
Current working directory/vagrantin the host (where ./manage.sh up has been done - /srv/salt: Salt state tree
- /srv/projects: makina Salt projects installation root
- /srv/pillar: Pillar data
- To edit or access the files from your host system, youn ll just have to use ./VM
which is a mountpoint for the``/`` of the vm exported from the vm as the root user.
- For example, you can configure <here>/VM/srv/projects/foo as the project
workspace root for your eclipse setup.
- You should do git or large operations from within the VM as it will not use the shared network and will be faster
At each vm access
We copy to the root and vagrant users:
- the current user ssh-keys
- the current user ssh-config
We copy vagrant authorized_keys to root/.ssh.
All of this is managed in /vagrant/vagrant/install_keys.sh
This allow you from the host:
- To log as vagrant or root user
- To mount the guest filesystem as root (used in the core setup)
- git push/pull from the guest as if you were on the host
If your project has custom users, just either (via saltstates):
- copy the vagrant ssh keys to your user $HOME
- Use an identity parameter pointing to the vagrant key pair
If the provision script of the vm halt on file share mounts you will have to check several things:
do you have some sort of firewalling preventing connections from your host to the vm? Maybe also apparmor or selinux?
do you have a correct /etc/hosts with a first 127.0.[0|1].1 record associated with localhost name and your short and long hostname?
did you clone this repository in an encrypted folder (e.g.: home folder on Ubuntu)?
On Mac OS X you can try sudo nfsd checkexports
try to run the commands but do prior to that:
export VAGRANT_LOG=INFOtry to run sudo exportfs -a for more debug information on host side.
On Mavericks, you may encounter several issues, usually you need at least to reinstall virtualbox:
There was an error while executing VBoxManage: hashicorp/vagrant#1809 try to usesudo launchctl load /Library/LaunchDaemons/org.virtualbox.startup.plist(4.3) andsudo /Library/StartupItems/VirtualBox/VirtualBox restart(before)There was an error executing the following command with VBoxManage: ["hostonlyif", "create"]: http://stackoverflow.com/questions/14404777/vagrant-hostonlyif-create-not-working- shutdown problems: https://www.virtualbox.org/ticket/12241 you can try
VBoxManage hostonlyif remove vboxnet0
Their use is to facilitate the learning of docker and to mitigate current installation issues by providing a ready-to-use docker+salt virtualised host. This vagrant Virtualbox management can be also used without Docker usage.
Master branch of this repository is using an Ubuntu Saucy Vagrantfile VM. Check other branches to find LTS precise versions.
check the Install part on this documentation for installation instructions
Notes for specific ubuntu release packages:
Run ./manage.sh release which will at once:
- Edit and increment version.txt's version
- Do a snapshot of the current vm to the desired release name (devhost-$branch_$ver.tar.tbz2)
- Upload the tarball to the CDN, actually sourceforge
All the images are constructed from ubuntu cloud archives images.
- Recent Virtualbox
- Linux hardware enablement stack kernel (3.8)
As of now, we needed to backport those next-ubuntu stuff (saucy) for things to behave correctly and efficiently:
- Lxc >= 1.0b
- Kernel >= 3.11
- Virtualbox >= 4.2.16
Mainline packages
Mainline packages
- We maintain some handmade Packer images from the official debian netinst iso
- (see packer subdir)
For packer, we use a docker subfolder with the appropriate stuff to build the base docker images insides. Goal is to use packer to construct base images for the vagrant ones when there are no base images avalaible from trusted sources.
apt-get -t wheezy-backports install linux-image-3.10-0.bpo.3-amd64 linux-headers-3.10-0.bpo.3-amd64 initramfs-tools
- Contruct base environments to work with docker. (kernel, aufs, base setup)
- Install a functional makina-states installation inside in
servermode - Whereas the single process docker mainstream approach, we want to use the init systems
providen by the underlying distribution to manage a bunch of things.
Goal is to have in working state:
- init system
- cron
- logrotate
- sshd
- sudo
- syslog
- screen
- makina-states in server mode (vm)
For now you need docker from git and lxc from git also to fix: - moby/moby#2278 - moby/moby#1960
You can install them in the vm with
vagrant ssh sudo su cd /srv/docker ./make.sh inst
And uninstall them with
vagrant ssh sudo su cd /srv/docker ./make.sh teardown
Some weird changes can happen in Vagrant file
Say that you want to put master commits in debian (vagrant-debian-7-wheezy64) branch:
git merge --no-commit --no-ff -e origin/master
Verify and discard or merge any changes to Vagrantfile:
git diff --cached Vagrantfile
Discard:
git show origin/vagrant-debian-7-wheezy64>Vagrantfile git add Vagrantfile
2 ways merge:
git show origin/master>Vagrantfile.a git show origin/vagrant-debian-7-wheezy64>Vagrantfile vimdiff Vagrantfile.a Vagrantfile git add Vagrantfile
commit && push the result:
git commit && push
Install your OS NFS server
Edit vagrant_config.rb and set
DEVHOST_HAS_NFS=true.The important thing here is to tuneup the number of avalaible workers for nfs server operations.
- NOTE: [RECOMMENDED] 256 threads == ~512MO ram allocated for nfs
- NOTE: 128 threads == ~302MO ram allocated for nfs
- 512 is a lot faster but the virtualbox ethernet interfaces had some bugs (kernel guest oops) at this speed.
On Debian / Ubuntu:
Install nfs:
sudo apt-get install nfs-kernel-server nfs-common portmap virtualbox
Edit /etc/default/nfs-kernel-server and increase the RPCNFSDCOUNT variable to 256.
Restart the server:
sudo /etc/init.d/nfs-kernel-server restart
On Archlinux:
Edit /etc/conf.d/nfs-server.conf and increase the NFSD_COUNT variable to 256.
Enable at boot / Restart the services:
modprobe nfs # may return an error if already loaded for i in rpc-idmapd.service and rpc-mountd.service nfsd.service;do systemctl enable $i service $i start done
On MacOSX:
- Edit /etc/nfs.conf and increase the nfs.server.nfsd_threads variable to 512 or 256.
- Select, active & restart the NFS service in server admin
For Vagrant you need to have a recent Vagrant version (vagrant is a virtualbox VM manager, to make it simple). But version 1.3.4 is broken, so use 1.3.3 or 1.3.5 or greater. Get latest vagrant from official download site, where you can find msi, dmg, rpm and deb packages.