Skip to content
This repository has been archived by the owner on Jul 1, 2020. It is now read-only.

How To (Main questions)

Jump to bottom
PICARD Michaël edited this page Mar 26, 2019 · 1 revision

Summary

  1. Introduction
  2. Installation
  3. Quick Start
  4. How To (Main questions)
  5. Details on main scripts
  6. Database

How To (Main questions)

How to use TUXML (in one command line)?

To use TuxML in one commande line, you need to use MLfood.py, this is the script which run the compilation process and send the result to the database. You can use it on a machine with Docker like:

./MLfood.py 100 --dev

in order to run 100 new compilations on a machine with the development image. All the compilations launch are sent to the same "prod" database.

If you want to run TuxML on IGRIDA, you will need to run:

./master_spring.py --nm 20 --nc 20 --walltime 6:0:0 --nb-core 8

Here, --nm is for the number of machines to use, --nc is the number of compilations to run, --walltime is the limit of time and --nb-core, the number of core per machines to use.

It will generate the script spring.sh and run it. If you want to repeat the action you can run oarsub -S ./spring.sh, it will run the exact same thing.

Further information in the Quick Start and the Manuals, the README.md and the --help ./MLfood.py --help.

How to update the Docker image?

To update the Docker image, you will need to download the script TPDIM.py. It has been written to update easily the debian image for Docker used in TuxML.

Two cases comes:

  • To add a new package/dependence to the debian Docker image, you need to add the package name at the end of the line in the "dependencies.txt" file.
  • To add a file (script, txt, ...) to the debian Docker image, you will need to edit the script itself at this line in the write() method. Inside, there are all the ADD xxx where ADD is the Docker keyword to add the file in the Docker image. To add a file name foobar.txt in the TuxML directory in the Docker image, you will need to edit this line and add at the beginning: ADD foobar.txt /TuxML/foobar.txt\n.

Once your modifications have been done, you are ready to run TPDIM.py

./TPDIM.py -a debian -t dev

The -a stands for 'all': generate, build, push to Docker image. The debian stands for the image do update (there is only debian for now). The -t stands for 'tag'. The dev is for the Docker tag chosen, it can be 'dev' or 'prod', "dev" is the one to update when you develop, "prod" is the one to update only when you have a stable version. The "prod" image should match with the Master branch of the project.

/!\ ./TPDIM.py -a debian -t prod must be used only when a new release is made and the Master branch is updated.

How to update the IGRIDA Debian image?

On IGRIDA, we reproduce the behavior of the debian Docker image we made by using a customize debian image without Docker.

You will need to upload the new file you want to add in the location you need in the Core Repository. This repository should have the exact same content as the debian Docker image. When you update one, you need to update the other. If you update the debian Docker image, you need to do the same manually in the Core repository and the opposite too.

On IGRIDA, connect to the path where the image is (currently /temp_dd/igrida-fs1/alemasle/images/TuxMLDebian.qcow2)

cd /temp_dd/igrida-fs1/alemasle/images/TuxMLDebian.qcow2

Then use:

oarsub -I -p "virt='YES'"

Now you are redirected on the new node, run:

module load veertuosa/0.0.1
veertuosa_launch --name tuxml --image /temp_dd/igrida-fs1/alemasle/images/TuxMLDebian.qcow2 --persistent
ssh-vm tuxml

(Do not forget the --persistent to keep the changes)
You are now connected on the debian image on a virtual machine.

You can now update the directory:

cd /TuxML
git pull

You debian image on IGRIDA is now up-to-date.

How to pre-set some options (specialization)?

To specialize the compiled kernel, you can edit the file tuxml.config to add pre-set options. See the issue #98
You can now update the Docker image and the IGRIDA image.

How to compute make tinyconfig?

To compile "tinyconfig" you need to do it manually. On docker:

sudo docker run -it tuxml/tuxmldebian:dev
cd /TuxML/linux-4.13.3
make tinyconfig
cd ..
cp linux-4.13.3/.config tiny.config
./runandlog.py --path tiny.config

The script runandlog.py will run the compilation process and send the result to the database.

An other option exist in TuxML, the possibility to make a random config (randconfig) mixed with the options of tinyconfig. To use it you can run only with Docker:

./MLfood.py 100 --dev --tiny

With the --tiny option enable.

How to (re-)compile a specific configuration?

To (re-)compile a specific configuration, you can use the --path option from MLfood.py.
You can use it with the command:

./MLfood.py 1 --dev --path myconfig.config

Where is the code to instrument/investigate incremental compilation?

To investigate more about the incremental compilation, you can go in tuxml.py. This is the script which run the compilation in incremental too.
If you want to run one or more incremental compilations, you can run the command:

./MLfood.py 1 1

The first "1" is the number of docker containers, the second "1" is the number of incremental compilations. When you do not precise the incremental number, it is set at 0. A mere compilation is launch on a container. But with the incremental parameter at "1", the Docker container will run a normal compilation, then "1" incremental.

If you need 3 incremental compilations, you can increase the second number:

./MLfood.py 1 10

Here it will compile 11 times, one normal, then ten incremental each after the other.

Incremental analysis

You can use the script kernel2csv.py to make incremental compilations which are used to be compared to their non incremental version. When you run ./kernel2csv.py 1, it will create a folder in compare/ that contains the config file .config, all versions of the kernels compiled in incremental way and in a basic way.

After the script finished to compile the number asked, it will launch a comparison with the help of flash_compare.py. It compare each incremental with their non-incremental version and gives the mean of all the comparison in size and time.

flash_compare.py can be used alone too, this is the script which make comparisons of time and size.

Let's say we want to launch a new campaign on another version of the Linux kernel. What's the impact on the overall code?

We do not know what will happen with a different Linux kernel, we need to try it. To do this, you will need to download the Linux Kernel you want to try and add the full folder "linux-X.XX" in the BuildImageInter instead of "linux-4.13.3", then you replace this line by the name of your new Linux kernel folder.

Once it is done, you can rebuild the image with TPDIM.py.

Return Home

Clone this wiki locally