Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc changes and more checks #3

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Doc changes and more checks #3

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eab14f7
Doc changes
r0mflip Feb 14, 2019
a4ec363
Check for gzip
r0mflip Feb 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
160 changes: 95 additions & 65 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,76 +1,97 @@
__Scripts to copy [Rocks Clusters](http://www.rocksclusters.org) roll isos to a generic (linux) webserver for faster local hosting__

Purpose: Rocks 7.0 frontends require a web server to have rolls "installed" to download. Sometimes distance or
other issues indicate that a local copy on an existing (non-Rocks) web-server would be much better. The scripts in this
repository help with that.

There are four scripts defined in this roll
* rollcopy.sh - copy a roll iso contents for web serving
* index.cgi - cgi to present roll directory in a standard format
* httpdconf.sh (optional) - defines a virtual server httpd configuration file
* unpack-guides (optional) - unpacks roll users guides into a web-servable directory

__0. Preparation__
* clone this repository `git clone https://github.com/rocksclusters/roll-server.git`
* download the roll isos from UCSD.

__I. Copy Roll(s) to a web directory__
For this example, will assume the `/var/www/html/rocks/7.0` is the (already created) parent directory into
which you want to copy rolls. The `rollcopy.sh` script will create the subdirectory `install/rolls`.
to copy the iso file, `CentOS-7.4.1708-0.x86_64.disk1.iso` to the the release directory:
```
# rollcopy.sh CentOS-7.4.1708-0.x86_64.disk1.iso /var/www/html/rocks/7.0
```
Repeat the above for each roll iso that you want to copy
# roll-server

Scripts to help hosting a custom [Rocks Clusters](http://www.rocksclusters.org) roll download
server using apache/httpd

Rocks 7.0 frontend installation downloads "rolls" from remote server and installs them. The minimum
rolls required would end up 10 GB or above and downloading them for every installation is hard.
The scripts in this repository can help with that. We setup a local server that can simulate the
rocks cluster download server for easy download.

__II. Make certain that the supplied `index.cgi` script gives the directory listing__
There are four scripts defined in this repo
* `index.cgi` - cgi script to serve roll directory in a standard format
* `rollcopy.sh` - extract roll iso contents for web serving
* `httpdconf.sh` (optional) - defines a virtual server httpd configuration file
* `unpack-guides` (optional) - unpacks roll users guides into a web-servable directory

## 0. Preparation
* Clone this repository `git clone https://github.com/rocksclusters/roll-server.git`
* Download the roll isos from [rockscluster.org](http://central-7-0-x86-64.rocksclusters.org/isos/)

## I. Extract roll(s) to a web directory
`rollcopy.sh` extracts the roll iso files. The first argument `<file>` is the iso roll and the
second one is the server directory. It creates `install/rolls/` directory in the target and extracts
the files to it.

``` sh
$ rollcopy.sh <file> <target>
```
# cp index.cgi /var/www/html/rocks/7.0/install/rolls

To extract the roll, `CentOS-7.4.1708-0.x86_64.disk1.iso` to the the server directory:
``` sh
$ rollcopy.sh CentOS-7.4.1708-0.x86_64.disk1.iso /var/www/html/
```

You will probably need to add something like the following in your Apache
configuration, so that the directory index is given via the index.cgi
Repeat the above for each roll iso that you want to copy. The rolls will be accessbile for download
from the path `http://localhost/install/rolls/`.


## II. Copy `index.cgi` script for directory listing
Rocks frontend installation program searches for all the rolls by checking the directory listing of
files from the server. The `index.cgi` script servers the response by mimicking the rolls download
server. It needs to be copied to every directory.

``` sh
$ find /var/www/html/install/rolls -type d -exec cp index.cgi {} \;
```

Now configure `httpd` to execute the cgi script for every request

``` apache
# allow all access to the rolls RPMS
<Directory /var/www/html/rocks/7.0/install/rolls>
AddHandler cgi-script .cgi
Options FollowSymLinks Indexes ExecCGI
DirectoryIndex /rocks/7.0/install/rolls/index.cgi
<Directory /var/www/html/install/rolls>
# don't use SetHandler
AddHandler cgi-script .cgi
Options +FollowSymLinks +Indexes +ExecCGI
DirectoryIndex /install/rolls/index.cgi
Allow from all
</Directory>
```
(See below for an example of how to test and verify that your index.cgi is being
called properly)

__III. (Optional) provide a virtual httpd server__
The script `httpdconf.sh` will write to standard output a reasonable stanza for Apache web server configuration.
Suppose that you wanted the virtual server to be called `rocks-7.my.org` and have it used the directory above.
Then
```
httpdconf.sh rocks-7-0.my.org /var/www/html/rocks/7.0
```
will generate an httpds conf file. It is your responsibility to place this on the appropriate directory and restart
your webserver.
## III. (Optional) serve directories on a virtual server
`httpdconf.sh` lets you generate a vhost configuration for the server

__IV. (Optional) unpack the userguides__
You can unpack userguides
```
unpack-guides.sh /var/www/html/rocks/7.0/install/rolls /var/www/html/rocks/7.0
``` sh
$ httpdconf.sh rocks-7-0.my.org /var/www/html/
```

__V. Verifying that your output is correct (that is, index.cgi is being called)__
try
Above command generates a httpd vhost conf file with cgi handlers for virtual server
`rocks-7.0.my.org`. If you have your rolls in `/var/www/html/rocks/7.0/[install/rolls/...]` then
change the second argument accordingly.

## IV. (Optional) unpack the userguides
You can unpack userguides
``` sh
$ unpack-guides.sh /var/www/html/rocks/7.0/install/rolls/ /var/www/html/rocks/7.0/
```
wget -O - http://central-7-0-x86-64.rocksclusters.org/install/rolls

## V. Verifying that cgi scripts are working

Try
``` sh
$ wget -O - http://central-7-0-x86-64.rocksclusters.org/install/rolls/
```

and compare the output of the rocks central server listing of rolls to
your website, something like
```
wget -O - http://rocks-7-0.my.org/install/rolls
your website, something like

``` sh
$ wget -O - http://rocks-7-0.my.org/install/rolls/
```
You should see output very similar to

You should see output very similar to
```
TTP request sent, awaiting response... 200 OK
Length: 877 [text/html]
Expand Down Expand Up @@ -124,23 +145,32 @@ Saving to: ‘STDOUT’
<tr><td>
<a href="zfs-linux/">zfs-linux/</a>
</td></tr>
100%[======================================>] 877 --.-K/s in 0s
100%[======================================>] 877 --.-K/s in 0s
```
__VI. Verifying that rolls directories can be listed__

## VI. Verifying that rolls directories can be listed

The rolls themselves need be listed. Try the following

``` sh
$ wget -O - http://rocks-7-0.my.org/install/rolls/base/7.0/x86_64
```
wget -O - http://central-7-0-x86-64.rocksclusters.org/install/rolls/base/7.0/x86_64
```
and verify that when you use your roll server instead that you get the same output
and verify that you get the same output from
`http://central-7-0-x86-64.rocksclusters.org/install/rolls/base/7.0/x86_64`

__VII. Other items to check__
## VII. Other items to check

The following may need to be checked/changed for your setup:

* Firewall needs modifying on your web server to allow http access (platform sepcific)

* SELinux may need to be modified or turned off (e.g., ```setenforce Permissive```)



* Firewall needs modifying on your web server to allow http access (platform sepcific)
* SELinux may need to be modified or turned off (e.g., ```setenforce Permissive```)
* Check your permissions on the directories, run
``` sh
$ chown www-data:www-data -R /var/www/html/rocks/7.0/
```
to fix permissions to apache user. (run with sudo if necessary)
* You might need root pemissions to run `rollcopy.sh`
* Need to enable `cgi` module in apache/httpd
``` sh
$ a2enmod cgi
```
* Disable `gzip` compression