Merge branch 'devel'

.gitignore

@@ -1,4 +1,5 @@
 /build.conf
+/buildroot.conf
 /bootfs/
 /config/
 !/config/apt/custom_sources.txt

.travis.yml

@@ -0,0 +1,6 @@
+language: bash
+sudo: false
+install: gem install mdl
+script:
+ - bash -c 'shopt -s globstar; shellcheck **/*.sh'
+ - mdl --verbose --rules '~MD013, ~MD036' .

BUILD.md

@@ -14,6 +14,7 @@ On a Debian system those are the following, excluding packages with priority ess
 - dosfstools
 - binutils
 - bc
+- cpio
 
 On Debian based systems you can install them as root or with sudo as follows:
 
@@ -24,11 +25,26 @@ aptitude install git curl bzip2 zip xz-utils gnupg kpartx dosfstools binutils bc
 The following scripts are used to build the raspberrypi-ua-netinst installer, listed in the same order they would be used:
 
 - `clean.sh` - Start with a clean slate by removing everything created by earlier builds. This is not needed on a first build, but won't hurt either.
+
 - `update.sh` - Downloads latest Raspbian packages that will be used to build the installer.
+
 - `build.sh` - Builds the installer initramfs and .zip package for Windows/Mac SD card extraction method. Transfer the .zip package to a Windows/Mac computer, then simply unzip it and copy the files onto a FAT formatted SD card.
-- `buildroot.sh` - Builds the installer SD card image, it requires root privileges and it makes some assumptions like not having any other loop devices in use. You only need to execute this script if you need more than a .zip package. The script produces an .img package and also its bzip2 and xz compressed versions.
+
+- `buildroot.sh` - Builds the installer SD card image, it requires
+  root privileges and it makes some assumptions like not having any
+  other loop devices in use. You only need to execute this script if
+  you need more than a .zip package. The script produces an .img
+  package and then its bzip2 and xz compressed versions, but this is
+  configurable (see below).
 
 To set build options, create a file named `build.conf`, which contains the appropriate variable settings. Supported variables are:
 
 - `mirror_raspbian_cache` - Sets a apt caching proxy for the raspbian.org repository. (e.g. "192.168.0.1:3142")
 - `mirror_raspberrypi_cache` - Sets a apt caching proxy for the raspberrypi.org repository. (e.g. "192.168.0.1:3142")
+
+To set buildroot options, create a file named `buildroot.conf`, which contains the appropriate variable settings. Supported variables are:
+
+- `compress_bz2=1` - create a bz2-compressed image
+- `compress_xz=1` - create a xz-compressed image
+
+By default both bzip2 and xz compressed versions of the image will be created and the uncompressed image will deleted, but either or both can be disabled. If both are disabled, the uncompressed image will be left in place.

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## raspberrypi-ua-netinst v2.2.0
+
+- support for model 3B+
+- added installer output via telnet
+- improved compatibility with RPi Compute Module 3
+- improved SD card compatibility
+- added or changed configuration variables
+  - disable telnet output
+- many build improvements and fixes
+
 ## raspberrypi-ua-netinst v2.1.0
 
 - moved `cpufrequtils` from base to minimal preset

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2016 Andreas Eberlein <foodeas@aeberlein.de>
+Copyright (c) 2016-2018 Andreas Eberlein <foodeas@aeberlein.de>
 Copyright (c) 2013 Toni Spets <toni.spets@iki.fi>
 
 Permission to use, copy, modify, and distribute this software for any

README.md

@@ -30,22 +30,22 @@ The installer with the default settings configures eth0 with DHCP to get interne
 
 ### Features
 
-- completely unattended, you only need a working internet connection through the ethernet port or use the onboard wireless LAN (supported on model 3B and 0W)
+- completely unattended, you only need a working internet connection through the ethernet port or use the onboard wireless LAN (supported on model 3B, 3B+ and 0W)
 - DHCP and static IP configuration (DHCP is the default)
 - always installs the latest version of Raspbian
 - configurable default settings
 - extra configuration over HTTP possible - gives unlimited flexibility
 - installation takes about **20 minutes** with fast internet from power on to sshd running
 - can fit on a 512MB SD card, but 1GB is more reasonable
-- default installation includes `fake-hwclock` to save time on shutdown
+- default installation includes `fake-hwclock` to save the current time at shutdown
 - default installation includes NTP to keep time
 - `/tmp` is mounted as tmpfs to improve speed
 - no clutter included, you only get the bare essential packages
 - option to install root to a USB drive
 
 ## Requirements
 
-- a Raspberry Pi (from model 1B up to 3B or Zero including Zero W)
+- a Raspberry Pi (from model 1B up to 3B, 3B+ or Zero including Zero W)
 - SD card with at least 1GB, or at least 128MB for USB root install (without customization)
 - ethernet or wireless LAN with a working internet connection
 
@@ -74,7 +74,7 @@ If you don't have a display attached, you can monitor the ethernet card LEDs to
 
 If the installation process fails, you will see **SOS** in Morse code (... --- ...) on an led.  In this case, power off the Pi and check the log on the sd card.
 
-If you do have a display, you can follow the progress and catch any possible errors in the default configuration or your own modifications.
+If you do have a display, you can follow the progress and catch any possible errors in the default configuration or your own modifications. Once a network connection has been established, the process can also be followed via telnet (port 23).
 
 If you have a serial cable connected, installer output can be followed there, too. If 'console=tty1' at then end of the `cmdline.txt` file is removed, you have access to the console in case of problems.
 
@@ -137,7 +137,7 @@ The default **root** password is **raspbian**.
 - Configure your timezone: `dpkg-reconfigure tzdata`
 
 Optional:  
-Create a swap file with `fallocate -l 512M /swap && mkswap /swap && chmod 600 /swap` (example is 512MB) and enable it on boot by appending `/swap none swap sw 0 0` to `/etc/fstab`.  
+Create a swap file with `dd if=/dev/zero of=/swap bs=1M count=512 && chmod 600 /swap && mkswap /swap` (example is 512MB) and enable it on boot by appending `/swap none swap sw 0 0` to `/etc/fstab`.  
 
 ## Logging
 

build.sh

@@ -22,8 +22,8 @@ else
 	zipfile="raspberrypi-ua-netinst-$(date +%Y%m%d).zip"
 fi
 
-INSTALL_MODULES="kernel/fs/btrfs/btrfs.ko"
-INSTALL_MODULES="$INSTALL_MODULES kernel/drivers/scsi/sg.ko"
+INSTALL_MODULES+=("kernel/fs/btrfs/btrfs.ko")
+INSTALL_MODULES+=("kernel/drivers/scsi/sg.ko")
 
 # defines array with kernel versions
 function get_kernels {
@@ -67,16 +67,16 @@ function check_dependencies {
 	local dep
 	# iterate over the passed modules
 	for mod in "${mods[@]}"; do
-		# find the modules dependencies, convert into array
-		mapfile -t deps < <(grep "^${mod}" "${depmod_file}" | cut -d':' -f2)
+		# find the module's dependencies, convert into array
+		deps=($(grep "^${mod}" "${depmod_file}" | cut -d':' -f2))
 		# iterate over the found dependencies
 		for dep in "${deps[@]}"; do
 			# check if the dependency is in $modules, if not, add to temp array
-			contains_element "${dep}" "${modules[@]}" || new_found[${#new_found[@]}]="${dep}"
+			contains_element "${dep}" "${modules[@]}" || new_found+=("${dep}")
 		done
 	done
 	# add the newly found dependencies to the end of the $modules array
-	modules=("${modules[@]}" "${new_found[@]}")
+	modules+=("${new_found[@]}")
 	# set the global variable to the number of newly found dependencies
 	new_count=${#new_found[@]}
 }
@@ -410,6 +410,9 @@ function create_cpio {
 	cp --preserve=xattr,timestamps tmp/etc/rpc rootfs/etc/
 	cp --preserve=xattr,timestamps tmp/etc/services rootfs/etc/
 
+	# netcat-openbsd
+	cp --preserve=xattr,timestamps tmp/bin/nc.openbsd rootfs/bin/nc
+
 	# ntpdate components
 	cp --preserve=xattr,timestamps tmp/etc/default/ntpdate rootfs/etc/default/
 	# don't use /etc/ntp.conf since we don't have it
@@ -467,6 +470,9 @@ function create_cpio {
 	# libblkid1 components
 	cp --preserve=xattr,timestamps tmp/lib/*/libblkid.so.1.* rootfs/lib/libblkid.so.1
 
+	# libbsd0 components
+	cp --preserve=xattr,timestamps tmp/lib/*/libbsd.so.0.* rootfs/lib/libbsd.so.0
+
 	# libbz2-1.0 components
 	cp --preserve=xattr,timestamps tmp/lib/*/libbz2.so.1.0.* rootfs/lib/libbz2.so.1.0
 

buildroot.sh

@@ -2,6 +2,20 @@
 
 set -e # exit if any command fails
 
+# Set defaults for configurable behavior
+
+# Controls production of a bz2-compressed image
+compress_bz2=1
+
+# Controls production of an xz-compressed image
+compress_xz=1
+
+# If a configuration file exists, import its settings
+if [ -e buildroot.conf ]; then
+	# shellcheck disable=SC1091
+	source buildroot.conf
+fi
+
 build_dir=build_dir
 
 version_tag="$(git describe --exact-match --tags HEAD 2> /dev/null || true)"
@@ -60,16 +74,24 @@ else
 fi
 
 # Create archives
-rm -f "${build_dir}/${imagename}.img.xz"
-if ! xz -9v --keep "${image}"; then
-	# This happens e.g. on Raspberry Pi because xz runs out of memory.
-	echo "WARNING: Could not create '${IMG}.xz' variant." >&2
+
+if [ "$compress_xz" = "1" ]; then
+	rm -f "${image}.xz"
+	if ! xz -9v --keep "${image}"; then
+		# This happens e.g. on Raspberry Pi because xz runs out of memory.
+		echo "WARNING: Could not create '${IMG}.xz' variant." >&2
+	fi
+	rm -f "${imagename}.img.xz"
+	mv "${image}.xz" ./
 fi
-rm -f "${imagename}.img.xz"
-mv "${image}.xz" ./
 
-rm -f "${imagename}.img.bz2"
-( bzip2 -9v > "${imagename}.img.bz2" ) < "${image}"
+if [ "$compress_bz2" = "1" ]; then
+	rm -f "${imagename}.img.bz2"
+	( bzip2 -9v > "${imagename}.img.bz2" ) < "${image}"
+fi
 
 # Cleanup
-rm -f "${image}"
+
+if [ "$compress_xz" = "1" ] || [ "$compress_bz2" = "1" ]; then
+	rm -f "${image}"
+fi

doc/INSTALL_CUSTOM.md

@@ -96,7 +96,7 @@
 
 | Parameter | Default | Options | Description |
 |-------------------------|------------|--------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `gpu_mem` |  |  | Specifies the amount of RAM in MB that should be reserved for the GPU. To allow the VideoCore GPU kernel driver to be loaded correctly, you should use at least "32". If not defined, the bootloader sets it to 64MB. The minimum value is "16". |
+| `gpu_mem` | `16` |  | Specifies the amount of RAM in MB that should be reserved for the GPU. To allow the VideoCore GPU kernel driver to be loaded correctly, you should use at least "32". If not defined, the bootloader sets it to 64MB. The minimum value is "16". |
 | `console_blank` |  |  | Sets console blanking timeout (screensaver) in seconds. Default kernel setting is 10 minutes (`600`). The value `0` disables the blanking completely. |
 | `hdmi_type` |  | `tv`/  `monitor` | Forces HDMI mode and disables automatic display identification. Choose between TV or monitor  mode and specify the resolution with the options below. If not defined, the automatic display setting is used to determine the information sent by the display. |
 | `hdmi_tv_res` | `1080p` | `720p`/  `1080i`/  `1080p` | Specifies the display resolution if `hdmi_type` is set to TV mode. |
@@ -126,9 +126,8 @@
 | `cleanup` | `0` | `0`/`1` | Remove installer files after success. To also remove log files, note the option below. |
 | `cleanup_logfiles` | `0` | `0`/`1` | Removes installer log files after success. |
 | `cmdline` | `"dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 elevator=deadline fsck.repair=yes"` |  |  |
-| `rootfs_install_mount_options` |  |  |  |
-| `rootfs_mount_options` |  |  |  |
 | `final_action` | `reboot` | `reboot`/  `poweroff`/  `halt`/  `console` | Action at the end of install. |
+| `installer_telnet` | `1` | `0`/`1` | Send installer console output via telnet. |
 | `installer_retries` | `3` |  | Number of retries if installation fails. |
 | `installer_networktimeout` | `15` |  | Timeout in seconds for network interface initialization. |
 | `installer_pkg_updateretries` | `3` |  | Number of retries if package update fails. |

scripts/opt/raspberrypi-ua-netinst/install.sh

@@ -76,7 +76,7 @@ variables_reset() {
 	usbboot=
 	cmdline=
 	rootfstype=
-	final_action=
+	installer_telnet=
 	installer_retries=
 	installer_networktimeout=
 	installer_pkg_updateretries=
@@ -152,6 +152,7 @@ variables_set_defaults() {
 	variable_set "cmdline" "dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 elevator=deadline fsck.repair=yes"
 	variable_set "rootfstype" "f2fs"
 	variable_set "final_action" "reboot"
+	variable_set "installer_telnet" "1"
 	variable_set "installer_retries" "3"
 	variable_set "installer_networktimeout" "15"
 	variable_set "installer_pkg_updateretries" "3"
@@ -362,8 +363,7 @@ install_files() {
 	echo
 }
 
-output_filter() {
-	local filterstring
+set_filter() {
 	filterstring="^$"
 	filterstring+="|^Setcap failed on \S.*, falling back to setuid$"
 	filterstring+="|^dpkg: warning: ignoring pre-dependency problem"'!'"$"
@@ -390,12 +390,12 @@ output_filter() {
 	filterstring+="|\(Reading database \.\.\. [0..9]{1,3}\%"
 	filterstring+="|^E$"
 	filterstring+="|^: $"
+}
 
-	while IFS= read -r line ; do
-		if [[ "$line" =~ ${filterstring} ]] ; then
-			:
-		else
-			echo "  $line"
+output_filter() {
+	while IFS= read -r line; do
+		if [[ ! "${line}" =~ ${filterstring} ]]; then
+			echo "  ${line}"
 		fi
 	done
 }
@@ -498,7 +498,7 @@ installer_swapfile=/rootfs/installer-swap
 wlan_configfile=/tmp/wpa_supplicant.conf
 rootdev=/dev/mmcblk0
 tmp_bootfs=/tmp/bootfs
-final_action=reboot
+set_filter
 
 mkdir -p /proc
 mkdir -p /sys
@@ -659,6 +659,7 @@ case "${rpi_hardware}" in
 	"a020a0") rpi_hardware_version="Compute Module 3 (Lite)" ;;
 	"a22082") rpi_hardware_version="3 Model B" ;;
 	"a32082") rpi_hardware_version="3 Model B" ;;
+	"a020d3") rpi_hardware_version="3 Model B+" ;;
 	*) rpi_hardware_version="unknown (${rpi_hardware})" ;;
 esac
 
@@ -939,6 +940,21 @@ else
 	echo "OK"
 fi
 
+# Start telnet console output
+if [ "${installer_telnet}" = "1" ]; then
+	mkfifo telnet.pipe
+	mkfifo /dev/installer-telnet
+	tee < telnet.pipe /dev/installer-telnet &
+	while IFS= read -r line; do
+		if [[ ! "${line}" =~ userpw|rootpw ]]; then
+			echo "${line}"
+		fi
+	done < "/dev/installer-telnet" | /bin/nc -klC -p 23 > /dev/null &
+	exec &> telnet.pipe
+	rm telnet.pipe
+	echo "Printing console to telnet output started."
+fi
+
 # This will record the time to get to this point
 PRE_NETWORK_DURATION=$(date +%s)
 
@@ -1483,9 +1499,9 @@ mdev -s
 
 echo -n "Initializing /boot as vfat... "
 if [ -z "${boot_volume_label}" ]; then
-	mkfs.vfat "${bootpartition}" &> /dev/null || fail
+	mkfs.vfat -F 32 -s 1 "${bootpartition}" &> /dev/null || fail
 else
-	mkfs.vfat -n "${boot_volume_label}" "${bootpartition}" &> /dev/null || fail
+	mkfs.vfat -F 32 -s 1 -n "${boot_volume_label}" "${bootpartition}" &> /dev/null || fail
 fi
 echo "OK"
 
@@ -1505,8 +1521,13 @@ if [ "${kernel_module}" = true ]; then
 	fi
 fi
 
+
 echo -n "Initializing / as ${rootfstype}... "
-eval mkfs."${rootfstype}" "${rootfs_mkfs_options}" "${rootpartition}" &> /dev/null || fail
+eval mkfs."${rootfstype}" "${rootfs_mkfs_options}" "${rootpartition}" | sed 's/^/  /'
+if [ "${PIPESTATUS[0]}" -ne 0 ]; then
+	fail
+fi
+
 echo "OK"
 
 echo -n "Mounting new filesystems... "
@@ -1685,6 +1706,7 @@ if [ -n "${username}" ]; then
 		if [ -z "${userpw}" ]; then
 			echo -n "  Setting '${username}' to sudo without a password... "
 			echo -n "${username} ALL = (ALL) NOPASSWD: ALL" > "/rootfs/etc/sudoers.d/${username}" || fail
+			chmod 440 "/rootfs/etc/sudoers.d/${username}" || fail
 			echo "OK"
 		fi
 	fi
@@ -1781,8 +1803,18 @@ if echo "${cdebootstrap_cmdline} ${packages_postinstall}" | grep -q "ifupdown";
 		ln -s /dev/null /rootfs/etc/udev/rules.d/75-persistent-net-generator.rules
 	fi
 
-	if [ "${ip_addr}" != "dhcp" ]; then
-		cp /etc/resolv.conf /rootfs/etc/ || fail
+	# copy resolv.conf
+	echo -n "  Configuring nameserver... "
+	if [ -e "/etc/resolv.conf" ]; then
+		if cp /etc/resolv.conf /rootfs/etc/; then
+			echo "OK"
+		else
+			echo "FAILED !"
+			fail
+		fi
+	else
+		echo "MISSING !"
+		fail
 	fi
 
 	echo "OK"
@@ -2214,6 +2246,11 @@ if [ -n "${wlan_country}" ]; then
 	fi
 fi
 
+# disable wlan country warning
+if [ -e "/rootfs/etc/wifi-country.sh" ]; then
+	sed -i "1 iexit 0" /rootfs/etc/wifi-country.sh
+fi
+
 # set hdmi options
 if [ "${hdmi_type}" = "tv" ] || [ "${hdmi_type}" = "monitor" ]; then
 	config_set "/rootfs/boot/config.txt" "hdmi_ignore_edid" "0xa5000080"