TL;DR: You can find latest kernel configs, pre-built kernel images, u-boot configs and pre-built u-boot binaries as well as a pre-built gentoo linux image here. I try to keep everything up to date:

• Kernel: 4.9 (stable)
• U-Boot: 2016.05-rc1
• Gentoo RootFS: 2018-06-04 (with systemd)

Contents

• Compile everything from scratch
  • Prerequisites
  • Prepare Gentoo Base System
  • Compile a custom kernel
  • Compile U-Boot
  • First Boot
• Compile Jailhouse
• Playing around with Jailhouse
• Use my precompiled RootFS + Bootloader

---

Compile everything from scratch

Prerequisites

• Gentoo Host
• ARMv7 Compiler
• Jetson TK1 board
• Serial cable (for jailhouse output and general purpose debugging)
• Micro USB cable

On your local gentoo box, you can install the cross-compiler with crossdev.

$ emerge crossdev
$ crossdev -S -v -t armv7a-hardfloat-linux-gnueabi


Prepare Gentoo Base System

On your gentoo box, create a workspace

$ mkdir /tmp/jetson
$ cd /tmp/jetson


Inside this directory, download and unpack the latest Gentoo Stage 3. You may also want to download the latest portage snapshot to usr/.

$ curl http://gentoo.oregonstate.edu/releases/arm/autobuilds/current-stage3-armv7a_hardfp/stage3-armv7a_hardfp-20150721.tar.bz2 | sudo tar -xvjp -C .
$ curl http://gentoo.oregonstate.edu/snapshots/portage-latest.tar.bz2 | sudo tar -xvjp -C usr


Adjust your Gentoo make.conf. You probably want to add something like

$ sudo vim etc/portage/make.conf
>> MAKEOPTS="-j5"


Get a salted hash of your root password using openssl and write it to /etc/shadow

$ openssl passwd -1 gentoo
$1$xQl51XUf$.eUKt5EynZu4OWQnnIdZf.
$ sudo vim etc/shadow
root:$1$xQl51XUf$.eUKt5EynZu4OWQnnIdZf.:10770:0:::::


Adjust the etc/fstab. The following line should remain the only one.

/dev/mmcblk0p1 / ext4 noatime 0 1


Change the speed of the serial console /dev/ttyS0 to 115200 Baud (later we have to comment out this line for using jailhouse)

$ sudo vim etc/inittab
s0:12345:respawn:/sbin/agetty -L 115200 ttyS0 vt100


Never forget to think about a nice hostname and to set the timezone

$ echo hostname=\"iridium\" > etc/conf.d/hostname
$ ln -sf usr/share/zoneinfo/Europe/Berlin etc/localtime


We will use auto-DHCP for our networking interface

$ ln -sf net.lo etc/init.d/net.enp1s0
$ echo config_enp1s0=\"dhcp\" > etc/conf.d/net


Time to send out RootFS to the Jetson TK1. A pretty nice feature of u-boot allows us to mount the eMMC of the Jetson as USB mass storage device. Connect the Mini-USB to your machine, push the reset button your jetson, interrupt the bootloader and type:

Hit any key to stop autoboot: 0
Tegra124 (Jetson TK1) # ums 0 mmc 0
UMS: LUN 0, dev 0, hwpart 0, sector 0x0, count 0x1d5a000


Now the whole eMMC of your Jetson is available as a mass storage device. I recommend to create one single huge ext4 partition using gparted. After creating the partition, mount it and copy the root file system:

mount /dev/sdX1 /mnt/temp/
cp -av /tmp/jetson/* /mnt/temp/
sync


Take a cup of coffee or tea.

Compile your own kernel

As your Jetson is unable to boot yet, you have to cross-compile the kernel on your local machine.

$ emerge gentoo-sources
$ cd /usr/src/linux
$ make ARCH=arm menuconfig # Adjust everything you need
$ make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- -j 5
# Copy the kernel
$ cp arch/arm/boot/zImage /mnt/temp/boot/
# Copy the device tree block
$ cp arch/arm/boot/dts/tegra124-jetson-tk1.dtb /boot
# Install all modules
$ make modules_install INSTALL_MOD_PATH=/mnt/temp/
# Sync file system and unmount the Jetson
$ sync
$ umount /mnt/temp


Next step is to place a file called /boot/boot.scr. It’s pretty similar to GRUB’s menu.cfg. U-Boot reads this file when booting and executes its commands. Unfortunately, U-Boot uses a binary format and it must be compiled first. But first of all, install the u-boot-tools:

$ emerge u-boot-tools


Create the file boot.script that contains the following content:

# Optionally add 'init=/usr/lib/systemd/systemd', if you want to use systemd insteas of OpenRC
setenv bootargs 'console=ttyS0,115200 root=/dev/mmcblk0p1 rw rootwait'

#Comment the line above and uncomment these lines for running jailhouse
#setenv bootargs 'root=/dev/mmcblk0p1 rw rootwait mem=1984M vmalloc=512M'
#setenv bootm_boot_mode nonsec

load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage
load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/tegra124-jetson-tk1.dtb
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}


Use the mkimage tool of the u-boot-tools to compile it to a .scr file:

$ cd /tmp/jetson/boot
$ wget https://ramses-pyramidenbau.de/~ralf/jetson-tk1/boot.script
$ mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n Boot-Script -d boot.script boot.scr


After rebooting, make sure that all CPUs are started in HYP mode:

$ root@iridium:~# dmesg | grep HYP
[ 0.101989] CPU: All CPU(s) started in HYP mode.


Here you can also download a prebuilt boot.scr. Copy the boot.scr file to the /boot directory of your jetson.

Compile U-Boot

Jailhouse requires the processor to run in non-secure HYP mode. The processor needs to be staged to HYP mode very early by the bootloader. HYP is not enabled in NVIDIA’s stock bootloader, so we need to compile and flash it on our own.

It took me quite some time to find out how to build the bootloader correctly.
Here’s the summary 🙂 Run the following commands:

$ mkdir /tmp/uboot
$ cd /tmp/uboot
# Get all the required stuff
$ git clone https://github.com/NVIDIA/tegra-uboot-flasher-scripts.git
$ git clone https://github.com/NVIDIA/tegrarcm.git
$ git clone https://github.com/NVIDIA/cbootimage.git
$ git clone https://github.com/NVIDIA/cbootimage-configs.git
$ git clone git://git.denx.de/u-boot-tegra.git u-boot

# build tegrarcm
$ cd tegrarcm
$ ./autogen.sh
$ make
$ cd ..

# build cbootimage
$ cd cbootimage
$ ./autogen.sh
$ make
$ cd ..

# build u-boot
$ export PATH=$PATH:/tmp/uboot/cbootimage/src:/tmp/uboot/tegrarcm/src
$ cd tegra-uboot-flasher-scripts
$ export CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi-
$ ./build --socs tegra124 --boards jetson-tk1 build


Et voilà, you just built your own U-Boot bootloader. Let’s flash it! Before sending the next command, bring your device in recovery mode by pressing the “RESET” + “FORCE RECOVERY” buttons.


$ cd ../tegra-uboot-flasher-scripts
$ ./tegra-uboot-flasher --data-dir ../_out flash jetson-tk1


With a bit of luck and some magic unicorn dust, your jetson will directly boot your custom bootloader that boots your custom kernel. After rebooting, you can check if HYP is enabled with

$ root@iridium:~# dmesg | grep HYP
[ 0.101989] CPU: All CPU(s) started in HYP mode.


First Boot

After your device was successfully flashed it will automatically reboot. You can now login to it by using a serial connection.

$ putty -serial /dev/ttyUSB0 115200 -sercfg 115200


As we don’t have NTP yet, our first step will be to set the current time.

On your TK1 login and type:

$ date --set="20150723 18:37"
$ ####
$ # Your network should be autoconfigured via DHCP, if not try
$ ifconfig enp1s0 a.b.c.d
$ route add default gw a.b.c.e
$ echo nameserver 8.8.8.8 > /etc/resolv.conf
$ ####
$ emerge --sync
$ emerge -av ntp
$ rc-update add ntp-client default
$ rc-update add sshd default
$ # Perform a full system upgrade
$ emerge -uDNav world


---

Compile Jailhouse

This will guide you how to compile Jailhouse on your Jetson TK1. Jailhouse comes with some dependencies and requires the mako python package in order to compile. Install mako as well as git before cloning and compiling jailhouse.

$ emerge git mako
$ git clone https://github.com/siemens/jailhouse.git
$ cd jailhouse
$ cat > hypervisor/include/jailhouse/config.h # Create this file with the following content
#define CONFIG_ARM_GIC_V2 1
#define CONFIG_MACH_TEGRA124 1
$ make
$ make install
$ depmod -a
$ modprobe jailhouse
$ lsmod
# and check if the module was successfully loaded


---

Playing around with jailhouse

This tutorial will give you a brief introduction on how to run a simple uart-demo inmate in jailhouse on the TK1. It is important not to use the UART in the root cell as it will be assigned to the non-root cell after creating it. Any further UART access from the root cell after creating the non-root cell will lead to a panic of the hypervisor. Additionally, jailhouse needs some memory for the hypervisor itself and its inmates that is not used by the kernel of the root cell. This results in a boot.script as follows:

# Reserve 64MiB for jailhouse
setenv bootargs 'root=/dev/mmcblk0p1 rw rootwait mem=1984M vmalloc=512M'
# This enables HYP mode (disabled by default)
setenv bootm_boot_mode nonsec
load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage
load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/tegra124-jetson-tk1.dtb
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}


Compile the boot.script using mkimage to boot.scr and copy it to ‘/boot/’.

$ mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n Boot-Script -d boot.script /boot/boot.scr


Also, comment out the serial console from ‘/etc/inittab’. Use ssh to access your device. After rebooting your device, you can load the jailhouse module and activate the hypervisor:

$ modprobe jailhouse
$ jailhouse enable ~/jailhouse/configs/jetson-tk1.cell

Dmesg should now tell you, that the jailhouse has opened:

$ dmesg | tail
[...]
[ 183.608569] The Jailhouse is opening.

On your serial console, you should see something like:

Initializing Jailhouse hypervisor v0.5 (132-g997802f) on CPU 0
Code location: 0xf0000020
Page pool usage after early setup: mem 22/16112, remap 64/32768
Initializing processors:
CPU 0... OK
CPU 3... OK
CPU 2... OK
CPU 1... OK
Page pool usage after late setup: mem 34/16112, remap 64/32768
Activating hypervisor

Now you can create a new non-root cell using the jetson-tk1-demo cell configuration:

$ jailhouse cell create ~/jailhouse/configs/jetson-tk1-demo.cell


Now that your cell is created, you can load a binary to it and start the cell:

$ jailhouse cell load jetson-tk1-demo ~/jailhouse/inmates/demos/arm/uart-demo.bin -a 0
$ jailhouse cell start jetson-tk1-demo

On your serial console, you can now see the output of the demo cell.

To stop and destroy the cell, just type:

$ jailhouse cell shutdown jetson-tk1-demo
$ jailhouse cell destroy jetson-tk1-demo


Use my precompiled RootFS + Bootloader

If you don’t want to compile everything on your own – which i truly can understand – then you can use my pre-compiled images.

I try to keep those images up to date. Look at the top of the page for the current versions. SSH server is enabled, root password is ‘gentoo’, dhcp client on wired ethernet. So just a few steps for you to start.

Reset your Jetson, interrupt the bootloader and attach it as USB mass storage device (explained above) by typing

Tegra124 (Jetson TK1) # ums 0 mmc 0

to your serial console. Mount and partition your Jetson and extract the root file system:

$ mount /dev/sdX1 /mnt/temp/
# Extract Root FS
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/rootfs/RootFS-latest.tar.gz | tar x -C /mnt/temp/
# Extract pre-compiled kernel
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/kernel/precompiled/latest.tar.bz2 | tar x -C /mnt/temp/
# Place boot.scr (non-jailhouse variant)
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/u-boot/boot.scr > /mnt/temp/boot/boot.scr
# Place boot.scr (jailhouse variant)
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/u-boot/boot.scr.jailhouse > /mnt/temp/boot/boot.scr


Next step is to flash the correct bootloader. On your local machine, type

$ git clone https://github.com/NVIDIA/tegra-uboot-flasher-scripts.git
$ git clone https://github.com/NVIDIA/tegrarcm.git
$ cd tegrarcm
$ ./autogen.sh
$ make
$ export PATH="$PATH:`realpath src`"
$ cd ../tegra-uboot-flasher-scripts
$ curl https://ramses-pyramidenbau.de/~ralf/jetson-tk1/u-boot/jetson-tk1-uboot-2016.05-rc1.tar.bz2 | tar -xzv -C .
$ ./tegra-uboot-flasher --data-dir . flash jetson-tk1


Et voilà. Reboot your device and check if HYP is enabled if you chose the jailhouse variant:

root@iridium:/# dmesg|grep HYP
[ 0.101987] CPU: All CPU(s) started in HYP mode.