NetBSD-SoC: Implementing HTTP support for libsa

What is it?

Have you ever wanted to compile a kernel for a specific use/device? You probably have gone through a zillion times of the
endless tftp-boot-test-reboot cycle. Chances are, you weren't that happy about tftp, right?

libsa is the shared code for the bootloaders, and architecture independent. thus if HTTP were implemented, you could just as easly load the kernel from an http address.

Status

2010-04-30: this page is set up 2010-07-13: the patch to use the UNDI API of PXE instead of the UDP is available for testing. 2010-09-23: project is completed. still has things to do, see below.

April 21, 2010: Community Bonding Period -- Students get to know mentors, read documentation, get up to speed to begin working on their projects.
May 24, 2010: Students begin coding for their GSoC projects; Google begins issuing initial student payments
July 12, 2010: Mentors and students can begin submitting mid-term evaluations.
July 13, 2010: Patch to restructure i386/stand/pxeboot to use the general UNDI PXE API is ready, and available here
July 14, 2010: Patch to bring preliminary HTTP support is available: pxeboot patch, libsa patch
July 16, 2010: Mid-term evaluation deadline; Google begins issuing mid-term student payments provided passing student survey is on file.
August 9, 2010: Suggested 'pencils down' date. Take a week to scrub code, write tests, improve documentation, etc.
August 16, 2010: Firm 'pencils down' date. Mentors, students and organization administrators can begin submitting final evaluations to Google.
August 20, 2010: Final evaluation deadline; Google begins issuing student and mentoring organization payments provided forms and evaluations are on file.
August 23, 2010: Completed the project. For patches and description, see below.

Deliverables

Mandatory (must-have) components:

Implement a minimalistic TCP stack into libsa (done)
Implement an HTTP stack on top of that, capable of GET-ting a kernel (done)

Optional (would-be-nice) components:

HTTP authentication in the form of http://username:password@host/path/imagename (done)
IPv6 support (WIP)

Documentation

Before using PXE, you need to have a DHCP and a TFTP server set up. I'm assuming this is the case.
(If it's not, see the relevant documentation here.)

You need to patch your kernel. You need two patches:
- i386-only: You need to patch the PXE code using this patch inside src/sys/arch/i386/stand/pxeboot.
- You need to patch libsa using this patch inside src/sys/lib/libsa.

The patches are against 5.0.2, and they add some clutter. I'll link new patches as they become available.

The boot process has two stages:
- first, the PXE client inside the NIC ROM (or BIOS, ...) downloads the NetBSD bootloader
- then the NetBSD bootloader starts the system

(Actually, it's irrelevant to libsa if the caller (the bootloader) was started by PXE or not.
Not all architectures ures libsa, however. For that, I'm planning to do a reference table on the wiki,
but still waiting for my developer application to be processed. For now, let's suppose
you're using an architecture that's already using libsa.)

Now, to boot a kernel from a http address, you'd setup your dhcpd.conf like this:

subnet 192.168.68.0 netmask 255.255.255.0 {
    option routers 192.168.68.11;
    option subnet-mask 255.255.255.0;
    option broadcast-address 192.168.68.255;
    option domain-name-servers 8.8.8.8;
    range dynamic-bootp 192.168.68.100 192.168.68.150;
    next-server 192.168.68.11;

    if substring (option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000" {
            filename "pxeboot_ia32.bin";
    } elsif substring (option vendor-class-identifier, 0, 17) = "NetBSD:i386:libsa" {
	    filename "http:http://user:password@192.168.1.1/netbsd";
    }
}

Two entries, for the two stages.
The general notation is "http:http://[username:password@]host/url".

If the host is not on the same subnet as the machine you're trying to boot, it's
going to use the IP specified in "next-server". This might change later, as I'll
add an option for specifying the gateway.

Things to do

The project has not ended with GSoC. Here is the current TODO:

BIG FAT WARNING: I have a nasty and serious bug somewhere. Once in 15 or so tries, the CPU freezes. I need to fix that.
Add #ifdefs for debug purposes, so debug/release can be decided by compiler flags
HTTP error code handling (right now, we just skip the HTTP headers. We must get a 200, or halt.)
handle TCP RST flag, especially on the handshake
i386(/amd64?) only: check if the PXE ROM supports UNDI. if it doesn't, we can't boot HTTP.
test it on different archs. martin@ said he'll help.
(after all of the above): merge it to -current ;)

Technical Details

The first thing needed to be done was to rework the pxeboot code. It had used the PXE API's UDP
sending/receiving feature, which is not suitable for general packet handling.
I had to implement the UNDI inferface.

The next step was to separate the IP and the UDP layers, thus adding the readip/sendip methods.
The last steps were to add the TCP and HTTP code on top of that. For now, the HTTP code just strips
the HTTP headers from the server without interpreting it. We use the a range specifier in the request,
so a HTTP/1.1 capable webserver is required.

For HTTP basic authentication, a Base-64 encoder was implemented (as per RFC 4648).

Zoltan Arnold NAGY <zoltan.arnold.nagy@gmail.com>

$Id: index.html,v 1.8 2010/08/23 11:26:06 nagyz Exp $