Setting up a netboot server in Fedora/CentOS

I’ve had a request for an explanation on how we use PXE and syslinux in our school. In a previous post, I talked a bit about chain-loading pxe, but didn’t explain much on how our system is set up.

So here goes…

Our primary goal in setting up a PXE environment was to have some way of imaging our computers without having to screw around with an ancient version of Norton Ghost and without having to put a floppy in every computer.

The problem was that we really didn’t want students to be able to reimage the computers whenever they wanted to, and there were other tools we wanted to use that we wanted restricted.

The solution was PXELINUX’s simple menu, and it works beautifully! This post will walk you through the process of setting up PXELINUX (and gPXE while we’re at it).

For this post, I am assuming that you already have a DHCP server and a web server set up.

There are three things we need to set up:

  1. TFTP server
  2. gPXE
  3. Syslinux

The first step is to set up a TFTP server to carry our gPXE images.

  1. Run
    yum install tftp-server
    
  2. Edit /etc/xinetd.d/tftp and change the line that says
    disable = yes
    
    to
    disable = no
    
  3. If this is the initial installation of xinetd, you may need to run chkconfig --levels 2345 xinetd on
    service xinetd start
    
    at this point. Otherwise, it might be a good idea to run
    service xinetd reload
    

Now, for the next step, we need to download our gPXE images. gPXE is an extended version of PXE that allows you to load images over http and https in addition to the usual tftp. As most (all?) network cards don’t come with gPXE drivers, we will be using PXE to download and bootstrap our gPXE drivers.

As mentioned in my previous post, some of our motherboards seem to have issues mixing PXE and their normal PXE UNDI drivers, so I prefer to use gPXE’s native drivers rather than its UNDI driver.

However, we have four computers whose network cards just don’t work with gPXE’s native drivers, so we will direct those four computers to the gPXE UNDI driver.

So let’s grab and setup these drivers:

  1. Go to ROM-o-matic and choose the latest production release
  2. For output format, choose “PXE bootstrap loader image [Unload PXE stack] (.pxe)”
  3. Choose NIC type “all-drivers”
  4. Click on “Customize”
  5. Check the box that says “DOWNLOAD_PROTO_HTTPS”
  6. Click on the button that says “Get Image”
  7. Save file to /tftpboot/gpxe.pxe
  8. Change NIC type to “undionly”
  9. Click on the button that says “Get Image”
  10. Save file to /tftpboot/undi.pxe

I’m assuming you’re running the ISC dhcp server (dhcp package on both Fedora and CentOS). If not, you’ll have to work out these next steps yourself.

You need to edit /etc/dhcpd.conf and add the following lines:

next-server     ip address;

if exists user-class and option user-class = "gPXE" {
    filename "http://webserver/netboot/pxelinux.0";
} else {
    filename "/gpxe.pxe";
}

Where ip address is the ip address of your TFTP server and webserver is the name/ip address of your web server.

If you have some computers that won’t pxeboot using gPXE’s native drivers (you’ll be able to tell because the computers will show the gPXE loading screen, but won’t be able to get an IP address using DHCP while in gPXE), change the last five lines above to:

if exists user-class and option user-class = "gPXE" {
    filename "http://webserver/netboot
} else {
    if binary-to-ascii(16, 8, ":",
       substring(hardware, 1, 6)) = "mac address 1"
    or binary-to-ascii(16, 8, ":",
       substring(hardware, 1, 6)) = "mac address 2" {
        filename "/undi.pxe";
    } else {
        filename "/gpxe.pxe";
    }
}

Where “mac address 1” and “mac address 2” are the MAC addresses of the computers that don’t work with gPXE’s native drivers. Please note the MAC address are without leading zeros (i.e. 00:19:d1:3a:0e:4b becomes 0:19:d1:3a:e:4b).

At this point, if you boot any computer on your network off the NIC, you should see something like this:

The next step is to setup PXELINUX, a part of the Syslinux Project. PXELINUX is a small bootloader designed for booting off a network.

  1. On your web server, create a directory called “netboot” in your web root (normally /var/www/html on Fedora/CentOS).

  2. Run

    yum install syslinux
    

    or, as an alternative, build a newer version of syslinux. I recommend at least 3.75 (the version in Fedora 12), though I’m using 3.82 at the school.

  3. Copy (at minimum) chain.c32, menu.c32, vesamenu.c32 and pxelinux.0 to “netboot” in your web root. (These files will be located in /usr/share/syslinux if you installed the package using yum.) At this point, you’ll probably want to check for other modules that might have some potential. We use ifcpu64.c32 to decide between 32-bit and 64-bit Fedora on the computers.

  4. Run:

    yum install memtest86+
    cp /boot/elf-memtest86+-4.00 \
    your_web_root/netboot/memtest
    

    (Note that “your_web_root” will most likely be /var/www/html)

  5. Download this picture and save it to your_web_root/netboot

  6. Change directory to your_web_root/netboot

  7. Run

    mkdir pxelinux.cfg
    cd pxelinux.cfg
    
  8. Create a file called “default” that contains the following:

    default vesamenu.c32
    timeout 40
    prompt 0
    noescape 1
    menu title Boot Options
    menu background menu.png
    menu master passwd $4$tek7ROr8$xzFCb2QVEWsc2msx3QsErbRuo0Y$
    
    label local
    menu label ^Boot from hard drive
    kernel chain.c32
    append hd0
    
    label admin
    menu label ^Administrative tools
    kernel vesamenu.c32
    append pxelinux.cfg/admin
    menu passwd $4$tek7ROr8$xzFCb2QVEWsc2msx3QsErbRuo0Y$
    
  9. Create a file called “admin” that contains the following:

    default vesamenu.c32
    timeout 40
    prompt 0
    noescape 1
    menu title Administrative Tools
    menu background menu.png
    menu master passwd $4$tek7ROr8$xzFCb2QVEWsc2msx3QsErbRuo0Y$
    
    label memtest
    menu label ^Memory tester
    kernel memtest
    

If you boot any computer on your network off the NIC, you should see something like this:

Main boot menu
Enter your password for Admin tools
Admin tools

So now you have a double layered menu system with a password required to get to the second layer. For reference’ sake, the current password is “purple”, and you can generate your own password by running sha1pass (included in the syslinux package).

If you wanted to add other administrative tools, you would add them to the file “admin” in netboot. For more information on how to add items to the menu, see this page.

Deltarpm problems (Part II)

About six weeks ago, I looked at one of the problems we currently face with deltarpm, that of colored executables.

Today, I want to look at one of the other major problems that we’ve currently papered over without really fixing. That is compression.

Background

When we switched over to using xz compression in Fedora 12, we ran into a two problems, one expected and one not. The first problem was that deltarpm didn’t have to code to handle xz-compressed rpms. That was solved quickly thanks to the work that SUSE developers had already put into handling lzma-compressed rpms. We had to do a little bit of adapting to xz, but it was pretty straightforward and trivial.

The second problem popped up right after the switchover and was completely unexpected. When doing some updates on a Rawhide machine, I noticed that a number of noarch deltarpms were giving me a checksum error on rebuild (prompting a download of the full rpm). It soon became obvious that xz wasn’t producing the same compressed files on PPC and x86 machines.

A noarch rpm (one that could be installed on any architecture machine) would sometimes be randomly built on a PPC builder, and a deltarpm for that package would be generated. The deltarpm would be applied on my x86 laptop and the resulting uncompressed data would be identical to the original rpm’s uncompressed data. However, when that uncompressed data was then recompressed so that we would have the original compressed rpm, it compressed slightly differently, breaking the package signatures.

The Problem

Most compression formats don’t guarantee repeatability. They do not promise that the compressed file you generated today will be identical to the compressed file you generate tomorrow. They just promise that you’ll be able to decompress your file tomorrow.

To understand this, remember that any compression format has a standard (which must always be followed) and an algorithm (which may change slightly). Look at the two following math formulas:

(1 + 5) / 3 + (3 + 9) / 4

(3 + 9) / 4 + (1 + 5) / 3

Though they are different, they still parse to exactly the same result. Now imagine that the formulas are two different compressed files, and the result is the uncompressed file. As far as the compression format is concerned, both compressed files are valid.

The problem is that because deltarpm must rebuild the original compressed rpm, it’s built on the assumption that compression is repeatable. And the assumption is mostly true, mainly because gzip and bzip2 haven’t made changes to their compression algorithms in years. But xz is a much newer algorithm that is still being fine-tuned.

One advantage of this is that upstream changed xz so it is repeatable across different architectures, fixing the PPC/x86 problem. However, upstream made it very clear that they were not promising repeatability over time. They may change the compression algorithm to improve speed or compression, while still sticking to the standard.

A Related Problem

This is closely related to another problem we hit when generating deltas: compressed files in rpms.

How many files are stored on the filesystem in a compressed format? All of our man pages, to start with. A lot of game data. And more…

Guess how good our deltas our of compressed data? Pretty bad, because a small change in an uncompressed file normally creates big changes in compressed files, throwing away the benefit of deltas.

And we can’t uncompress those files before doing a delta on them because we can’t guarantee that they will be recompressed to the exact same file (were they compressed with gzip -5? gzip -9? gzip -1?).

The Current Situation

So where does this leave us? In a bit of a mess with deltarpms if xz does change its compression output. It is a solvable mess, but it’s still a mess.

We also have lousy deltas if there are any compressed files in the rpm. In many ways, this problem is more immediate.

What we need is some way to recompress data in such a way that guarantees that it’s identical to the original compressed data…