Today is the last day of Flock in Dresden, and it has been a really good Flock! On Thursday, I got the opportunity to build my first module for Fedora Modularity, and I just wanted to document the experience. As things go, it was quite easy, but there were a couple of things that tripped me up.

Please note that the tooling is still being worked on, so if you’re reading this much after it was published, some things will have probably changed.

Background

First, for those who haven’t been following, Modularity is a system that allows the user to select different streams of software in the same release. Traditionally in Fedora, there’s only one available version of any given package. There are situations where this can be limiting (say, if you need a particular version of a certain Python framework, or if you want to test drive the latest release of a package without upgrading your whole system to something unstable), and Modularity tries to fill this gap.

I thought this might be useful for LizardFS in Fedora. The current stable version of LizardFS is 3.12.0 (available in all active Fedora releases and in EPEL), but a release candidate for 3.13.0 has been published, and I thought it would be useful for those who actively want to test LizardFS to be able to install it.

On Thursday, I went to the Expert Helpdesk: Module Creation session and noticed that the room was fairly full. I was a bit concerned that I might not get the expert help that I needed, but that worry was put to rest when I found out that I was the only person in the room that hadn’t built a module before.

A certain Modularity developer (who shall remain unnamed to protect the guilty) volunteered me to go to the front and plug my laptop into the projector so the room full of experts could watch and advise me as I built my first module. No pressure at all.

I was advised that for my suggested use-case, I should just keep packaging the stable releases in Fedora as usual, but create a devel module stream for the unstable releases.

The process

The first step in building a module is to read the documentation. If you’re like me, you’ll read the first step, see that it talks about getting your package into Fedora, and quickly move to the next step. It turns out, though, that you do need to go back and read that section because you’ll need to create a new branch in dist-git for each module stream you are going to create.

Branching

One thing that isn’t (at the time of writing) documented is that, when requesting a branch, you must specify a service level, even though it seems that these aren’t going to be used. The date must end in 12-01 or 06-01, so I just made something up.

To request my branches, I ran:

fedpkg --module-name=lizardfs request-branch devel --sl rawhide:2020-12-01

Mohan approved my requests in record time (there are advantages to having everybody watching you), and I was ready to work with both dist-git, and the modulemd git repo. I pushed the release candidate to my new devel branch in dist-git, and then it was time to create the module definition.

Creating the module definition

Step two, creating the module definition, should have been straightforward, but there was talk about using fedmod to automate the process, so I installed it, ran into problems getting it working, and we decided it would be easier to just generate the modulemd file manually. I used the minimum template here as a starting point, saving it as lizardfs.yaml in the modulemd git repo. It was pretty simple to setup and the only slightly tricky thing to remember is that the license is the license of the module definition, not the package itself.

One thing I didn’t do, but really need to, is to setup some profiles for LizardFS. A profile is a group of packages that fit a use-case, and make it easier for the end-user to install the packages they need.

Building the module

Once I had pushed the modulemd, it was time to build! I pushed the build and waited… and waited… The way my module was defined, it was going to be built for every Fedora release that has modules (currently F28 (Server) and F29), so it took a while. Our time was running out, and we had a walking tour and scavenger hunt in Dresden (a brilliant idea, huge thanks to the organizers!), so that was the end of the workshop.

While on the scavenger hunt (our team placed second, thanks to some very competitive members of the team), I got a notification that the module build had failed. I found that I was missing a dependency, so I fixed that and rebuilt, but ran into a rather strange problem: it tried to redo the first build rather than doing a new one.

It turns out that, as documented, you need to push an empty commit to the modulemd git repo in order for it to build from the latest dist-git repo.

I had another failed build because lizardfs-3.13.0 doesn’t build against 32-bit architectures, so I wrote a small patch to fix that, pushed another empty commit to the modulemd git repo, and finally managed to build my very first module! Cue slow clap.

Thoughts

The module building process is actually quite simple and being able to build a module stream for all active Fedora releases simplifies the workload quite a bit. I’m seriously thinking about retiring LizardFS in Rawhide and only providing it via modules (with the stable stream being the default stream that you would get if you ran dnf install lizardfs-client).

The workflow is still a bit clunky, especially with having to manage both dist-git and the modulemd git repo, and I’d love to see that simplified, if possible, but it’s not nearly as difficult as I thought it might be.

A huge thanks to everyone (even Stephen) who put the time and effort into making modules work! I think they have a lot of potential in making the distribution far more relevant to developers and users alike. And a huge thanks to all the experts at the session. I know how hard it is to sit and watch someone make mistakes as they try to use something you’ve created, and you all were very patient with me.

Over the past few months, I’ve been working on zchunk, a compression format that is designed to allow for good compression, but, more importantly, the ability to download only the differences between an old version of the file and a new version.

The concept is similar to both zsync and casync, but it has some important differences. Let’s first look at how downloading a zchunk file works.

Downloading a chunk file

A zchunk file is basically a bunch of zstd-compressed “chunks” concatenated together with a header specifying the location, size and checksum of each chunk. Let’s take an example with only a few chunks:

Note that the file has three chunks, labeled A, B and C, each with a unique checksum. These checksums are stored in the header.

Now let’s imagine that we want to download a new version of the file:

Note that the new file has two chunks that are identical to the original file and one new chunk. The header in the new file contains the checksums of chunks A, C and D. We start by downloading just the header of the new file:

We then compare the chunk checksums in the old file’s header with the chunk checksums in the newly downloaded header and copy any matching chunks from the old file:

We finish by downloading any remaining chunks, reducing the number of http requests by combining the range requests, and then inserting the downloaded chunks into the appropriate places in the final file:

When we’re finished, you have a file that is byte-for-byte identical to the new file on the server:

Background

What inspired this format is the ridiculous amount of metadata you download every time you check for updates in Fedora. Most of the data from one day’s updates is exactly the same in the next day’s updates, but you’ll still find yourself downloading over 20MB of metadata.

When I first took a look at this problem, there were two potential solutions: casync and zsync.

casync

At first glance, casync looked like it provided exactly what we need, but real-world testing showed a couple of problems. Because casync puts each chunk into a separate file, we downloaded hundreds (and sometimes thousands) of individual files just to rebuild the original metadata file. The process of initiating each http request is expensive, and, in my testing, downloading only the changed chunks took much longer than just downloading the full file in the first place.

The more I looked at casync, the more obvious it became that it’s designed for a different use-case (delivering full filesystem images), and, while close, wasn’t quite what I needed.

zsync

zsync approaches the problem a completely different way, by requiring you to use an rsyncable compression format (gzip –rsyncable is suggested), splitting it into chunks and then storing the chunk locations in a separate index file. Unfortunately, it also sends a separate http request for each chunk that it downloads.

Add to that the fact that zsync is unmaintained and somewhat buggy, and I didn’t really feel like it was the best option. I did find out later that OpenSUSE uses zsync for their metadata, but they put all the new records at the end of their metadata files, which reduces the number of ranges (and, therefore, the number of http requests).

zchunk

After looking at the drawbacks of both formats, I decided to create a completely new file format, with one major design difference and one major implementation difference compared to both casync and zsync.

Unlike both casync and zsync, zchunk files are completely self-contained. For zsync, you need the archive and its separate index, while casync requires that each chunk be stored in separate files alongside the index. Casync’s method fit its use-case, and zsync’s method works, given that it’s meant to be a way of extending what you can do with already-created files, though it’s hobbled by the fact that you have to intentionally use special flags to make compressed files that give good deltas.

The downside of having a separate index is that you have to make sure the index stays with the file it’s pointing to, and, since we’re creating a new format, there wasn’t much point in keeping the index separate.

The implementation difference is the ability that zchunk has to combine range requests into one http request, a rarely used http feature that is part of the spec. Zsync could theoretically add this feature, but casync can’t because it downloads separate files.

Zchunk will automatically combine its range requests into the largest number that the server will handle (the nginx default is 256 range requests in a single http request, while Apache’s default is to support unlimited range requests), send them as one http request, and then split the response into the correct chunks.

The zchunk format is also designed to store optional GPG keys, so zchunk files will be able to be signed and verified without needing to store the signature in a separate file.

What still needs work in zchunk

• The C API for downloading needs to be finalized. I’m leaning towards not actually providing an API for downloading, but rather providing functions to generate the range requests and providing a callback that re-assembles the downloaded data into the correct chunks
• Full test cases need to be written
• GPG signature integration needs to be written
• Python extensions need to be written

What’s needed to get zchunk-enabled Fedora repositories

• I’ve written patches for createrepo_c that allow it to generate zchunk metadata, but it needs some work to make sure there are test cases for all the code
• I’ve written a patch for libsolv that allows it to read zchunk files, but I still need to submit it for review
• I’ve started on the work to get librepo to download zchunk metadata, but I’m not finished yet.