Introducing zchunk, a new file format that’s highly delta-able (is that even a word?), while still maintaining good compression. This format has been heavily influenced by both zsync and casync, but attempts to address the weaknesses (for at least some use-case) in both formats. I’ll cover the background behind this in a later post.

Like casync and zsync, zchunk works by dividing a file into independently compressed “chunks”. Using only standard web protocols, the zchunk utility zckdl downloads any new chunks for your file while re-using any duplicate chunks from a specified file on your filesystem.

Zchunk is a completely new compression format, and it uses a new extension, .zck. By default, it uses zstd internally, but, because it compresses each chunk separately, a zchunk file cannot be decompressed using the zstd utilities. A zchunk file can be decompressed using the unzck utility and compressed using the zck utility.

Zchunk also supports the use of a common dictionary to help increase compression. Since chunks may be quite small, but have repeated data, you can use a zstd dictionary to encode the most common data. The dictionary must be the same for every version of your file, otherwise the chunks won’t match. For our test case, Fedora’s update metadata, using a dictionary reduces the size of the file by almost 40%.

So what’s the final damage? In testing, a zchunk file with an average chunk size of a few kilobytes and a 100KB dictionary ends up roughly 23% larger than a zstd file using the same compression level, but almost 10% smaller than the equivalent gzip file. Obviously, results will vary, based on chunk size, but zchunk generally beats gzip in size while providing efficient deltas via both rsync and standard http.

The zchunk file format should be considered fixed in that any further changes will be backwards-compatible. The API for creating and decompressing a .zck file can be considered essentially finished, while the API for downloading a .zck file still needs some work.

Future features include embedded signatures, separate streams, and proper utilities.

zchunk-0.4.0 is available for download, and, if you’re running Fedora or RHEL, there’s a COPR that also includes zchunk-enabled createrepo_c (Don’t get too excited, as there’s no code yet in dnf/librepo to download the .zck metadata).

Development is currently on GitHub.

Updated 05/03/2018 to point to new repository location

In case you can’t tell by the title of this post, I’m… mildly… annoyed with HP right now. The story starts with my just-over-a-year-old HP Pavilion laptop that has been having problems with its power brick.

While I was at work on Monday, the laptop started running on batter power even when it was plugged in, but when I got home, everything worked perfectly. I assumed it had something to do with the power at work, and wondered whether it might even be firmware related. I looked through the newest BIOS’s changelog, and, sure enough there was something mentioned about power and charging, so I downloaded it and updated my BIOS.

At least, that’s what I would have done if I was running Windows. Unfortunately for me, I’m not. I don’t even have a dual-boot system because I haven’t really needed Windows for years, and, when I do, a VM does the job just fine.

The only BIOS update HP offers is a Windows exe file, so I downloaded it, and ran cabextract to get the files off it. The tool lshw told me that my motherboard was an 0820D, and the zip contained a file called 0820DF45.bin (the BIOS revision is F.45), so I had everything I needed. I put the bin file on a USB, rebooted into HP’s recovery tools, and then went to firmware management, selected the bin file… and fail! It sat there telling me that I need a signature file for the firmware.

So I searched for the signature file, but it wasn’t in the exe. I googled for it, and found a lot of people who seem to be in the same boat. One suggestion was to run the exe on a Windows system and select the “Put BIOS update on USB” option. Sounded easy enough, so I booted my Windows VM, ran the exe, accepted the stupid EULA (I’m pretty sure I saw something in the forty-third paragraph about dancing on one leg while balancing a cupcake on my nose), installed the BIOS updater, and… nothing. After twenty seconds or so, a message popped up, “This program might not have installed correctly. Install using compatibility settings?” Yeah, thanks. After multiple attempts at different compatibility settings… still nothing.

I googled around a bit more, and found a 2GB HP USB image that you can use to recover your BIOS if it gets corrupted. Sweet! I downloaded it, and several hours later, I found out it only has the original BIOS revision (complete with a signature file!), but not my latest update!

At this point, I was desperate. My final hope was to figure out some way to boot my laptop into Windows. I have an 500GB SSD with a grand total of 30GB free, so that wasn’t an option. What about a Live USB? I mean, Linux distributions have had Live CDs and USBs forever, so it must just work in Windows, right?

Nope. Not unless you have Windows 10 Enterprise with it’s Windows-to-go feature. Luckily, the guys over at Hasleo software have created a nifty little tool called WinToUSB that does the same thing. I copied my VM image over to a USB, booted from it, and ran the BIOS update.

It worked perfectly and even offered to put the BIOS update on a USB! It seems that HP, in their infinite wisdom, have designed the updater so it refuses to start unless you’re on an HP machine.

To add insult to injury, all the updater does is copy the BIOS bin file and its signature onto the EFI partition, where it gets updated after a reboot. As far as I can tell, the signature file is generated on the fly by the updater, which begs the question… Why? Why generate the signature on the fly, rather than just stick it in the embedded CAB file with the BIOS images? Why require an HP system to generate a USB image containing the BIOS update? Why require your users to dance on one foot while balancing a cupcake on their nose?

And, as further insult (or maybe we’re back to injury), the BIOS update didn’t fix my charging problem, and it turns out that my just-out-of-warranty power brick is dying. Thanks, HP. You guys rock! After three HPs laptops in a row, I think it may be time for a change.

Picture of burning laptop by secumem, used under a CC BY-SA 3.0 license