Eulogy for yum-presto

With Fedora 19 comes a change that is rather bitter-sweet. Yum-presto, the plugin that I originally wrote to allow the use of deltarpms in Fedora, has been retired, its functionality merged directly into yum. While this is a necessary step to achieve things like parallel downloads of deltarpms and regular rpms, it’s still hard to see the death of a project that I’ve been involved with since its beginning, six years ago.

I started using Linux in 2000, switching from Red Hat Linux to Mandrake in 2001, and then to Fedora Core in September 2004, just before Fedora Core 3 was released. I had just returned to the Lebanon Evangelical School as a teacher and sysadmin after finishing university, and I remember downloading the FC3 Test 2 ISO image on our impressively fast 128kbps link. And I still remember my disgust when Test 2 was still quite buggy and I had to download the FC2 ISO.

Somewhere around the end of 2005, I came up with the brilliant idea of running a local mirror of Fedora Core for our school. The problem was that updates took forever to download, so I had a flash of inspiration. What if… we could download only the parts of the rpm that had changed? Updates would be much smaller and running a local mirror would become feasible. With that kind of motivation, I quickly hacked together a brilliantly elegant mess of code that was able to delta two small binary files in sometime under a day and apply the resultant patch in sometime under an hour. Definitely not my proudest coding moment, but I announced my work in the Fedora Forum (which I’m sure made sense to me at the time).

I got a single response to my post. Rahul Sundaram pointed out, much to my disgust, that Michael Schroeder of Suse had already created an efficient and fast (at least compared to my mess of code) program that could create and apply deltas to rpms. This program was cleverly named deltarpm. After looking at deltarpm and verifying that, yes, it did do what I wanted, I then put the whole project on the back burner for a while.

2006 passed. I got married, in the process missing a war in Lebanon, and spent most of rest of the year trying to adapt to the new rules of married life. Apparently leaving my dirty clothes all over the house was no longer acceptable, and there was some difference of opinion on what the definition of a well-balanced breakfast was. I thought that the peanuts in the Snickers bar did a great job of balancing out my Coke, while my new Irish wife thought that my logic needed to be balanced with a sharp smack on the head. She won. She usually does.

Early in 2007, Ahmed Kamal started posting about some work he was doing on a yum plugin that would download deltarpms. This plugin was named yum-presto. In early March, I got involved and started reworking some of the code. By the end of March, we started testing the plugin with Fedora Core 6, and, in early April, Kevin Fenzi sponsored me and reviewed yum-presto for Fedora Extras. We hit bugs. Then we fixed them. We were on a roll.

Then things ground to a halt. We had the client side working reliably, but there was no way that the Fedora infrastructure was going to use my hacky scripts for generating deltarpms. Someone needed to do some work to properly integrate deltarpm generation into Fedora’s infrastructure, but I had no idea how to go about it.

So I ended up in a rut, daily generating deltarpms for our FC6 i386 mirror, but it was anything *but* official. If anyone wanted to use them, they had to manually change the url in fedora-updates.repo. FC6 turned into Fedora 7, then Fedora 8, then Fedora 9, and finally Fedora 10. Jeremy Katz, Casey Dahlin and James Antill all contributed major changes to the client code, with James essentially rewriting the whole thing, but the infrastructure side of things stayed stagnant.

I opened a ticket against bodhi to try to figure out where the deltarpm generation should be happening, and, after a few rounds of pass the parcel, Seth Vidal finally decided to put it straight into createrepo. With that work done in 2009, Fedora 11 was the first release with official deltarpms across all the platforms that Fedora supported.

Four years have passed since Fedora 11 came out. Now I’m the one trying to convince my kids that Snickers and Coke weren’t meant for breakfast. And yum-presto has been mostly in maintenance mode. Last May, Lars Gullik Bjønnes added code to rebuild the rpms in parallel, and, in August, Zdenek Pavlas added code to download drpms in parallel. Both features made it into Fedora 18. I didn’t realize at the time that those would be the last features added to yum-presto.

On February 21, Zdenek posted this on yum-devel:

The native drpm support is complete enough to be used and tested. I’m quite satisfied with the performance, but the applydeltarpm backend needs more work. I’m going to make a rawhide release today.

Yum-presto had just died. And yum was now able to do things with deltarpms that we never could do as a plugin.

I do want to say thank you to everyone who helped test and develop yum-presto. A special thank you to Michael Schroeder, whose work on deltarpm made yum-presto possible. Thank you to the Lebanon Evangelical School for hosting the i386 deltarpms and Angel Marin for hosting the x86_64 deltarpms for the two years it took to get the Fedora infrastructure up and running. And thank you to all those whose hard work makes Fedora the exciting place that it is.


Comments

Juan Rodriguez
Saturday, Jun 1, 2013

Thank you very much for working on yum-presto. I loved the feature since the day it was announced, and remember testing it before it was widely available / the default in Fedora.

Bandwidth isn’t cheap in many places (Even inside USA), so this plugin was a blessing.

The plugin may be gone, but the concept lives on! :D

matejcepltest
Saturday, Jun 1, 2013

Who was first … your first initial efforts and deltarpm, or the similar functionality for apt-get (I know, that I finished it first … remember partial packages were old stuff in Debian when I was leaving it in 2006)?

Rahul Sundaram
Saturday, Jun 1, 2013

Thank you for all your work. I remember several posts in fedoraforum and users list expressing sheer glee and surprise from the % reduction in download size. That was a nice touch :-) I am glad it is finally merged into yum as well.

Jonathan Dieter
Saturday, Jun 1, 2013

You could very well be right as I’ve never used Debian, but deltarpms were around in Suse before they ever made it into Fedora. My initial efforts were actually quite late in the game, but I didn’t realize that until Rahul pointed it out to me.

Jonathan Dieter
Saturday, Jun 1, 2013

Thank you. I don’t know if you were aware that it was your comment on my original post in fedoraforum that got me involved with Fedora.

Rahul Sundaram
Saturday, Jun 1, 2013

I had no idea until I read your blog post and I don’t remember posting that reply (I used to reply to pretty much every thread at that time) but I am glad I did.

Jim
Saturday, Jun 1, 2013

Thank you for presto, it made me switch to Fedora.

matejcepltest
Saturday, Jun 1, 2013

of course my post should read “it has been finished first”.

Jerez Spa
Sunday, Jun 2, 2013

Great read. I always liked fedora for deltarpm.