[GLLUG] Ubuntu versus Debian (was: Re: GLLUG still alive?)

[Alan Pope]

Hi John,

Gosh, this is a long reply. Sorry. It's an email on the list. The LUG is
now officially *active*.

This isn't a dig at you, John, but more a lament that the Ubuntu
community has moved on, so fewer people are contributing now. There are a
hundred reasons why, but. I only mention a few.

On Tue, 13 Aug 2024 at 21:23, John Edwards via GLLUG <
gllug at mailman.lug.org.uk> wrote:

> But recently we have found that even though packages are within
> support in Ubuntu, fixes which have been made by Debian are not
> backported.
>

I've seen this lament a few times.

Note: They weren't backported before, either, on the whole.

The difference is, that now you're aware of that fact because motd spams
you with adverts for Ubuntu Pro.

You're being deliberately misleading with your words above, and burying the
actual truth much further down the email. They may well be 'within support
in Ubuntu' but as you know, up until recently Canonical only had a
commitment to *some* packages in the repo, the rest (universe) are
maintained by the community. Canonical != Ubuntu.

It's not that they are holding back fixes to packages that were previously
available. They simply were not available. Ubuntu Pro is providing updates
to packages that previously were not in scope, and so didn't happen.

Unfortunately, the amount of work to update packages and backport fixes far
exceeds the number of open source community members to maintain them. Same
as many open source projects these days.

What's changed is Canonical are offering this (backporting selected fixes)
as a paid service to their enterprise customers. Supporting those paying
customers by backporting fixes they would never have had before. Canonical
is enabling customers to accelerate and focus development effort on
packages Canonical would previously not have touched. Like most software
companies, if you pay, you get attention. This isn't exactly new.

They then make that paid work available for free to the Ubuntu community if
they enable Ubuntu Pro.

If Canonical just published the updates immediately, in the clear, where's
the motivation for an enterprise to pay them for the fix?

"Why should we pay you for this security update, you're publishing it in
the clear"
"You're paying us to do the work"
"We'll wait then"

The work then doesn't happen, the updates don't exist, and nobody gets
them. Everybody loses.

Paying enterprise customers can hold off their distro upgrades, retain
20.04 or even 18.04, Ubuntu Pro Free Tier users (the majority) get updates
they never got before, and those who refuse to sign up for Ubuntu Pro on
principle get to bitch about Canonical online.

What's not to love, everyone's a winner! :)


> It was a simple fix to do with library dependencies, which had been
> already been fixed in Debian months *before* the Ubuntu package was
> released:
>         https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000034
>
>
A "simple fix" in one package of thousands. You, me, and countless others
will likely say the exact same thing about a hundred different packages. We
all have our favourite "surely they can fix this in a minute" bugs. Yet if
we all put a patch in Launchpad, and nudged a Canonical or community
developer to sponsor, chances are it could be picked up and everyone would
benefit.

The very point of the community. You know this. :)

I suspect if Ubuntu didn't have so much process, and wasn't so wedded to
Launchpad, they might get more drive-by fixes. One can dream :D

So we've ended up having to backport packages from Debian, at which
> point we thought we may as well use Debian.
>
>
Or, in another timeline, you contribute that work to Ubuntu, and share the
workload, like you're asking Canonical to do for free, at their cost?


> Yes, these packages are in the Ubuntu "universe" rather than the
> "main" repository and are supposed to be community maintained, but
> community users can not upload to the release repositories (only their
> own repositories) so most people do not get to see those fixes unless
> they carefully read the bug reports, and so end up with broken or
> vulnerable software.
>

No. Members of the community can have ownership of packages in the universe
pocket of the repo via "MOTU (Mistresses Of The Universe) and
other initiatives. That's been the case forever. Sadly, with numbers
dwindling, there's fewer community members on hand to do that work, and an
ever increasing number of packages to maintain.

Here's the package uploaders to the last ~two weeks of changes to the most
recent LTS only.

* https://lists.ubuntu.com/archives/noble-changes/2024-August/thread.html

$ lynx
https://lists.ubuntu.com/archives/noble-changes/2024-August/thread.html
--dump --width 132 | grep "(Accepted)" | rev | cut -d ')'  -f 1 | rev |
sort | uniq

These (26) people are (probably) indeed paid Canonical employees (to my
best guess):
   Aaron Rainbolt
   Alex Murray
   Andreas Hasenack
   Andy Whitcroft
   Benjamin Drung
   Brian Murray
   Chloé 'kajiya' Smith
   Dave Jones
   Eduardo Barretto
   Evan Caville
   Felipe Reyes
   Ian Constantin
   Jeremy Bícha
   Julian Andres Klode
   Lena Voytek
   Leonidas S. Barbosa
   Łukasz Zemczak
   Luci Stanescu
   Marc Deslauriers
   Matthias Klose
   Michael Hudson-Doyle
   Nathan Pratta Teodosio
   Nick Rosbrook
   Paride Legovini
   Simon Chopin
   Utkarsh Gupta

These (4) are not paid by Canonical, but are able to update packages in
universe (and perhaps other specific packages):
   Erich Eickmeyer
   Scarlett Moore
   Sudip Mukherjee
   Lukas Rettler

They're volunteering their time - significant amounts of it in the case of
Scarlett, to maintain a ton of packages that Canonical are not, and never
were on the hook for.

I'm not sure if this is lack of staff within Canonical or a breakdown
> between them and the wider community.
>

Canonical has the highest number of employees now than it has ever had in
the last 20 years. So it's not just numbers of people, I suspect, but their
priorities. They will have paid work doing hardware-enabling new machines,
backporting patches for OEMs, building packages for partners and all manner
of other things to keep the wolf from the door.

Those engineers have less and less time to take contributions from the
community and sponsor them for release. It's very sad. but actually, it's
always been the case. When the flood of new patches from the community
turns into a trickle, it gets worse because everyone took their eye off the
ball.

I think (throwing back to the OP of this thread) it's at least partly a
by-product of community people moving on to other more interesting things.
Some will have burned out, others switched distro, may have taken up
hardware hacking, died or something else.

Some enthusiastic contributors have switched to rolling, or
faster-releasing platforms like NixOS and Arch. It's a massive motivator to
see your contributions picked up quickly, rather than languish in a queue
waiting for some lone developer at Canonical to pick them up.

Something Canonical (and the wider community) has never been good at is
recruiting new people to replace those that age out or burn out. So over
time the community has atrophied, and very few people stepped up to take
their place. Some have, of course, but not enough.

And to add insult to injury Ubuntu now pridely displays a message at
> every login telling you how many packages you have installed for which
> you will not get any security updates unless you sign up for Ubuntu
> Pro.


Canonical has never been "allowed" to advertise any paid service
whatsoever. Despite Ubuntu Pro not actually being a paid service for the
vast majority of users. I have Ubuntu Pro enabled on 8 machines out of my
allocation of 50, with no expiry. Never paid a dime, but I have contributed
over the years.

To my mind, the packages are there, the work is already been done,
> mostly by people not employed by Canonical, so to release vulnerable
> software and not fix it is rather unethical.
>

How do you solve this problem, exactly? Put them all up in the clear? As I
said before, then nobody will pay for them, because they can just enable
the repo that contains these fixes and never pay a dime. Someone, somewhere
has to pay for this. Just like people pay Debian developers. Indeed,
Canonical pays Debian developers, too.


> ps. As a business, the company I work for did used to pay Canonical
> for Ubuntu support many years ago. But they don't any more (lack of
> quality and poor response times).
>

I've heard that before too, sadly.

Okay, everyone can go back to sleep again now.

Cheers,
Al.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.lug.org.uk/pipermail/gllug/attachments/20240814/996ff76b/attachment-0001.htm>