Investigate AppStream metadata for building app directory

[probonopd]

The AppStream specification describes the AppStream upstream XML format which is used by upstream software authors to describe the software component (application, firmware, font, ...) they provide, as well as the AppStream distro XML format that is used by distributions to ship the metadata in bundled and processed form to their users.

Ideally, apps would be packaged by the upstream projects, and a repository of apps including version and update information, screenshots, as well as stars (or AppStream kudos) could be built. When I tried to do this a decade ago for klik and later for PortableAppImages, cross-distribution metadata infrastructure was not available, but now it begins to fall into place.

AppStream distribution XML files could be parsed.
Debian collects icons and app metadata here:
https://appstream.debian.org/data/unstable/main/

It looks like AppStream is built with Xapian in mind. Xapian is an Open Source Search Engine Library, released under the GPL. It's written in C++, with bindings to allow use from Perl, Python, PHP, Java, Tcl, C#, Ruby, Lua, Erlang and Node.js (so far!)

[probonopd]

Talked to @aleixpol and he suggests that we should be providing a list of available AppImages in the format of archlinux-appstream-data (use this format but serve the icons on demand under a URL). This would make it easiest for software centers like App Center and Discover to pick the information up. It is especially important to use the same <id> tag for each app as in archlinux-appstream-data so that the same app, when provided via different channels, can be shown to the user as the same app in different variants.

I can imagine (at least) three ways where software centers could get the information about AppImages from:

1. Download it from a database via an URL (the user could have the option to add such URLs in order for the system not to be centralized)
2. Extract it from AppImages already present on the system
3. From the AppStream data that comes with the distribution (it might take quite some time to trickle down)

[probonopd]

For presenting the app directory in the web (in addition to software centers coming with the desktop), possibly @ximion has some code that could serve as a starting ground.

[ximion]

I would need more information on what you are planning ;-)
In general, code to read AppStream metadata no matter in which form and also to do basic writing of the data can be found at https://github.com/ximion/appstream - it is used in Limba as primary data source for pretty much everything (except for some details of the bundle itself).
If you have a repository of AppImage bundles, you could also write a backend for appstream-generator ( https://github.com/ximion/appstream-generator ) which is the tool used at Arch and Debian to build the final metadata that is shipped to users. At time, asgen is built with distro repositories in mind as a data source, but its backends concept is abstract enough to allow consuming not only distro packages. What asgen does compared to "raw" usage of libappstream is rendering icons to the appropriate sizes, caching screenshots and doing a couple of other things (font rendering, searching of icons across multiple packages, ...).

For an app-directory on the web, I am currently experimenting with the vibe.d ( http://vibed.org/ ) framework for developing an online repo service. Vibe has some very nice properties, mainly being similar to NodeJS, but using a language which doesn't drive you insane with crazy levels of callback-nesting. But I am not yet 100% sure whether Vibe will be the final solution, although it's looking quite good so far.
The Xapian cache is only a client-thing, while it can also be used for websites (it's certainly fast enough), some properties, like its focus on only one language, make it less ideal for web use. It is used on the clients to reduce memory consumption of software-centers and to allow some advanced search features, like stemming. Using AppStream and libappstream without Xapian is perfectly possible though (just use AsDataPool instead of AsDatabase in the API).

[probonopd]

Thanks @ximion for the detailed explanation. As for what exactly we are trying to do, well, we are still trying to figure out what would make most sense for a decentral, community-based/crowd-sourced/chain-of-trust type of system.

What seems clear is that with more and more options for users to get software (native distribution package, ppa, COPR, Flatpak, Snappy, AppImage - stable, rolling, nightly) if seems to be common sense that it would be beneficial to offer these choices to the user in a convenient way inside the software centers.

[ximion]

Well, AppStream is tying all of these together and allows them to share metadata.
See the "bundle" tag for metainfo files: https://www.freedesktop.org/software/appstream/docs/chap-DistroData.html#tag-distro-bundle
So, displaying that stuff in software-centers in a way which is nice for users will be a rather simple task, if everything uses AppStream.
For Snappy, this might take a while because apparently someone higher up overruled the engineers to not implement AppStream - but I will be at a Snappy sprint this summer, maybe there still is a way to come to an agreement.

[probonopd]

Looks like you have thought of everything @ximion, yay for the <bundle> tag.

Would something along the lines of

<bundle type="appimage" channel="releases" >Foobar (releases)</bundle>
<bundle type="appimage" channel="development" >Foobar (development builds)</bundle>
<bundle type="appimage" channel="continuous" >Foobar (continuous builds)</bundle>

be allowable, with the actual version numbers being determined on demand, in order to save bandwidth? I am especially thinking of continuous where you'd have to check for the version essentially all the time.

Or is the <bundle/> tag meant to be used inside the <releases> tree structure? In that case, would a release called "continuous" be allowable?

[probonopd]

Note to self, also check https://sessellift.wordpress.com/2016/04/25/a-usability-guys-journey-to-creating-his-first-kde-tool-part-2-a-vision/

[aleixpol]

Don't you want to specify the URL instead of a channel?

[probonopd]

Correct @aleixpol.
In fact, not only direct http(s) download links to the AppImage but alternatively also the update information which can not only be used to update an AppImage but also to download it from scratch.

Will think about how to do this best.

Edit: Maybe it would be best to link to the download page, and have the user download the AppImage from there. This way the website could count downloads, etc. in any way it wants.

[probonopd]

@hughsie had been mentioning the need for a "central registry of AppImages" on https://blogs.gnome.org/hughsie/2016/04/27/3rd-party-fedora-repositories-and-appstream/#comment-5183

So, how would one do a "central registry of AppImages" without being too restrictive about what gets "in"? Right now I can imagine

1. Set clear and lightweight rules and every AppImage that matches those rules gets in
2. Have the possibility to add more than one "central registry of AppImages"

Thoughts welcome.

[ximion]

@probonopd Please read the AppStream spec, especially the distinction from metainfo files vs. distro-metadata.
To put it simply, right now AppStream only allows one installation candidate to be there by design. The component-id (org.example.MyApp.desktop) is supposed to be unique for this particular application. Bundling systems can offer AppStream metadata via some way, having a <bundle type="whatever">${ID}</bundle> tag, where ${ID} refers to some identifier used by the bundling system to install the particular app. Could be, for example, a web URL, or anything the software-center frontend can pass to the app-installer to actually install an application.

If you want to have different "channels" of an application, the only way to realize that right now would be giving them a different component-id. For example org.mozilla.Firefox vs. org.mozilla.FirefoxAurora. Those would have their individual bundle-tag to identify what needs to be installed to make that app available. It also allows an easy way of displaying them in the UI, and, most importantly for me, allows upstreams to set a different description and other metadata for their development versions (e.g. include a warning message that this is untested software).

In my personal opinion, alpha and beta releases should not be in the software sources at all, because having them is incredibly confusing for people, and it creates overcomplicated UI and presents users a decision of taking the newest version, or some older release, which they might no be able to make and usually don't need to make, because there is a released, known to be woring version available.
For the cases where these snapshots are useful, e.g. when working on bugs, providing a bundle outside of the software centers main sources does make a lot of sense.
For the rare cases where having different release channels is useful, giving the software different IDs is justified, IMHO.

[probonopd]

If you want to have different "channels" of an application, the only way to realize that right now would be giving them a different component-id. For example org.mozilla.Firefox vs. org.mozilla.FirefoxAurora.

I can see how this would work for Aurora (which in fact can be seen as a "different application" (has its own name, icon etc.). But if you take Scribus or the GIMP then i'm not sure that treating the development version as a "different application" is the best way.

I also understand that in order not to confuse ordinary users, development/nightly/continuous versions should not be shown too prominently, and there might be a warning when installing them, or a GUI checkbox that would beed to be checked in order for the user to see such versions. Especially with AppImages, I expect one major use case to be development versions (for the very reason that one app = one file makes it super easy to just try something out). So I am convinced that it could be beneficial to have no limit on the number of different versions possible to be offered, including historical ones. Take web browsers, for example - I am sure that web designers, for example, might want to test something not only on the latest but also on other versions of browsers.

That being said, I am even more inclined now to just put in the web link to the project's download page instead of a deeplink to the AppImage. This way, the project can explain the different versions on the download page and provide as many options, versions, or variants as they want, give warnings regarding development versions, etc.

On the following question I would really appreciate some input - what exactly is supposed to go into the bundle tag. An URL from where the bundle(s) can be downloaded? Pro: User can decide which bundle to download, contra: cannot automatically download/update/"install". An identifier that allows an AppImage helper (AppImageUpdate or a gnome-software/discover plugin) to download/update the AppImage? Pro: More automation. Con: How to handle more than 1 version/channel?

Is there any existing convention on what goes into the bundle tag?

<bundle type="appimage">https://krita.org/en/download/krita-desktop/</bundle>

Correct? Or is this reserved for non-HTTP-links and we should use something like

<url type="appimage">https://krita.org/en/download/krita-desktop/</url>

[probonopd]

@ximion One open question is, how do software centers "learn" about bundled software (Flatpak, AppImage etc.)

An AppImage that contains AppStream data should be able to place this to a location in the system where software centers can pick it up.
Looking at the AppStream spec, XMLs are expected to live in /usr/share/app-info/xmls/ or /var/cache/app-info/xmls but for AppImage we need a way to get data in as non-root.

https://mail.gnome.org/archives/gnome-software-list/2016-June/msg00000.html talks about ~/.local/share/app-info/xmls/ - @ximion is this an allowable location?

[ximion]

@probonopd

Looking at the AppStream spec, XMLs are expected to live in /usr/share/app-info/xmls/ but for AppImage we need a way to get data in as non-root.

https://mail.gnome.org/archives/gnome-software-list/2016-June/msg00000.html talks about ~/.local/share/app-info/xmls/ - is this an allowable location?

Not yet, it's something GNOME Software specific, so you can not rely on it being present or used at all.
However, it's on my todo-list to maybe add it to the spec and libappstream.
(in general, in my opinion installing executable apps into the user's $HOME directory is a really bad thing to do, and they should always be stored in a system location (for enhanced security, sharing of resources, ...) But I do see that having some metadata in .local might be useful, so that path might be available for string distro XML in future.
(I will comment on your other post later, lunchtime now!)