Free Software Directory:Infrastructure

From Free Software Directory
 
Jump to: navigation, search

Development server

See Free_Software_Directory:Backlog_Admin_Group#Testing_instance

Coming changes in MediaWiki

MediaWiki 1.38 (that eventually will be upgraded on Directory) comes the Vector 2022 desktop skin

"Vector 2022 is a desktop skin developed between 2019 and 2023 by the Wikimedia Foundation Web team with the goal of making the interface more welcoming and usable for readers and maintaining utility for existing editors... Today (January 18, 2023), the skin was turned on as the default on the desktop site. Users of non-default skins (Monobook, Timeless, etc.) will not see any changes. "

"its first new skin since 2010". - https://slate.com/technology/2023/01/wikipedia-redesign-vector-2022-skin.html

Vector 2022 comes with MediaWiki 1.38: https://www.mediawiki.org/wiki/Bundled_extensions_and_skins#Skins

Current

As described in our stack, the FSD specifically uses:

Download the Free Software Directory MediaWiki database (obsolete)

"Wikipedia offers free copies of all available content to interested users." - https://en.wikipedia.org/wiki/Wikipedia:Database_download

The FSD followed this approach and offered free copied of the wiki in the past:

People interested:

See also:

Archiving

Send archives to the Software Heritage (supported by UNESCO).

Tracker

This page has to many bugs to effectively organize them. We need a dedicated bug tracker for the Directory.

The concept for public bug trackers for FSF run websites has been posted in Directory-Discuss mailing-list.

Matt Lee said that he back in January 2017 but nothing happened and he never replied!

Ian discussed this with Ruben who suggested GitLab Community Edition since ruben has used it for the trisquel.info. Ian is willing to adapt it but found [some problems with it regarding the license fields]. Ian plans to submit a patch programmed in Ruby to solve the issue.

Bug Tracker Comparison by Sonali

GitLab

    Famous
    Attractive Web UI
    Can appoint multiple assignees per issue
    Set Milestones
    Categorise and filter Issues
    Granular user roles
    Community edition is free
    More features than just a bug tracker
    Very versatile and extensive features


Trac

    Minimalistic web ui
    Used by VideoLan (https://trac.videolan.org/vlc/ticket/19703)
    Allows wiki markup
    Has main features
    Written in python
    Lacks some collaborative features present in GitLab
    Not very user friendly: for the people who have a habit of
    working on GitHub and similar platforms


Redmine (demo on http://demo.redmine.org/)

    Very less content visible unless logged in 
    Simple web interface
    Multi language support though translations may
    not be up to date (written on their site)
    Provides a Gantt chart
    Base design isn’t as flexible
    Setup of ruby and rails require some expertise 


Debbugs

    Used by Debian (https://www.debian.org/Bugs/Reporting)
    Tickets open only through emails
        Might be hard for people who work only on web based platforms
    Useful for people who are already familiar with it
    Search is mainly based on looking up the email threads 
    May be hard to keep track

Debbugs + mailing list

"As previously discussed on bug-gnuzilla, the Bug Tracker for GNUzilla on Savannah is now closed [(Fri 20 Dec 2019)] to new issues, and new issues are to be reported to our bug-gnuzilla mailing list, now integrated with the Debbugs-based GNU Bug Tracker.

Note that this Bug Tracker is currently not in a complete read-only state, and commenting on existing items is still enabled." - https://savannah.gnu.org/bugs/?57455

Has the FSF discussed whether the same solution should be used for the Free Software Directory?

Resolution

We have developed our own tracker.

New anti-feature templates

This is an excellent first task for our new infrastructure.

The anti-feature text for all entries in https://directory.fsf.org/wiki/Free_Software_Directory:Antifeatures should not be part of the main page since we want to update the main page with pure .wiki files generated from Debian meta-data. Instead the anti-feature wiki text should be placed in special pages that are included in the main page.

Example for Adblock Plus, place this text in Antifeature:Adblock Plus.

{{Antifeature
|Antifeature_description={{AttentionBox
 |title=Antifeature: Tracking comment
 |color=red
 |icon=<span style="font-size: 250%;">⚠</span>
 |message=Adblock Plus is ineffective for surveillance protection by default as it comes with ''Acceptable Ads'' enabled: These ads are not meant to be "ads that don't track you".
}}
|Antifeature_tags={{Project antifeature
|Antifeature=Adware
}}
{{Project antifeature
|Antifeature=Tracking
}}
}}

Data

See also Free_Software_Directory:Backlog_active#Import

Debian Packaging System

Disclaimer: The Debian Package Tracking System provides no software description needed for |Short description= and |Full description= in the entry pages.

The Debian Package Tracking System produces RDF metadata and is already included in DBpedia. For example:

External links

Repology

Import

"Repology monitors a huge number of package repositories and other sources comparing packages versions across them and gathering other information. Repology shows you in which repositories a given project is packaged, which version is the latest and which needs updating, who maintains the package, and other related information."

Export unique FSD entries

New repository: Free Software Directory

F-Droid

Flatpak

flatpak remote-modify --subset=floss flathub

Not FSF approved currently.

FreeAMO

http://git.savannah.gnu.org/cgit/directory.git/tree/subprojects/freeamo

FreeAMO is a project used to get meta-data for free add-ons at
addons.mozilla.org (AMO) used to produce free add-on repositories for
free Mozilla-based programs.  contain nonfree add-ons and Mozilla
Foundation's addons.mozilla.org contains nonfree add-ons, and their
programs including Firefox, Thunderbird, SeaMonkey, and Sunbird are
trademarked and thus nonfree.

Data access

Public API

Related: MediaWiki API (example, https://www.mediawiki.org/w/api.php?action=query&list=search&srwhat=text&srsearch=GNU&format=json&srsort=last_edit_asc)

From: Bone Baboon
Subject: Re: FSD as a Git repository
Date: Sat, 17 Jul 2021 18:30:23 -0400
Archive link: https://lists.gnu.org/archive/html/directory-discuss/2021-07/msg00035.html

Has the FSD's MediaWiki API been made publicly accessible?

If the FSF has not made the FSD's MediaWiki API publicly available
yet then it would have to consider if it wants to make the API public
and maintain it.

I asked some question in #mediawiki@libera about MediaWiki's API.  These
links for the API documentation where shared:

<https://www.mediawiki.org/w/api.php>
<https://www.mediawiki.org/wiki/API:Edit>
<https://www.mediawiki.org/wiki/API:Main_page>

There is also an in development REST API that has "much less
functionality": 

<https://www.mediawiki.org/wiki/API:REST_API>

...

When I asked about MediaWiki API clients in #mediawiki@libera:
* this link was shared <https://www.mediawiki.org/wiki/API:Client_code>
* someone suggested pywikibot

Data model

See also: https://www.mediawiki.org/wiki/No-JavaScript_notes

Git

From: Bone Baboon
To: directy-discuss@gnu.org
Subject: FSD as a Git repository
Date: 2021-07-12 22:41
Archive link: https://lists.gnu.org/archive/html/directory-discuss/2021-07/msg00010.html

I wanted to ask what people think about the idea of the FSD being a git
repository.  This idea is motivated by this email thread.
<https://mail.gnu.org/archive/html/directory-discuss/2021-06/msg00017.html>

Here is an example of how the FSD could work as a Git repository:

* The Git repository would have the normal readme, contributing, license
  files. 
* The FSD contents could be organized with a clear and deliberate
  directory hierarchy. 
* The directories and files would have clear and deliberate names.
* The files could be written in a simple plain text format (Markdown,
  org-mode, etc). 
* A static website of the Git repository and it's source code could be
  generated using free software tools like stagit and cgit. 
* The FSD's plain text files could be compiled to HTML and served as a
  static web site without JavaScript using free software (discount,
  org-mode, etc). 

There would be trade off between using the current MediaWiki setup and
using a Git setup.

Both Git and MediaWiki provide:
* Metadata on contributions
* History of changes

Git advantages:
* No graphical web browser required.
* No JavaScript required.
* A text web browser would be optional.
* A contributor does not need a computer with a graphical environment.
  Not all GPU's have free firmware. 
* A contributor would not have to sign up to be able to make
  contributions.  (signing up currently does not work without JavaScript
  and cookies) 
* The Git repository could be cloned and worked on offline.
* Free software tools could be used to work on the Git repository.
* Contributors can make edits with their free software text editor of
  choice. 
* Static websites without JavaScript for browsing the Git repository and
  source code would be accessible with text web browsers.
* Static websites without JavaScript rendered from the FSD's Git
  repository would be accessible to browse with a text web browsers.
* Once the content is in a plain text format it would be easier to move
  to another plain text format using free software (Pandoc for example)
  if that was desirable in the future.
* Simpler server administration without a database.
* Uses less system resources on the server.

Git disadvantages:
* Requires learning how to use Git.
* Requires learning a simple markdown format.
* Requires a data migration initiative to move the FSD's data from it's
  current MediaWiki format into plain text files.
* Requires individuals with commit access to the Git repository to
  accept patches submitted by contributors.

MediaWiki advantages:
* The FSD already uses MediaWiki.
* Currently works for contributors who have graphical browsers.
* Has forms that provide validation checks on entered data.
* Has reusable templates for common content.

MediaWiki disadvantages:
* Requires learning the how to use MediaWiki.
* Does not work well with text web browsers.  Can not sign in with EWW
  Emacs's built in web browser for example.  Not all GPU's have free
  firmware and are able to run a graphical web browser.
From: bill-auger
To: directy-discuss@gnu.org
Subject: FSD as a Git repository
Date: 2021-07-14 08:17
Archive link: https://lists.gnu.org/archive/html/directory-discuss/2021-07/msg00022.html

FWIW, mediawiki has a complete REST API - it can be accessed from
the command line with curl, etc - all of the discussion so far
has the regrettable presumption, that a web browser is required
to read/write data to/from the internet, or to do so in an
"accessible" way

API clients are easy to write; and there are clients made for
mediawiki already (weboob alone, has multiple plugins for
mediawiki) - AFAIAC, that would satisfy the text-only use-case,
just as well as gopher or anything else would, but without
replacing the wiki, migrating all of it's data, or even patching
any upstream code - the FSF would only need to enable API
access, if it is not already

if some web service has a complete remote API, that is already
the ideal option for text-only, no-js, accessibility, etc - an
API is fully accessible by its nature (no colors, no mouse
buttons to locate, etc)

in short, the best javscript is 'none', and the best web browser
is 'none' - no need to re-invent the wheel - mediawiki already
supports it

Wikibase

I propose Wikibase: wikidata.org uses Wikibase -- It goes _much_ faster wikidata.org than directory.fsf.org. (david hedlund)

Wikibase is a triplestore implementation. "Technical support: the data model should allow for adequate representations in existing data formats, e.g., JSON or RDF/OWL" - https://www.mediawiki.org/wiki/Wikibase/DataModel


my evaluation: Iank (talk) 10:40, 18 July 2018 (EDT)

Upsides:

It would give an api to update data. With semantic mediawiki, you have to update entire pages. This would allow better syncing from sources like mozilla and debian.

We could also use it to pull in data from wikidata, but that could also be done without adopting wikibase for our own data.

Downsides: wikipedia's use of it seems to be mostly around linking between languages, and a very small use of the actual data, and where it is used, it's not user friendly, the data magically appears from a template that that has a wikidata reference number and there is no easy way for users to navigate to that data to edit it.

https://en.wikipedia.org/wiki/Wikipedia:Wikidata

At this point it does not seem high priority to adopt it, but it's got potential if wikipedia creates a better user interface for it.

Page structure

#####################################################################
Import and export data

https://directory.fsf.org/wiki/<entry>
|
|--<entry>-debian.wiki -- generated by https://savannah.gnu.org/projects/directory
|		* This is a Debian package: Yes
|		* Package name
|		* Version
|		* Release date
|		* Dependencies (depends, recommends, suggests, enhances)
|--<entry>-user.wiki -- user contributions on https://directory.fsf.org/wiki/
		* Download link
		* OpenPGP signature URL
		* Donate
		* VCS Checkout
		* Categories
		* IRC channel
		* Etc

Example:

https://directory.fsf.org/wiki/Emacs
|
|--Emacs-debian.wiki
|		* This is a Debian package: Yes
|		* Package name: Emacs
|		* Version: 26.1
|		* Release date: 28 May 2018
|		* Dependencies: <will be listed here>
|--Emacs-user.wiki
		* Download link: http://ftpmirror.gnu.org/emacs/
		* VCS Checkout: git clone git://git.sv.gnu.org/emacs.git
		* IRC channel: irc.freenode.net/emacs


#####################################################################
Form edit

https://directory.fsf.org/wiki?title=Emacs&action=formedit&debug=true

if debian package == yes
	Package name: Readonly
	Version: Readonly
	Release date: Readonly
	Dependencies: Readonly
	Download link: Write
	VCS Checkout: Write
	IRC channel: Write
else
	Package name: Write
	Version: Write
	Release date: Write
	Dependencies: Write
	Download link: Write
	VCS Checkout: Write
	IRC channel: Write

Issues

Resource links

Bendikker has added hundreds (if not thousands) of resource links to various entries. I think we should import them if possible. Here's what he added to Emacs for example

{{Resource
|Resource audience=Python (Ref)
|Resource URL=https://pypi.org/project/emacs
}}
{{Resource
|Resource audience=Debian (Ref) (R)
|Resource URL=https://tracker.debian.org/pkg/emacs23
}}
{{Resource
|Resource audience=Debian (Ref) (R)
|Resource URL=https://tracker.debian.org/pkg/emacs24
}}
{{Resource
|Resource audience=Debian (Ref) (R)
|Resource URL=https://tracker.debian.org/pkg/emacs25
}}
{{Resource
|Resource audience=Debian (Ref)
|Resource URL=https://tracker.debian.org/pkg/emacs
}}
{{Resource
|Resource audience=Savannah (Ref)
|Resource URL=https://savannah.gnu.org/projects/emacs
}}
{{Resource
|Resource kind=VCS Repository Webview
|Resource URL=https://git.savannah.gnu.org/cgit/emacs.git
}}
{{Resource
|Resource kind=Mailing List
|Resource URL=https://lists.gnu.org/mailman/listinfo/help-gnu-emacs
}}
{{Resource
|Resource kind=Mailing List
|Resource URL=https://lists.gnu.org/mailman/listinfo/bug-gnu-emacs
}}
{{Resource
|Resource kind=Mailing List
|Resource URL=https://lists.gnu.org/mailman/listinfo/emacs-devel
}}
{{Resource
|Resource kind=Mailing List
|Resource URL=https://lists.gnu.org/mailman/listinfo/gnu-system-discuss
}}
{{Resource
|Resource kind=Download
|Resource URL=https://ftp.gnu.org/gnu/emacs
}}

Rejected

Single-platform free software

Currently only free software that run on GNU/Linux have their own pages on the FSD.

https://en.wikipedia.org/wiki/Category:Single-platform_free_software is a useful resource of notable non-GNU/Linux software.

Candidate OS by priority

  1. Android apps: Replicant is a fully free Android-like distribution that meets the Free System Distribution Guidelines (GNU FSDG).[1]
    1. Android is partially proprietary but Replicant is fully free. So adding Android app pages to the FSD could be compatible with Free_Software_Directory:Requirements: "If a software program runs on proprietary operating system, then the version of the program that runs on free software operating system should be as good or better." -
  2. Proprietary OS: Windows, iOS, and macOS
Android (Replicant) apps

From https://lists.gnu.org/archive/html/directory-discuss/2021-07/msg00066.html "Currently the Directory (or FSD for short, without G at the end) proudly states that all software listed there runs on GNU/Linux ([1]), but propose that we lift this requirement.

This would allow small projects such as Replicant to:

  • Start contributing to reviews of packages that are specific to their

base kernel (Android, CyanogenMod or LineageOS as far as I can remember).

  • Take advantage of the review process and structure of the FSD

(including some of the MediaWiki and Semantic MediaWiki features and current entry data organization).

  • Have even more visibility in the long run. I know this might be

unrelated or could be done anyways but, to give an example, it would be possible for a Replicant contributor to schedule on their agenda to also do Replicant-specific reviews at the same time as the FSD Friday meetings and so focus only on two or three chat windows, and when someone comes asking for Android/mobile freedom-related topics the Replicant contributor will already be there to catch the opportunity.


  1. References

[1]: https://directory.fsf.org/wiki/Category/Runs-on/GNU/Linux" - Adonay

Why single-platform free software should have their own pages instead of external links in the FSD

From https://lists.gnu.org/archive/html/directory-discuss/2021-08/msg00000.html

"I don't think we should let external links to possible free/libre software appear in public-facing pages unless we make create proper entries for those software.

The Directory is a place where anyone can add software entries for review, then some evaluators/reviewers do one or two review checks (with varying depth of detail) and approve it or not in a way similar to saying that is, at that time and to that specific version, considered free/libre. If we simply link to a possibly free/libre software without actually reviewing it, we run the risk of, for example, end up linking to a software for Windows that depends on Microsoft Visual C++ Redistributable.

Also, I don't know if you recall this David Hedlund, but when we created the Collection namespace, both of us agreed that it would be used as a means of listing already-reviewed and curated/selected free/libre software that do some task to replace non-free ones." - Adonay


Group chat system for the FSD Friday IRC Meetings

Matrix and XMPP

From: David Hedlund
Subject: Re: Matrix and XMPP (was: hackint IRC channel for FSD)
Date: Sat, 17 Jul 2021 04:08:00 +0200

On 2021-06-30 16:27, Adonay Felipe Nogueira via wrote:
> Em 30-06-2021 10:01, David Hedlund escreveu:
>> Can you start a new thread on this discussion and suggest Matrix in
>> addition to XMPP?
> I wouldn't be in favor of Matrix, various clients have accessibility
> issues, specially for people like me that have subnormal monocular sight
> (since many clients often impose font sizes, element colors or spacing
> in a way that conflicts with system-wide preferences or with their own
> styles). For example, for Element Web, if one tells the browser to force
> system-wide fonts and sizes or when setting Element Web's font size to
> 20pt, the default style turns into a headache waiting to happen (see
> attached screenshot), so one still has to learn how to program in their
> browsers' native extension language, how to fiddle with CSS, and figure
> out how to bring the style inspector and build the CSS selectors for the
> right elements they want to tweak in order to have a
> subnormal-monocular-sight-friendly Element Web Matrix client. For other
> clients, like Revolt and Quaternium, the configurations for that are yet
> to be known.
>
> Also, contrary to XMPP which has at least two fully-functional
> free/libre software clients, for Matrix there is only Element non-Web
> has full functionalities of the protocol, and that is not even
> free/libre, because at least in Element Web you can't use the client's
> own search feature for conversation or media histories in encrypted
> conversations.
>
> Furthermore, the Matrix protocol itself is not internationally
> standardized (but rather auto/self-standardized, meaning that they can
> change that at any time), and some clients allow for link forgery since
> the Matrix protocol didn't dictate what would be the message format and
> clients decided to implement Markdown formatting, which is dangerous in
> the context of instant communications.
>
> Discarding audio/video conferencing support, Matrix servers take up more
> resources ([1]) than XMPP with all suites/parts of XEP-0443 ([2], again
> discarding audio/video conferencing).
>
> Also just like Matrix, XMPP servers and clients that support XEP-0313
> for both individual chats and group ones (MUC or MIX, although MUC is
> more popular) do provide ways for the user to catch room history
> (messages received by the room while that person was offline)
>
> While FSDG-fit distros have plenty of XMPP libraries, clients and
> servers, Matrix-related ones are lacking there, so one often has to
> thrust the single-language package manager installation instructions
> given by the library/client/server that you want to install, unless
> there is a FSDG-centric review of each of those
> apk/pip/npm/gems/docker/flatpak/rocks/cargo/snap packages/dependencies.
>
> To balance the sides, both Matrix and XMPP *do not* provide default
> bots/puppets/bridges/transports to other communication protocols. I was
> once lured by a person to investigate that since they claimed that
> Matrix had bridges for IRC, XMPP, and others *by default*. But it turns
> out that this is not the case, and you have to either trust the
> centralized Matrix.org or risk installing one yourself through the
> methods in the previous paragraph.
>
> Finally, there is also the privacy and centralization issues regarding
> Matrix, since it is mostly centered on Matrix.org (while there are
> instances that do provide account creation, only Matrix.org has
> bots/bridges/puppets which are clearly documented and exposed/discoverable).
>
>
> # References
>
>
> [1]: https://disroot.org/en/blog/matrix-closure .
>
> [2]: https://xmpp.org/extensions/xep-0443.html .
>
> [3]: https://xmpp.org/extensions/xep-0313.html .
>
It seems like the FSF already has evaluated the possibility to use XMPP and Matrix, they declined both of them.

From https://www.fsf.org/news/fsf-and-gnu-move-official-irc-channels-to-libera-chat-network: "Despite its age, IRC remains a strong favorite of the free software community. Although we are optimistic about the Matrix protocol and remain committed to following its development closely, we were not able to justify a full relocation of the FSF and GNU's official channels to a Matrix server. Doing so would create the unacceptable situation of encouraging a large number of users to run nonfree software in the form of nonfree _javascript_, which is used by the flagship Matrix.org server to authenticate users.

At the same time, we could not commit to moving fully over to XMPP, which would impose certain technical limitations on both users and FSF staff, and which does not offer many compelling advantages over IRC. We've also definitely heard from many of our members showing renewed interest in the XMPP server the FSF provides as an associate membership benefit, and we are looking at the possibility of devoting more resources to it. To reiterate, though IRC remains a key venue for communication in and around GNU and FSF, we are keeping an open mind and eye towards other existing or new communication protocols and software, including Matrix and XMPP, that enable users to communicate in freedom."


Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the page “GNU Free Documentation License”.

The copyright and license notices on this page only apply to the text on this page. Any software or copyright-licenses or other similar notices described in this text has its own copyright notice and license, which can usually be found in the distribution or license text itself.