Free Software Directory talk:Artificial Intelligence Team

Software classification for FSF assessment

To assist the FSF in evaluating the page at https://www.fsf.org/news/fsf-is-working-on-freedom-in-machine-learning-applications, we organized the software categories here accordingly. This structure aims to clarify the criteria for determining when a machine learning application can be considered free, ensuring that users are empowered to control their computing.

Examples

Opening-up-chatgpt.github.io

• Availability: Free code, LLM data, LLM weights, RL data, RL weights, License
• Documentation: Code, Architecture, Preprint, Paper, Modelcard, Datasheet
• Access: Package, API

USA Bipartisan House Task Force Report on AI (Dec '24)

Federal AI Governance and Transparency (Page 4)

• Data and Metadata: information about the data that was used to train, test, or fine-tune the model, including information about the data’s sources or provenance, collection methods, sample size, procedures for cleaning the data, bias and skewness, inclusion of protected characteristics or proxy features, and ultimate integrity.
• Software: information about the software components and their origins.
• Model Development: information about the training, tuning, validation, and testing of the AI system, who requested its development, who developed the model, communities consulted in development, the development process, tools used in development, the model’s intended uses and known limitations, and metrics pertaining to the model’s efficiency, performance, bias, and energy usage.
• Model Deployment: information about model deployment and monitoring, including metrics identified in model development, plans to provide notice and explanation of the use of AI models to members of the public impacted by the model’s use, and any ongoing training, validation, and testing.
• Model Use: information about how the model is used, including the organizational context and design of the entire system in which the model is deployed, specific use case applications of the model, the information that a deployed model utilizes, the types of determinations or decisions the model is intended to inform, meaningful explanations of the model and its outcomes given relevant stakeholders, the policies for how to handle outputs, the risks of harm identified, and risk mitigation plans including human oversight or intervention.

Mechanics

It's differentiation on N-dimensional arrays. (n-dimensional means n-amount of axes i.e. x,y,z)

Mess with variables: predicted, actual, and error.

Python

Array libraries

• Numpy

Differentiation libraries

• PyTorch
• TensorFlow
• JAX
• MyGrad

Combatting censorship

Decentralized LLMS

Distributed training

Prime Intellect [Apache 2.0]

Peer-to-peer generation

"LocalAI [MIT] uses https://github.com/libp2p/go-libp2p [MIT] under the hood, the same project powering IPFS. Differently from other frameworks, LocalAI uses peer2peer without a single master server, but rather it uses sub/gossip and ledger functionalities to achieve consensus across different peers.

EdgeVPN [Apache 2.0] is used as a library to establish the network and expose the ledger functionality under a shared token to ease out automatic discovery and have separated, private peer2peer networks.

The weights are split proportional to the memory when running into worker mode, when in federation mode each request is split to every node which have to load the model fully." - source

Isolating a web service like Gradio from internet access

Ensures less snooping on local models - Linux Safe Web Service

Decensoring censored models

A guide to decensoring models; Exercise caution, as it stands to reason an inherently uncensored model would perform better than needing the legwork of decensoring one (and then making mistakes + missing some of the censorship)

Examples of censorship

• If you ask Deepseek-V2 (through the official site) 'What happened at Tienanmen square?', it deletes your question and clears the context.

Testing models

Tools are needed to assess the pros/cons of each model.

Meta-Leaderboards

• HuggingFace LeaderboardFinder
• LLM Explorer

Leaderboards

Due to the issue of merely training a model to become good at whatever tests are on a leaderboard, multiple leaderboards are preferential (hence not putting HuggingFace on the main page). A more comprehensive evaluation would be a meta-analysis of existing leaderboards.

Text

• AlpacaEval

• Another LLM Roleplay Rankings

• Ayumi Benchmark ERPv4

• Azure AI Studio Model benchmarks

• Chaiverse

• EQ-Bench (Emotional Intelligence Benchmark)

• EQ-Bench (Creative Writing Benchmark)

• Googlesheet of models, AI labs, datasets, and various other ML info by Alan Thompson

• HuggingFace Open LLM

• HuggingFace Uncensored General Intelligence

• Kagi Search's llm chess puzzles

• LiveBench: A Challenging, Contamination-Free LLM Benchmark

• LLM Openness

• LMSYS Chatbot Arena

• MixEval

• NoCha

• OpenCompass (China)

• OpenRouter LLM Rankings

• Predibase's Open-source Model Fine-Tuning Leaderboard

• SEAL leaderboards

• Simple Bench - Basic Reasoning

• Toqan's ProLLM Benchmarks: Summarization

• Toqan's ProLLM Benchmarks: Q&A Assistant

Coding

• Aider

• ClassEval

• CRUXEval

• EvalPlus

• LiveBench: A Challenging, Contamination-Free LLM Benchmark

• LiveCodeBench

• SEAL leaderboards' coding prompt set

• TabbyML Coding LLMs

• Toqan's ProLLM Benchmarks: Coding Assistant

Voice

• Artificial Analysis Speech to Text (ASR) Leaderboard
• Huggingface TTS Arena: Benchmarking TTS Models in the Wild
• Open TTS Tracker

Benchmarks

• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• PremAI's inference engine benchmarks Hint: top ones are Nvidia's TensorRT, exllamav2, vllm, llama.cpp

Architectures

• Transformers
  • Package manager and player: Transformers - https://github.com/huggingface/transformers/
  • Original science paper: Attention Is All You Need

• Hyena
• Mamba/Jamba

• List of popular text-to-image generative models with their respective parameters and architecture overview

Ordinal value scales could exist for

• indistinguishability from human creations - e.g. Human or Not? social turing chat game
• inference speed
• length of memory
• trivia accurateness
• computation costs: cpu/vram/ram mhz

Source of model training data

• amount of data
• date range (e.g. distinguishing old science from new science for smaller scale models)
• level of censorship (important to make personal+research use distinct from business use)

Problem solving

• math
• creative problem solving (there exists methodology for testing this in humans)

"The present findings suggest that the current state of AI language models demonstrate higher creative potential than human respondents." Nature (Feb 10, '24)

Computation Costs

• Computation costs and datasets for many models
• LLaMA 2 performance reached with $100k
• Image Stable Diffusion performance reached with $2k

General trends

• Larger models are more prone to human superstition[1], but also generate more human-like readability.
• Quantization (a la GPT-Q) allows consumer hardware to run large models.

Legal

• UN Resolution: Seizing the opportunities of safe, secure and trustworthy artificial intelligence systems for sustainable development

Potential Freedom issues

• Dependencies need to be checked. (e.g. LibreChat)
• Verify whether a workflow requires non-free GPU or if CPU can be used.
• The training data often contains non-free licensed material.
  • According to current copyright laws, this does not impact the license of the model or the output of the model. According to current copyright laws, the output is public domain. Mmcmahon (talk) 11:48, 2 May 2023 (EDT)

Model licenses

There appears to be a swath of custom model licenses being used independent of the more standardized software licenses used to interact with models. This presents a conflict as to what license is deemed applicable to the files contained in any repo.

Reddit - Security PSA: huggingface models are code. not just data.

This video (starting at 16:50) illustrates a good argument that model checkpoints may not fall under copyright protection so traditional software licenses that depend on copyright law would be invalid. The video does illustrate that contract law may try to be used it place of copyright. I would advise not using YouTube directly and instead using yt-dl or Invidious.

Worth noting: Open LLaMA (out-of-date) is an example of removing the issue of model cards for text generation.

StabilityAI keeps (Nov '22) updating (July '23) its licenses (Nov '23), so I've removed them from the main page until they settle down.

GLIGEN and GLIGEN GUI is quite neat, but states strict model terms and conditions associated with using it.

Image Models

Stable Diffusion

Stable Diffusion model files (.ckpt) are released under a non-free license.

Here's the stable diffusion beginning point: https://huggingface.co/CompVis/stable-diffusion-v1-4 https://huggingface.co/spaces/CompVis/stable-diffusion-license

stable-diffusion-webui

• https://github.com/AUTOMATIC1111/stable-diffusion-webui
  • Demo, and guide - https://www.youtube.com/watch?v=R52hxnpNews
  • Depends on stable-diffusion repository instead of diffusers. The current branch of stable-diffusion only has a model license which does not make sense for code. Older commits are MIT.
  • Extensions:
    • Deforum

Free software replacements that are missing

• AI Research Assistant
  • https://elicit.org/ - Elicit uses language models to help you automate research workflows, like parts of literature review.
• Voice to instrument: Tone Transfer-like
• Identification
  • Photo
    • Pl@ntNet for Android - Pl@ntNet is a citizen science project for automatic plant identification through photographs and based on machine learning. "The observations shared by the community are published with the associated images under a Creative Common CC-BY-SA license (visible author name)." - https://plantnet.org/en/2020/08/06/your-plntnet-data-integrated-into-gbif/
  • Audio
    • Shazam: Shazam is an application that can identify music, movies, advertising, and television shows, based on a short sample played and using the microphone on the device.
    • A Shazam-like software that is identifying genres instead of songs.
    • A free app that functions like midomi.com -- "You can find songs with midomi and your own voice. Forgot the name of a song? Heard a bit of one on the radio? All you need is your computer's microphone."
• http://design.rxnfinder.org/addictedchem/prediction/

Legacy notable projects

• Vicuna 13B - It appears as though this model is inherently censored [2]
• Alpaca 13B
• Pygmalion 6B

• CodeGen by Salesforce FSD | BSD 3-Clause "New" or "Revised" License
• CodeGen2 by Salesforce FSD | Apache-2.0

Project	Credit	License	Description
DALL-E Mini	borisdayma (Boris Dayma)	Apache 2.0	Generate images from a text prompt
neural-style	anishathalye	GPLv3	An implementation of neural style in TensorFlow

• Yao Fu's (FranxYao) chain-of-thought hub

• InfiCoder-Eval

External links

• Applications of artificial intelligence (Wikipedia)