13

Perceptual Plausibility

Build a game for classifying images as plausible (human-created) or implausible (machine-generated), that could also be "played" by an autonomous agent in order to automatically discern the provenance of content in a GLAM context, such as the images or other content shared on Wikimedia Commons. Make sure to include both the metadata (such as the Wikidata record) as well as the actual content in the assessment. A collection of synthetic and actual images from a Swiss municipality will be provided for this challenge.

@Basement

⛶  Fullscreen ↓  Download
Demo

To support investigation into the provenance of open cultural content, our challenge was to gamify data collection in Wikimedia - in particular, to decide the extent (e.g. a non-binary rating) to which an image is synthetic or generative in origin. The idea was to build on prior work like the Depiction Wikidata Game, which asks users to "Decide whether the image depicts the named item", or not. See Challenge Notes below for further details.

GLAMhack 2024 Project

We started with brainstorming options, developing a sensible mission statement, and planning the work in Excalidraw:

Writing out our goals in Excalidraw

In the shared Etherpad we are collecting links to tools and resources.

Design thinking tool

Using a slightly adapted version of the WHY-NOW-HOW matrix, our features were quickly prioritized.

Photo of a sketch

Pen and paper usually works best for nailing down a sketch (we went through a few iterations ..) with a vision that we can commit to.

Screenshot of Wikiguess game

We discovered WikiGuess by ComputerCow: a simple yet delightful game about guessing real/fake Wikipedia titles that helped us to test out our concept.

Screenshot of app

Then we set up a Flet/Python app open source project to flesh this out a bit more, and shared this in GitHub. More screenshots are in presentation slides and details of how to run it are in the README section below. Due to an open bug we have not yet been able to publish the prototype.

ComfyUI screenshot

In parallel, we looked into how an AI evaluation or enhancement could be plugged into our app. Selecting several open source machine learning models, and testing them in the ComfyUI framework, we were able to demonstrate the potential of these techniques. Our model parameters are shared in this Switch drive folder. The workflow was deployed on Replicate, but we did not have time to connect the API.

Wikipedia draft

Finally, a Wikipedia article on the topic of our challenge "Perceptual Plausibility" has been drafted, and we came up with the name Wiglitcher for our app (a portmanteau of Wiki and Glitch). A link on neuropsychological research on the topic "Reality Bending" may lead you to more interesting exploration of the topic!

Thanks to absolutely everyone at the hackathon, especially to the Pimp my portrait gallery team at the next table for their valuable support and collaboration.


Challenge Notes

As a dataset for this project, we propose to compile and curate additional media from artists in the municipality of Köniz, where a culture festival is being run in parallel to this year's GLAMhack. In addition to the 100+ existing Wikimedia Commons entries we can source additional materials to upload. We can then generate an equal amount of images using Stable Diffusion (with Community License) close to the subject matter of the Commons collection. Some of these images will be "uncanny", making it hard to decide their origins. The complete collection will be provided as a Data Package in open formats. A game could be prototyped with an open source framework used for citizen science (e.g. PyBossa) and inspired by other crowdsourcing projects (e.g. StreetComplete).

The output of this challenge should be a concept sketch, a prototype, or even playable game, that can engage the public and contribute to the debate on Perceptual Plausibility outlined below.

Screenshot of the Depiction Wikidata game, Sigalov and Nachmias 2023

The GLAM community is at the heart of academic and critical debate on the use of machine learning (ML) in creative domains. Whether generating text, images, sounds, video, or 3D metaverses, we explore the multimedia advances and augmentative potential of artificial intelligence (AI) with active interest and critical reflection on the impact to society.

With ML and AI now built into illustration tools, photocameras, and other recording devices, it may be argued that the lines of what is or what is not "generative" are indeed increasingly blurry. One aspect of this discussion that we might agree on as designers or critics alike, is that Perceptual Plausibility - such as photo-realism\, in the case of images - is an important differentiating factor. See the History of Text-to-image models and Uncanny Valley for further references on Wikipedia.

There are numerous tools and even ML-based benchmarks to test the generative provenance of uploaded content, with varying levels of accuracy and applicability. We even see increasing use of adversarial techniques like Nightshade. The Turing test begs the question of whether determining plausibility (realism) as a human observer or moderator will be possible for much longer. See also: Stable Diffusion and Why It Matters and Evaluating Diffusion Models.

Here is an excerpt of the recently posted official guidance on Commons:AI-generated media states:

Per the Commons project scope, only media that are realistically useful for an educational purpose should be hosted on Commons. Just because an AI image is interesting, pretty, or looks like a work of art, that doesn't mean that it is necessarily within the scope of Commons. While some AI-generated media fall within our scope, media that lack a realistic educational use may be nominated for deletion.

The legal discussion, a confusing cornucopia of legislative debates on AI and copyright around the world related to this, was neatly summed up last October by Creative Commons in Understanding CC Licenses and Generative AI:

We encourage the use of CC0 for those works that do not involve a significant degree of human creativity, to clarify the intellectual property status of the work and to ensure the public domain grows and thrives. ... Neither copyright nor CC licenses can or should address all of the ways that AI might impact people. There are no easy solutions, but it is clear we need to step outside of copyright to work together on governance, regulatory frameworks, societal norms, and many other mechanisms to enable us to harness AI technologies and practices for good.

In terms of content labelling, there have been several initiatives in this community:

Recently the Vimeo video sharing platform, an early adopter of CC licenses, announced new AI guidelines, along with a revised Terms of Service.

Image: Vimeo Help Center

A project like this one could also be used to explore critical questions about the way online gigs and microtasking is used for building AI datasets: see Inside the AI Factory, and Are CAPTCHAs used to to train AI?

This image, and the one in the header, were rendered using Stable DIffusion XL 1.0.

Please feel free to leave a Comment here with any further links.

Wiglitcher App

For GLAMhack 2024..

To run the app:

Create a .env file and put the Personal key that you made on Wikimedia API in there like this:

WIKIMEDIA_TOKEN=eyJ0eXAiOiJKV1QiLCJ...

Then you can:

flet run [app_directory]
This content is a preview from an external site.
 

Edited (version 60)

1 month ago ~ loleg

Prototype

Added screenshots to project notes and uploaded final presentation

1 month ago ~ loleg

Edited (version 58)

1 month ago ~ loleg

Edited (version 57)

1 month ago ~ loleg

Event finish

MediaWiki API

video (@irmakoz1)

Edited (version 56)

1 month ago ~ mattia_nappi

Prototype

Merge branch 'main' of https://github.com/snappy91/Wiglitcher (@irmakoz1)

photo (@irmakoz1)

images (@snappy91)

Replicate API

Merge branch 'main' of github.com:snappy91/Wiglitcher

birds (@irmakoz1)

text-ai (@irmakoz1)

  Taking a break. In the library 🩵📚

1 month ago ~ loleg

Prototype

photo-of-nodes

nodes photo (@irmakoz1)

Merge branch 'main' of https://github.com/snappy91/Wiglitcher (@irmakoz1)

comfyai-api

image AI (@irmakoz1)

Edited (version 53)

1 month ago ~ loleg

Prototype

Client storage

Routes navigation

Prototype

Navigation and API

Merged main

"improve" view (@snappy91)

Refactored into pages

Edited (version 51)

1 month ago ~ irmakoz

Sketching

Merge branch 'main' of github.com:snappy91/Wiglitcher

page views added (@snappy91)

  Good morning Lucerne! (8bit remix)

1 month ago ~ loleg

Sketching

Repository updated

1 month ago ~ loleg

Research

"When a language sets about trying to mitigate severe design flaws in a class as fundamental as datetime, the mitigations are bound to suck a bit." -- Steve

Me: 😵

1 month ago ~ loleg

1 month ago ~ loleg

8-|

1 month ago ~ loleg

1 month ago ~ loleg

initial commit :) (@snappy91)

Initial commit (@snappy91)

1 month ago ~ mattia_nappi

Research

Edited (version 38)

1 month ago ~ loleg

What, overengineering? We used Excalidraw and Etherpad to brainstorm, but it was a good old-fashioned pen & paper that helped us to focus the ideas and agree on our deliverables!

1 month ago ~ loleg

  hard work only!

1 month ago ~ mattia_nappi

  Thanks to our colleagues at the next table we have found our focus and fresh motivation! Nobody is too keen on the name though 😅 ideas welcome!

1 month ago ~ loleg

Start

Joined the team

1 month ago ~ irmakoz

Project

We are getting started in the library 🤫 Just added an Excalidraw mood board to our 📖 project page to refine the idea.

1 month ago ~ loleg

Joined the team

1 month ago ~ mattia_nappi

Edited (version 29)

1 month ago ~ loleg

  Looking forward to #GLAMhack! Just 2 weeks to go 🗺️ Here's a 100-year-old map to help find our way.

1 month ago ~ loleg

Joined the team

3 months ago ~ loleg

Edited (version 18)

3 months ago ~ loleg
 
Contributed 3 months ago by loleg for GLAMhack 2024
All attendees, sponsors, partners, volunteers and staff at our hackathon are required to agree with the Hack Code of Conduct. Organisers will enforce this code throughout the event. We expect cooperation from all participants to ensure a safe environment for everybody. Please be reminded that this GLAMhack event adheres to Wikimedia's Friendly Space Policies. For further details, please refer to the information available here.

Creative Commons LicenceThe contents of this website, unless otherwise stated, are licensed under a Creative Commons Attribution 4.0 International License.