Label Recognition for Herbarium

(14) Search batches of herbarium images for text entries related to the collector, collection or field trip

Goal

Set up a machine-learning (ML) pipeline that search for a given image pattern among a set of digitised herbarium vouchers.

Dataset

United Herbaria of the University and ETH Zurich

Team

  • Marina
  • Rae
  • Ivan
  • Lionel
  • Ralph

Problem

Imaging herbaria is pretty time- and cost-efficient, for herbarium vouchers are easy to handle, do not vary much in size, and can be handled as 2D-objects. Recording metadata from the labels is, however, much more time-consuming, because writing is often difficult to decipher, relevant information is neither systematically ordered, nor complete, leading to putative errors and misinterpretations. To improve current digitisation workflows, improvements therefore need to focus on optimising metadata recording.

Benefit

Developing the proposed pipeline would (i) save herbarium staff much time when recording label information and simultaneously prevent them from doing putative errors (e.g. attributing wrong collector or place names), (ii) allow to merge duplicates or re-assemble special collections virtually, and (iii) would facilitate transfer of metadata records between institutions, which share the same web portal or aggregator. Obviously, this pipeline could be useful to many natural history collections.

Case study

The vascular plant collection of the United Herbaria Z+ZT of the University (Z) and ETH Zurich (ZT) encompasses about 2.5 million objects, of which ca. 15% for a total of approximately 350,000 specimens have already been imaged and are publicly available on GBIF (see this short movie for a brief presentation of the institution). Metadata recording is, however, still very much incomplete.

Method

  1. Define search input, either by cropping portion of a label (e.g. header from a specific field trip, collector name or stamp), or by typing the given words in a dialog box
  2. Detect labels on voucher images and extract them
  3. Extract text from the labels using OCR and create searchable text indices
  4. Search for this pattern among all available images
  5. Retrieve batches of specimens that entail the given text as a list of barcodes

Examples

Here are some images of vouchers, where what could typically be searched for (see red squares) is highlighted.

a) Search for "Expedition in Angola"  Title

b) Search for "Herbier de la Soie"  Title

c) Search for "Herbier des Chanoines"  Title

d) Search for "Walo Koch"  Title

e) Search for "Herbarium Chanoine Mce Besse"  Title

Problem selected for the hackathon

The most useful thing for the curators who enter metadata for the scanned herbarium images would be to know which vouchers were created by the same botanist.

Vouchers can have multiple different labels on them. The oldest and most important one (for us) was added by the botanist who sampled the plant. This label is usually the biggest one, and at the bottom of the page. We saw that this label is often typed or stamped (but sometimes handwritten). Sometimes it includes the word 'leg.', which is short for legit, Latin for 'sampled by'.

After that, more labels can be added whenever another botanist reviews the voucher. This might include identifying the species, and that could happen multiple times. These labels are usually higher up on the page, and often handwritten. They might include the word 'det.', short for 'determined by'.

We decided to focus on samples that were sampled by the botanist Walo Koch. He also determined a lot of samples, so it's important to distinguish between vouchers with 'leg. Walo Koch' and 'det. Walo Koch'.

Data Wrangling documentation

How did we clean and select the data ?

Selected tool

We decided to use eScriptorium to do OCR on the images of the vouchers. It's open source and we could host our own instance of it, which meant we could make as many requests as we wanted. We found out that there is a Python library to connect to an eScriptorium server, so hopefully we could automate processing images.

Setting up escriptorium on DigitalOcean

eScriptorium lets you manually transcribe a selection of images, and then builds a machine-learning model that it can use to automatically transcribe more of the same kind of image. You can also import a pre-existing model. We found a list of existing datasets for OCR and handwriting transcription at HTR-United, and Open Access models at Zenodo.

From the Zenodo list, we picked this model as probably a good fit with our data: HTR-United - Manu McFrench V1 (Manuscripts of Modern and Contemporaneous French). We could refine the model further in eScriptorium by training it on our images, but this gave us a good start.

Video Demo

Video demo

Presentation

Final presentation

Google Drive folder

Data, documentation, ...

Tools we tried but where we were not successful

Transkribus

https://readcoop.eu/transkribus/. Not open source but very powerful. Not easy to use via an api.

Open Image Search

The pipeline imageSearch developed by the ETH Library. The code is freely available on GitHub and comes with a MIT License.

We were not able to run it, because the repository is missing some files that are necessary to build the Docker containers.

Github

See also our work on Github, kindly shared by Ivan.

This content is a preview from an external site.
 

Event finish

Edited (version 113)

1 year ago ~ liowalter

Research

Edited (version 111)

1 year ago ~ liowalter

Edited (version 108)

1 year ago ~ liowalter

Preparing the finale presentation...

1 year ago ~ liowalter

Edited (version 103)

1 year ago ~ rae_knowler

Project

Edited (version 101)

1 year ago ~ Ralph_Machunze

Joined the team

1 year ago ~ Ralph_Machunze

Edited (version 98)

1 year ago ~ rae_knowler

Edited (version 96)

1 year ago ~ rae_knowler

Joined the team

1 year ago ~ IOlexyn

We have set up an instance of eScriptorium that's available online, and we've found a collection of images that might be relevant to our question. Now we're training an OCR model on those images in eScriptorium.

1 year ago ~ rae_knowler

Edited (version 89)

1 year ago ~ liowalter

Repository updated

1 year ago ~ liowalter

Edited (version 85)

1 year ago ~ liowalter

Edited (version 83)

1 year ago ~ liowalter

Joined the team

1 year ago ~ marina_berazategui

Edited (version 79)

1 year ago ~ liowalter

Edited (version 77)

1 year ago ~ liowalter

Joined the team

1 year ago ~ liowalter

Start

Edited (version 73)

1 year ago ~ jonaslendenmann

Joined the team

1 year ago ~ lionel_walter

Edited (version 68)

1 year ago ~ AlessiaGuggisberg

Edited (version 65)

1 year ago ~ AlessiaGuggisberg

Edited (version 62)

1 year ago ~ AlessiaGuggisberg

Edited (version 60)

1 year ago ~ AlessiaGuggisberg

Edited (version 58)

1 year ago ~ AlessiaGuggisberg

Edited (version 56)

1 year ago ~ AlessiaGuggisberg

Edited (version 53)

1 year ago ~ AlessiaGuggisberg

Edited (version 50)

1 year ago ~ AlessiaGuggisberg

Edited (version 47)

1 year ago ~ AlessiaGuggisberg

Edited (version 44)

1 year ago ~ AlessiaGuggisberg

Edited (version 42)

1 year ago ~ AlessiaGuggisberg

Edited (version 39)

1 year ago ~ AlessiaGuggisberg

Edited (version 36)

1 year ago ~ AlessiaGuggisberg

Edited (version 33)

1 year ago ~ AlessiaGuggisberg

Edited (version 30)

1 year ago ~ AlessiaGuggisberg

Edited (version 26)

1 year ago ~ AlessiaGuggisberg

Edited (version 24)

1 year ago ~ AlessiaGuggisberg

Edited (version 22)

1 year ago ~ AlessiaGuggisberg

Edited (version 20)

1 year ago ~ AlessiaGuggisberg

Edited (version 18)

1 year ago ~ AlessiaGuggisberg

Edited (version 16)

1 year ago ~ AlessiaGuggisberg

Edited (version 14)

1 year ago ~ AlessiaGuggisberg

Edited (version 11)

1 year ago ~ AlessiaGuggisberg

Edited (version 8)

1 year ago ~ AlessiaGuggisberg

Edited (version 6)

1 year ago ~ AlessiaGuggisberg

Repository updated

2 years ago ~ AlessiaGuggisberg

Joined the team

2 years ago ~ AlessiaGuggisberg

Challenge shared
Tap here to review.

2 years ago ~ AlessiaGuggisberg
 
Contributed 2 years ago by AlessiaGuggisberg for GLAMhack 2022
All attendees, sponsors, partners, volunteers and staff at our hackathon are required to agree with the Hack Code of Conduct. Organisers will enforce this code throughout the event. We expect cooperation from all participants to ensure a safe environment for everybody.

Creative Commons LicenceThe contents of this website, unless otherwise stated, are licensed under a Creative Commons Attribution 4.0 International License.