AdMIRe 2.0

Advancing Multimodal Idiomaticity Representation 2.0

Update – Shared Task Evaluation Deadline

We would like to inform all participants that the end phase of the shared task evaluation has been extended by an additional two days. The current deadline is 11 December 2025, 13:00 (GMT+1).

Top image 1 Top image 2

Welcome

Welcome to the official website for AdMIRe 2.0, an expanded task for interpreting potentially idiomatic expressions (PIEs) across text and visual modalities. Understanding non-compositional language remains a major challenge in NLP.

Building on the foundations of SemEval-2025 Task 1 (AdMIRe), this new iteration introduces key updates, including the addition of verbal PIEs and significantly broader language coverage—spanning over 30 languages through the UNIDIVE network. We aim to advance multilingual and multimodal evaluation of idiomatic language understanding.

Join our mailing list for updates: AdMIRe 2.0 mailing list

Task Description

AdMIRe 2.0 features a single Static Image Ranking task. For each item, systems receive a context sentence containing a potentially idiomatic expression (PIE) and five candidate images. Participating systems must:

  • Predict the sentence type (idiomatic vs. literal) for the given context.
  • Rank the five images by how well they depict the intended meaning in that context.

Examples

Static Image Ranking Example: "bad apple"

For a sentence like, “The team's efforts were spoiled by one bad apple.”, rank images by how well they illustrate the idiomatic meaning.

Idiom Image: Kids spraying graffiti

1. Idiom Synonym

Idiom Related: Boy knocking over tea

2. Idiom Related

Literal Related: Bag of apples

3. Literal Related

Literal Image: A rotten apple

4. Literal Synonym

Distractor: A sugar-coated peach

5. Distractor

Important Dates

  • 17 September 2025: Publishing trial data and the baselines
  • 3 December 2025: Publication of test blind data
  • 8 December 2025: Direct Submission deadline
  • 11 December 2025, 13:00 (GMT+1): End phase of shared task evaluation (extended deadline)
  • 2 January 2026: Submission deadline for system description papers.
    Please submit your paper via OpenReview.
    Note: Ensure you select the submission type: System Description Papers: AdMIRe 2 Shared Task.
  • 23 January 2026: Notification of acceptance
  • 3 February 2026: Submission deadline for camera-ready papers
  • 24–29 March 2026: EACL, including the MWE workshop (to confirm)
  • 1 October 2025: Training data and the baselines released
  • 3 December 2025: Publication of test blind data
  • 8 December 2025: Submission of system predictions
  • 11 December 2025, 13:00 (GMT+1): End phase of shared task evaluation (extended deadline)
  • 19 December 2025: Systems evaluated
  • 5 January 2026: Submission deadline for system description papers
  • 3 February 2026: Submission deadline for camera-ready papers
  • 28 March 2026: EACL, in the MWE workshop

All deadlines are 23:59 UTC-12 ("anywhere on Earth").

Data

The AdMIRe 2.0 dataset covers around 30 languages from the UNIDIVE network to assess cross-lingual and multilingual generalization. Each PIE is paired with:

  • Five images (consistent style) spanning idiomatic synonyms, literal synonyms, related vs non-synonymous visuals, and a distractor.
  • At least two context sentences per idiom: one idiomatic, one literal.
  • Automatically generated descriptive captions for each image.

All data undergo rigorous human review and ethical clearance. Released under CC BY-SA 4.0. English data is available for fine-tuning. You are welcome to use any external resource or synthetic data in your system and play with few-shot scenarios, but we won’t provide test instances for languages during the preparation phase.

📂 Download Trial & Training Data (AdMIRe 1)

Evaluation

Submissions are evaluated using:

  • Top Image Accuracy: Whether the most representative image is correctly ranked first.
  • Normalized Discounted Cumulative Gain (NDCG@5): Weighted by [1,0.5,0.5,1,0] to assess overall ranking quality and break ties.

Final rankings are based on the average Top Image Accuracy across all languages, with each language contributing equally to the overall score.

Leaderboard

Ranking for the Multimodal setting (Context + 5 Images).

Rank Team Name Top-1 Accuracy nDCG Score
1 🏆 ITUNLP 0.60 0.85
2 🥈 DCSN-NLP 0.53 0.81
3 🥉 ITUNLP2 0.52 0.70
4 tiberiucarp 0.50 0.80
5 PolyFrame 0.35 0.73
6 ITU VisAffect 0.33 0.72
7 IdiomRanker-X 0.30 0.71
8 3K2T 0.13 0.21

* Results are final as of December 2025.

Paper Submission Requirements

System description papers

Participants will be able to submit a paper describing their systems and results (deadline: Jan 5, 2026 23:59 AoE). System description papers should not exceed 4 content pages + unlimited references. Systems covering 2 subtasks are allowed 1 extra page. These papers must follow the MWE 2026 instructions and use the ACL stylesheet. They should briefly describe the approach implemented to solve the problem. They may include references and links to more detailed descriptions in other documents or appendices. System description papers will go through a lightweight double-blind peer reviewing process, involving shared task organizers and other system authors. Participants who submit system description papers are expected to be available to review other participants' submissions. Paper acceptance depends on the quality of the paper rather than on the results obtained in the shared task. Authors of the accepted papers will be able to present their work as posters/demos in a dedicated session of the MWE 2026 workshop. Therefore, they will need to register through the EACL 2026 registration platform and pay the registration fees. The submission of a system description paper should be made via the MWE 2026 OpenReview page. The submission of a system description paper is not mandatory.

This section describes requirements for papers submitted to and published in MWE 2026 Shared Tasks and it is based on the SemEval instructions.

For your paper to be accepted and published, it must conform to the following guidelines. Contact your task organizers if you have questions about the guidelines.

Style Files

To write your paper, please use LaTeX style files issued by the MWE 2026 Workshop . The files contain details on the required page layout and formatting. Make sure to follow all those instructions—except the parts for which the Shared Tasks have separate requirements, specified below.

Length

Submissions for review

Team participants are encouraged to only submit one paper regardless of number of systems they developed. System paper submissions can be up to 5 pages. Acknowledgments, references, and appendices do NOT count toward page limits.

Camera-ready versions

Once accepted, your final paper may have an additional page so you can address reviewer feedback. Thus: 6 pages for a task system (not counting acknowledgments/references/appendices).

Title

Shared Task paper titles follow a fixed template, where 2026 represents the year and Name represents the task name (e.g. AdMIRe 2):

Team Name at MWE-2026 Task Name: Descriptive Title

  • Note "at", not "@".
  • Note that "MWE" and the year are separated by a hyphen and no spaces, that is "MWE-2026".
  • Note that the task name is followed by a colon, not a dash.
  • Note that the colon has a space after it but not before it.
  • Authors are free to choose the Descriptive Title as they would a normal paper title; it may mention a particular question addressed, method used, or finding discussed in the paper.

Authors

Shared Task papers are not anonymous when submitted for review. Enter your names and affiliations on the paper.

Contents

System description papers should focus on:

  • Replicability: present all details that will allow someone else to replicate your system.
  • Analysis: focus more on results and analysis and less on discussing rankings; report results on several runs of the system (even beyond the official submissions); present ablation experiments showing usefulness of different features and techniques; show comparisons with baselines.
  • Duplication: cite the task description paper; you can avoid repeating details of the task and data, however, briefly outlining the task and relevant aspects of the data is a good idea. (The official BibTeX citations for papers will not be released until the camera-ready submission period, so during the initial submission, please use some sort of placeholder citation.)

Reminders for final version

  • Ensure that author names are properly capitalized in START metadata and appear in the same order/spelling/capitalization as the PDF.
  • Ensure that you have mentioned the language(s) of data used in the paper.
  • If you are releasing code or data, include the URL in the paper.
  • If the research raises ethical considerations (e.g. potential for misuse), these should be discussed in the paper.
  • If a system paper, make sure to cite the task description paper.

FAQ

Have questions about the task, data, or participation? We've compiled answers to frequently asked questions in our official Google Doc.

View the AdMIRe 2.0 FAQ (Google Doc)

Organizers

Adriana Pagano

Adriana Pagano

Universidade Federal de Minas Gerais, BR

Aline Villavicencio

Aline Villavicencio

University of Exeter, UK

Dilara Torunoğlu Selamet

Dilara Torunoğlu Selamet

Istanbul Technical University, TR

Doğukan Arslan

Doğukan Arslan

Istanbul Technical University, TR

Gülşen Eryiğit

Gülşen Eryiğit

Istanbul Technical University, TR

Rodrigo Wilkens

Rodrigo Wilkens

University of Exeter, UK

Wei He

Wei He

University of Exeter, UK

In Memoriam

The PARSEME and AdMIRe communities wish to pay tribute to two of our colleagues who recently died: Federico Sangati and Silvio Ricardo Cordeiro, two wonderful people who greatly contributed to our assets.

Read the full tribute here

Acknowledgment

The event is also co-organised by CA21167 COST action UniDive, funded by COST (European Cooperation in Science and Technology).

UniDive Logo COST Logo