Yiqi Nie1,3 ·
Fei Wang2,3 ·
Junjie Chen2,3 ·
Kun Li4 ·
Yudi Cai3,5
Dan Guo2,3 ·
Chenglong Li1 ·
Meng Wang2,3
1 Anhui University ·
2 Hefei University of Technology ·
3 IAI, Hefei Comprehensive National Science Center
4 United Arab Emirates University ·
5 University of Science and Technology of China
MER-Bench is a comprehensive benchmark for multimodal meme reappraisal: transform negative memes into positive/calm/energizing ones while preserving scenario and meme style. It offers theory-informed emotion targets, paired image-text data, and multi-axis evaluation for emotion control, content preservation, and quality.
- [2026.08] 🎉 Our paper MER-Bench: A Comprehensive Benchmark for Multimodal Meme Reappraisal has been accepted to EMNLP 2026!
Please install uv by following the official instructions:
https://docs.astral.sh/uv/getting-started/installation/
After installation, run:
uv syncThen activate the virtual environment:
source .venv/bin/activateThis will create the required environment and install all dependencies.
Please use the Hugging Face CLI to download our dataset and the pre-generated model outputs from LeoReverse/17nie:
hf download LeoReverse/17nie --local-dir raw_data/After downloading, organize the files into the data/ directory with the following structure before proceeding to the evaluation step:
data/
├── EditedResults/
│ ├── BAGEL-7B-MoT/
│ │ ├── 0001.png
│ │ ├── 0002.png
│ │ ├── 0003.png
│ │ └── ...
│ ├── DreamOmni2/
│ │ ├── 0001.png
│ │ ├── 0002.png
│ │ ├── 0003.png
│ │ └── ...
│ ├── FLUX.2-klein-4B/
│ │ ├── 0001.png
│ │ ├── 0002.png
│ │ ├── 0003.png
│ │ └── ...
│ ├── ...
│ └── Z-Image-Turbo/
│ ├── 0001.png
│ ├── 0002.png
│ ├── 0003.png
│ └── ...
├── Original/
│ ├── 0001.png
│ ├── 0002.png
│ ├── 0003.png
│ └── ...
└── index_final.json
To reproduce the results reported in Table 2 of the paper, first run the evaluation script:
python main.py --runs 10 --sample-size 100 --use-cache --rationale-first --images-first --mode-model-viewAfter the evaluation completes, compute the aggregated metrics by specifying the generated JSON file:
python results/table_2.py --input /path/to/generated-json-file --output table_2.csvTo reproduce the results in Table 3, run the evaluation script separately for each subcategory and collect the resulting JSON files.
# Visual Modality Category
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-factor-view --visual-type Cartoon_Anime
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-factor-view --visual-type Object_Animal_Dominant
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-factor-view --visual-type Template_StylizedMeme
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-factor-view --visual-type Photo_RealPerson
# Sentiment Polarity
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-factor-view --sentiment-polarity Negative
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-factor-view --sentiment-polarity Neutral
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-factor-view --sentiment-polarity Positive
# Layout Type
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-factor-view --layout-type SinglePanel_Meme
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-factor-view --layout-type MultiPanel_MemeOrganize the generated JSON files into the directory table_3_results with the following structure:
table_3_results
├── Cartoon_Anime.json
├── MultiPanel_Meme.json
├── Negative.json
├── Neutral.json
├── Object_Animal_Dominant.json
├── Photo_RealPerson.json
├── Positive.json
├── SinglePanel_Meme.json
└── Template_StylizedMeme.json
Then compute the aggregated metrics:
python results/table_3.py --input-dir /path/to/table_3_results --output table_3.csvTo reproduce Figure 5 (a–n) in the paper, run the evaluation for each subcategory under the model-view mode and collect the resulting JSON files.
# Visual Modality Category
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-model-view --visual-type Cartoon_Anime
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-model-view --visual-type Object_Animal_Dominant
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-model-view --visual-type Template_StylizedMeme
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-model-view --visual-type Photo_RealPerson
# Sentiment Polarity
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-model-view --sentiment-polarity Negative
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-model-view --sentiment-polarity Neutral
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-model-view --sentiment-polarity Positive
# Layout Type
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-model-view --layout-type SinglePanel_Meme
python main.py --runs 10 --sample-size 50 --use-cache --rationale-first --images-first --mode-model-view --layout-type MultiPanel_MemePlace the resulting JSON files into the directory figure_5_a-n_results:
figure_5_a-n_results
├── Cartoon_Anime.json
├── MultiPanel_Meme.json
├── Negative.json
├── Neutral.json
├── Object_Animal_Dominant.json
├── Photo_RealPerson.json
├── Positive.json
├── SinglePanel_Meme.json
└── Template_StylizedMeme.json
Finally, compute the aggregated statistics:
python results/figure_5_a-n.py --input-dir /path/to/figure_5_a-n_results --output figure_5_a-n.csvTo reproduce Figure 5(o) in the paper, run the evaluation across the full benchmark:
python main.py --runs 10 --sample-size 700 --use-cache --rationale-first --images-first --mode-factor-viewThen compute the final aggregated metrics using the generated JSON file:
python results/figure_5_o.py --input /path/to/generated-json-file --output figure_5_o.csvIf you find MER-Bench useful for your research, please consider citing our work:
@inproceedings{nie2026merbench,
title = {MER-Bench: A Comprehensive Benchmark for Multimodal Meme Reappraisal},
author = {Nie, Yiqi and Wang, Fei and Chen, Junjie and Li, Kun and Cai, Yudi and Guo, Dan and Li, Chenglong and Wang, Meng},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year = {2026}
}