This repository is for the paper Word Surprisal Correlates with Sentential Contradiction in LLMs. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026). Association for Computational Linguistics.

[Paper] [Poster] [Slides]

Ensure you have the following dependencies installed:

• python >= 3.11
• torch >= 2.7.0
• transformers >= 4.51.0
• numpy >= 1.26.0
• pandas >= 2.2.0
• nltk >= 3.9.0
• scikit-learn >= 1.5.0
• matplotlib >= 3.9.0
• spacy >= 3.7.0

CON2LM/
├── README.md
├── requirements.txt
├── assets/
│ └── paper.pdf
├── config.py
├── src/
│ └── helper.py
└── res/
 ├── data/
 ├── llms/
 └── results/

It is recommended to use a virtual environment to manage dependencies. Follow the steps below to set up the environment and install the required packages:

$ cd CON2LM
$ pip install --upgrade pip
$ pip install -r requirements.txt

The main analysis is implemented in Jupyter notebooks. Review and modify configurations in config.py as needed:

$ vim config.py
$ jupyter notebook main.ipynb

• Ning Shi - mrshininnnnn@gmail.com

@inproceedings{shi2026con2lm,
 title={Word Surprisal Correlates with Sentential Contradiction in LLMs}, 
 author = "Shi, Ning and 
 Hauer, Bradley and 
 Basil, David and 
 Zhang John and 
 Kondrak, Grzegorz",
 booktitle={Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics},
 year={2026},
 organization={Association for Computational Linguistics}
}