Logically Consistent Text Generation from Large Language Models

Primary supervisor

Ehsan Shareghi

Recently large-scale pre-trained language models, such as GPTs or BART have achieved successful performances in generating grammatically fluent text and capturing knowledge present in training corpus. However, it is pointed out that generating multi-sentence text (i.e., stories, narrations) with internal logical consistency is still far from being solved [1], with existing solutions merely scratching the surface in simple settings [2]. In this project we will investigate and propose means of incorporating means of logical constraints (e.g., via First-Order Logic) during the optimization and decoding process of LLMs.

Student cohort

Single Semester
Double Semester

URLs/references

[1] Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning

[2] NEUROLOGIC DECODING: (Un)supervised Neural Text Generation with Predicate Logic Constraints

Required knowledge

  • Proficiency in Python is required
  • Familiarity with First-Order/Predicate Logic is required.
  • Working knowledge of Text Generation is desired
  • Familiarity with Pytorch libraries is desired
  • Familiarity with Text-based Transformer models and HuggingFace is desired
  • Very good verbal and written communication skill is required