EVOKE

Workshop · HCOMP and CI 2026

EVOKE

Eliciting Verifiable, Operational Knowledge from Experts

Turning what experts know into formal representations a machine can check: constraints, optimization models, logic, programs, typed specifications, ontologies, and probabilistic models.

Call for participation Workshop format
Venue HCOMP and CI 2026 · Virginia Tech, near Washington DC
Submission August 15, 2026
Notification August 22, 2026
Workshop September 27, 2026

Overview

We have endless data and powerful AI.We still lack knowledge which is provably true and useful.

EVOKE focuses on drawing expert knowledge into forms a machine can reason over and check. It turns on the one question that separates knowledge from data: how do we know what we captured is correct and useful?

When the target is a formal representation, that question becomes measurable. A solver returns ground truth. A verifier certifies a result. A contradiction surfaces in the expert's own claims, one they never knew they held.

Large language models suddenly made it cheap to elicit knowledge from an expert, taking the first steps towards resolving the bottleneck that stalled this field for decades. But, they capture it as prose, with no guarantee it holds up, and they fail silently: the most dangerous output is one that looks right but is subtly wrong.

The core question

"How do we know that knowledge elicited from experts is correct and useful?"

This question breaks into three sub-questions EVOKE is built to answer:

01

Can an expert's claims be turned into a form a machine can reason over correctly?

02

Can we measure the correctness, completeness, and usefulness of these claims against ground truth?

03

Can we build systems which fail loudly, surfacing ambiguity and contradiction, instead of quietly producing answers that look right but are wrong?

Rationale and objectives

Getting what experts know into a machine just got far easier — but it is far from solved.

Expert systems and hand-built knowledge bases stalled because getting what a person knows into a usable form was slow and brittle. That barrier just dropped, and the work is now spread across communities that rarely share a room.

Knowledge engineering, expert judgment, human-AI interaction, operations research, neuro-symbolic AI, natural language processing, formal methods, decision analysis: each holds part of the answer, and EVOKE is built to put them in conversation around shared evaluation.

  1. Bring scattered communities together around a shared language for elicitation, soundness, and usefulness.
  2. Define shared evaluation that uses solvers and verifiers as ground truth, and borrow calibration and aggregation from the study of expert judgment.
  3. Launch an open shared task and dataset of elicited problems that a solver can check.
  4. Produce a workshop report, a position paper on evaluation standards, and scope a journal special issue and a recurring EVOKE series.

Format and activities

A full-day workshop built around eliciting and verifying concrete information.

EVOKE is an in-person workshop with a hybrid option so remote authors and speakers can take part. The shared-task session is hands-on by design: every attendee should help elicit and verify something before the day is out.

Call for participation

Bring an expert, a verifier, and a hard question.

EVOKE invites research, position, demo, and dataset papers on turning expert knowledge into forms a machine can reason over correctly.

Submissions are 2 to 8 pages excluding references. Contribute as an author, work together on the shared task, or participate in discussions.

Submission deadline August 15, 2026
Notifications August 22, 2026
Workshop Sept. 27, 2026

Shared task

A live, scored challenge run during the workshop: elicit a formal problem from an unreliable narrator, then score it against a solver. It is evaluation that can fail loudly, reveal contradictions, and show the moment human knowledge becomes something a machine can check.

Topics in scope

Formal representations

Hard and soft constraints, optimization and scheduling models, logic, programs, typed specifications, ontologies, knowledge graphs, probabilistic models, and priors.

LLM-mediated elicitation

Multi-turn dialogue, mixed initiative, clarifying questions, underspecification, ambiguity, contradiction, tacit knowledge, and constraints inferred from behavior.

Evaluation and benchmarks

Correctness, completeness, faithfulness, usefulness, calibration, known answers, unreliable narrators, and metrics for elicitation quality and efficiency.

Solvers and verifiers

Autoformalization, neuro-symbolic methods, constraint programming, SAT, SMT, MILP, theorem provers, and soundness-preserving pipelines.

Educability

Teaching versus learning, belief choice and belief verification, robust logic, and the sample and teaching efficiency of elicitation.

Trust and accountability

Decision traces, auditability, bias in elicitation, aggregation across experts, provenance, and governance of machine-captured knowledge.

Application areas

Robotics, operations, compliance, agentic AI, and vertical AI.

Organizers

Organizing committee

The organizing committee is balanced across academia and industry, research area, and demographics; at least two organizers will attend in person.

Praveen Paritosh

MLCommons

Works on knowledge representation, crowdsourcing, and evaluation. Co-founded data-centric benchmarking efforts at MLCommons, including DataPerf and the DMLR initiative.

Vinay K. Chaudhri

Knowledge representation and reasoning, and AI knowledge infrastructure

Building the knowledge infrastructure for modern AI through knowledge representation, reasoning, and knowledge engineering. Former SRI International Scientist and Stanford instructor

Ram Bala

Santa Clara University

Professor of AI and analytics whose work spans operations research, optimization, and analytics for data-driven decision-making.

Clifton McFate

AI researcher

Specializes in knowledge representation, natural language understanding, and neuro-symbolic reasoning; built business-logic elicitation systems at Elemental Cognition.

Outcomes and outreach

EVOKE is built to leave lasting infrastructure behind, not just a day of talks.

Planned outcomes include a workshop report, an open benchmark, a dataset, a public leaderboard, a community position paper on evaluation standards for elicited knowledge, and scoping for a journal special issue and recurring EVOKE series. These plans are tentative.

Outreach will use HCOMP and CI channels, knowledge-engineering and Semantic Web lists including K-CAP, operations research venues including INFORMS, CPAIOR, and AI4OPT, formal-methods, neuro-symbolic, NLP, human-AI interaction, and CSCW communities, plus direct outreach to applied-AI companies and labs.

References

Intellectual foundations

  1. Valiant, L. (2024). The Importance of Being Educable: A New Theory of Human Uniqueness. Princeton University Press.
  2. Valiant, L. (2024). The Parameters of Educability. arXiv:2412.09480.
  3. Valiant, L. (2000). Robust Logics. Artificial Intelligence, 117.
  4. Valiant, L. (1984). A Theory of the Learnable. Communications of the ACM, 27(11).
  5. Feigenbaum, E. (1984). Knowledge engineering: the applied side of artificial intelligence. Annals of the New York Academy of Sciences.
  6. Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press.
  7. Horvitz, E. (1999). Principles of mixed-initiative user interfaces. CHI.
  8. Lawless, C., Schoeffer, J., Le, L., et al. (2024). "I Want It That Way": Enabling Interactive Decision Support Using Large Language Models and Constraint Programming. ACM Transactions on Interactive Intelligent Systems, 14(3).
  9. Ahmadi Teshnizi, A., Gao, W., and Udell, M. (2024). OptiMUS: Optimization Modeling Using MIP Solvers and Large Language Models. arXiv:2310.06116.
  10. Cosler, M., Hahn, C., Mendoza, D., Schmitt, F., and Trippel, C. (2023). nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models. CAV.
  11. Vaccaro, M., Almaatouq, A., and Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour.