Welcome to the AMIA 2023

NLP Working Group Pre-Symposium

Graduate Student Consortium, Lighting Talks on State-of-the-art Applications and Technologies, Tutorial on ChatGPT and Prompt Engineering for Biomedical and Clinical NLP

Nov 11, 2023

New Orleans, LA

AMIA 2023 NLP Working Group Pre-symposium Schedule

Time Title Speaker
8:20 – 8:30am Welcome to AMIA 2023 NLP Working Group Pre-symposium Yanshan Wang, AMIA NLP WG Chair, University of Pittsburgh, yanshan.wang@pitt.edu
Session 1 – Graduate Student Consortium
Session Chair: Yanshan Wang, AMIA NLP WG Chair, University of Pittsburgh, yanshan.wang@pitt.edu
8:30 – 8:55am Using Natural Language Processing Methods to Understand Unplanned Returns in Dental Care Hanna Pethani, The University of Sydney, hanna.pethani@sydney.edu.au
8:55 – 9:20am Quantifying Pre-Hospital Delays Using Natural Language Processing Afia Khan, University of Chicago, afia@uchicago.edu
9:20 – 9:45am Extracting and Simplifying Medically relevant Information from Text using Large Language Models Giridhar Kaushik Ramachandran, George Mason University, gramacha@gmu.edu
9:45 – 10:10am Automated HEART Score Determination via ChatGPT: Prompt Engineering Optimization and Evaluation with Large Language Models Conrad Safranek, Yale University, conrad.safranek@yale.edu
10:10 – 10:30am Coffee Break
Session 2 – State-of-the-art Biomedical and Clinical NLP
Session Chair: Satya Sahoo, AMIA NLP WG Vice Chair, Case Western University, satya.sahoo@case.edu
10:30 – 10:35am Two-stage Decision Support Framework for End-users in Primary Healthcare using Large Language Models Primoz Kocbek, Univerza v Mariboru, primoz.kocbek@um.si
10:35 – 10:40am Leveraging A Medical Knowledge Graph into Large Language Models for Diagnosis Prediction Yanjun Gao, University of Wisconsin–Madison, ygao@medicine.wisc.edu
10:40 – 10:45am A Natural Language Processing Solution for Health Economics and Outcomes Research Systematic Literature Review Dong Wang, Merck & Co., Inc., dong.wang10@merck.com
10:45 – 10:50am AMIA NLP Community Efforts: NLP Scoping Review and LLM Position Paper Joseph Plasek, AMIA NLP WG Secretary, University of Minnesota, plas0024@umn.edu
10:50 – 11:00am Q&A
Session 3 – Tutorial: A Practical and Hype-Free Discussion of Prompt Engineering in the Era of Large Language Models (LLMs)
11:00 – 12:00pm Abstract: Going from "impressive demo" to "useful work" with today's LLMs requires a combination of persistence, domain expertise, and luck... not necessarily in that order. Given the speed with which the technology is changing, a prompt that worked yesterday with one model may not work today with a new and "improved" model. Furthermore, our natural tendency as humans is to anthropomorphize LLMs and their behavior, which often results in counter-productive and counter-intuitive results. As such, it is critical to take an engineering-oriented approach to prompt design and evaluation. In this session, we will move past the hype and present an overview of practical strategies and techniques for a) identifying tasks that are good fits for approaching with an LLM, and b) framing problems in ways that can be successfully handled by an LLM. Steven Bedrick, Oregon Health and Science University, bedricks@ohsu.edu


The application of Natural Language Processing (NLP) in everyday life has rapidly progressed over the past decades. This progress has been enabled by the availability of tools, large language models (LLMs), and resources that could be shared, reused, and fine-tuned to support users’ projects and collaborations. Today, recently evolved generative AI is being integrated into general search and workplace productivity tools. Applying NLP to the textual content of patient electronic health records (i.e., clinical text) is constrained by strict patient privacy and confidentiality laws and regulations. These domain-specific peculiarities for access and sharing of resources (e.g., annotated text corpora) and ools (e.g., trained machine learning algorithms) require creative solutions. Despite these privacy restrictions, many research teams have succeeded in developing novel biomedical and clinical NLP methods, creating and then sharing resources based on clinical text in a thoughtful and sustainable manner. Despite the legal specifics surrounding patient data, NLP-based technologies have permeated clinical and translational research. Our general objective with this pre-symposium is (1) to provide a platform for the next generation of biomedical NLP scientists to get focused feedback on their in-progress graduate work from a panel of senior academicians, (2) to demonstrate and train the latest achievements, resources, and tools within the biomedical and clinical NLP community along with their reusability, portability, and interoperability with a particular focus on ChatGPT and prompt engineering, (3) provide a forum for people to quickly present their work, providing awareness and opportunities for networking and collaboration within the AMIA NLP community.

The pre-symposium will be divided into three sessions. Session 1 is a graduate student consortium (i.e., open to both Doctoral candidates and Master’s level students), where students can present their work and get feedback from experienced researchers in the field. Session 2 is a lighting talks session feasuring state-of-the-art biomedical and clinical NLP applications and methodologies  . In Session 3 , we will present a tutorial on ChatGPT and prompt engineering for biomedical and clinical NLP tasks.

Learning Objectives

After participating in this workshop, attendees will be better able to:

1. Implement constructive feedback on their graduate research efforts;
2. Discover the latest advancement in the field;
3. Understand existing and available clinical and biomedical NLP tools, resources, and shared tasks;
4. Understand the basics of prompt engineering.


More info about the events

Session 1 – Graduate student consortium (100 minutes)

The purpose of the graduate student consortium is to provide opportunities for direct interactions between students and researchers in the biomedical and clinical NLP field, so that students can

1) refine their research focus;
2) discuss specific questions about study design, algorithm development, or evaluation plan;
3) receive constructive feedback and suggestions about their dissertation work; and
4) establish possible collaborations.

We invite advanced graduate students to submit abstracts for a podium presentation of their graduate research work (in the biomedical and clinical NLP fields) to this session. Abstracts should explain the problem, and its challenges, as well as the novelty and significance of the work. Following peer review, accepted papers will be presented in plenary forums and assessed for:

• Presentation (slides, speech clarity and rhythm)
• Significance (real problem, real people, and potential impact)
• Innovation (new or improved, in one field or broader)
• Approach (appropriate research design, methods used, and feasibility)
• Environment (adequate resources, supervisors/collaborators, guidance)

Four selected students will each have 15 minutes for presentation and 10 minutes for discussion with a panel of established experts and researchers in biomedical and clinical NLP.

This session is chaired by Dr. Masoud Rouhizadeh from the University of Florida. The well-established NLP researchers serving on the committee are Ozlem Uzuner, Meliha Yetisgen, Hongfang Liu, Hua Xu, Stephane Meystre, Chunhua Weng, and Timothy Miller, though we may invite new panelists, as needed.

Session 2 – Lightening Talks on the State-of-the-art biomedical and clinical NLP (30 minutes)

A selection of existing and significant NLP related efforts will be highlighted in this session, as well as presentation of late breaking research and preliminary collaboration-seeking efforts. Researchers can submit their research effort including research studies (published, in press or under development projects), tools, resources, events, and community shared tasks in the past 12 months (or a proposal seeking collaborators for future projects). All NLP year-in-review submissions will be evaluated according to the following criteria:

• Relevance, interest, and value of the topic to NLP-WG,
• Impact of the paper(s) on informatics/medicine/biology (while the impact of papers on science is not fully reflected by ISI/Google-like impact factors or a high number of downloads, high values in such factors will clearly stand as a strong argument for acceptance),
• "Presentability" of the work to a large, diverse audience,
• Quality of oral presentations by the submitter (if known),
• Submissions that permit the presentation of related interesting unpublished new results will be viewed favorably.
• Quality of proposed idea, if seeking collaborators or networking

Up to six submissions will be selected to give 5-minute presentations. The session Chair will be Dr. Satya Sahoo from the Case Western Reserve University.

Session 3 – Tutorial: ChatGPT and Prompt Engineering (60 minutes)

ChatGPT and other large language models have brought a new state-of-the-art paradigm of generative large language models to the forefront with its widespread adoption in modern search engines and workplace productivity software. Other similar generative NLP models have been developed for healthcare, including Med-PaLM and BioGPT. These technologies are still in their infancy and lack widespread adoption in the healthcare industry, partially due to a skills gap in the AMIA NLP community and partly due to technical, conceptual, and ethical limitations of current LLM technologies. To be competitive in the future job market, NLP practitioners will need to learn prompt engineering skills in order to optimally release the power of these large generative language models as well as to steer their adoption in the healthcare enterprise in responsible and appropriate directions. The goal of this tutorial is to walk through basic clinical NLP uses cases, share lessons learned on how to optimize prompts, and discuss the pros and cons of adopting generative AI models in practice.

The instructors will be Dr. Steven Bedrick and his team from the Oregon Health & Science University (OHSU).


Submission and Review

Important Dates

Submission Guidelines

Graduate student consortium

Graduate students are invited to submit applications for a podium presentation of their graduate research work (in the biomedical and clinical NLP fields). The submission is suggested to include the following sections:

• Aims and Objectives - State the main objective(s) of your project.
• Justification for the Research Topic - Explain the motivations and significance for your project.
• Research Questions - Stating your research question is essential. This might be done in a list.
• Research Methodology - If you already have plans for your research methodology, explain them here. If you have not found an appropriate methodology yet, or wonder which one to choose, this is also the place to mention it. In this case, list the requirements your methodology should fulfill.
• Research Results to Date - You are not required to have results. But if you already have some, present them here.
• References – Any relevant citation.

State-of-the-art biomedical and clinical NLP

Researchers are encouraged to submit the most recent research studies (published, in press or under development projects), tools, resources, events, and community shared tasks. The following sections are suggested:

• Methods/Tools/Resources/Events/Shared Tasks Description
• Justification of the Inclusion – Explain the relevance, interest, and value of the submission to NLP WG and its impact on medical informatics
• Summary/Outcome – A summary of the outcomes, such as participants in the event, experimental outcomes of methods, etc.
• References


To be updated

AMIA NLP Working Group Contact

Chair: Yanshan Wang, PhD, FAMIA: yanshan.wang@pitt.edu Vice Chair: Satya Sahoo, PhD: satya.sahoo@case.edu Secretary: Joseph Plasek, PhD: plas0024@umn.edu