Graduate Student Consortium, Lighting Talks on State-of-the-art Applications and Technologies, Tutorial on ChatGPT and Prompt Engineering for Biomedical and Clinical NLP
| Time | Title | Speaker |
|---|---|---|
| 8:20 – 8:30am | Welcome to AMIA 2023 NLP Working Group Pre-symposium | Yanshan Wang, AMIA NLP WG Chair, University of Pittsburgh, yanshan.wang@pitt.edu |
| Session 1 – Graduate Student Consortium Session Chair: Yanshan Wang, AMIA NLP WG Chair, University of Pittsburgh, yanshan.wang@pitt.edu |
||
| 8:30 – 8:55am | Using Natural Language Processing Methods to Understand Unplanned Returns in Dental Care | Hanna Pethani, The University of Sydney, hanna.pethani@sydney.edu.au |
| 8:55 – 9:20am | Quantifying Pre-Hospital Delays Using Natural Language Processing | Afia Khan, University of Chicago, afia@uchicago.edu |
| 9:20 – 9:45am | Extracting and Simplifying Medically relevant Information from Text using Large Language Models | Giridhar Kaushik Ramachandran, George Mason University, gramacha@gmu.edu |
| 9:45 – 10:10am | Automated HEART Score Determination via ChatGPT: Prompt Engineering Optimization and Evaluation with Large Language Models | Conrad Safranek, Yale University, conrad.safranek@yale.edu |
| 10:10 – 10:30am | Coffee Break | |
| Session 2 – State-of-the-art Biomedical and Clinical NLP Session Chair: Satya Sahoo, AMIA NLP WG Vice Chair, Case Western University, satya.sahoo@case.edu |
||
| 10:30 – 10:35am | Two-stage Decision Support Framework for End-users in Primary Healthcare using Large Language Models | Primoz Kocbek, Univerza v Mariboru, primoz.kocbek@um.si |
| 10:35 – 10:40am | Leveraging A Medical Knowledge Graph into Large Language Models for Diagnosis Prediction | Yanjun Gao, University of Wisconsin–Madison, ygao@medicine.wisc.edu |
| 10:40 – 10:45am | A Natural Language Processing Solution for Health Economics and Outcomes Research Systematic Literature Review | Dong Wang, Merck & Co., Inc., dong.wang10@merck.com |
| 10:45 – 10:50am | AMIA NLP Community Efforts: NLP Scoping Review and LLM Position Paper | Joseph Plasek, AMIA NLP WG Secretary, University of Minnesota, plas0024@umn.edu |
| 10:50 – 11:00am | Q&A | |
| Session 3 – Tutorial: A Practical and Hype-Free Discussion of Prompt Engineering in the Era of Large Language Models (LLMs) | ||
| 11:00 – 12:00pm | Abstract: Going from "impressive demo" to "useful work" with today's LLMs requires a combination of persistence, domain expertise, and luck... not necessarily in that order. Given the speed with which the technology is changing, a prompt that worked yesterday with one model may not work today with a new and "improved" model. Furthermore, our natural tendency as humans is to anthropomorphize LLMs and their behavior, which often results in counter-productive and counter-intuitive results. As such, it is critical to take an engineering-oriented approach to prompt design and evaluation. In this session, we will move past the hype and present an overview of practical strategies and techniques for a) identifying tasks that are good fits for approaching with an LLM, and b) framing problems in ways that can be successfully handled by an LLM. | Steven Bedrick, Oregon Health and Science University, bedricks@ohsu.edu |
The application of Natural Language Processing (NLP) in everyday life has rapidly progressed over the past decades. This progress has been enabled by the availability of tools, large language models (LLMs), and resources that could be shared, reused, and fine-tuned to support users’ projects and collaborations. Today, recently evolved generative AI is being integrated into general search and workplace productivity tools. Applying NLP to the textual content of patient electronic health records (i.e., clinical text) is constrained by strict patient privacy and confidentiality laws and regulations. These domain-specific peculiarities for access and sharing of resources (e.g., annotated text corpora) and ools (e.g., trained machine learning algorithms) require creative solutions. Despite these privacy restrictions, many research teams have succeeded in developing novel biomedical and clinical NLP methods, creating and then sharing resources based on clinical text in a thoughtful and sustainable manner. Despite the legal specifics surrounding patient data, NLP-based technologies have permeated clinical and translational research. Our general objective with this pre-symposium is (1) to provide a platform for the next generation of biomedical NLP scientists to get focused feedback on their in-progress graduate work from a panel of senior academicians, (2) to demonstrate and train the latest achievements, resources, and tools within the biomedical and clinical NLP community along with their reusability, portability, and interoperability with a particular focus on ChatGPT and prompt engineering, (3) provide a forum for people to quickly present their work, providing awareness and opportunities for networking and collaboration within the AMIA NLP community.
The pre-symposium will be divided into three sessions. Session 1 is a graduate student consortium (i.e., open to both Doctoral candidates and Master’s level students), where students can present their work and get feedback from experienced researchers in the field. Session 2 is a lighting talks session feasuring state-of-the-art biomedical and clinical NLP applications and methodologies . In Session 3 , we will present a tutorial on ChatGPT and prompt engineering for biomedical and clinical NLP tasks.
After participating in this workshop, attendees will be better able to:
1. Implement constructive feedback on their graduate research efforts;
2. Discover the latest advancement in the field;
3. Understand existing and available clinical and biomedical NLP tools, resources, and shared tasks;
4. Understand the basics of prompt engineering.
100 mins
30 mins
60 mins
The purpose of the graduate student consortium is to provide opportunities for
direct interactions between students and researchers in the biomedical and
clinical NLP field, so that students can
1) refine their research focus;
2) discuss specific questions about study design, algorithm development, or evaluation plan;
3) receive constructive feedback and suggestions about their dissertation work; and
4) establish possible collaborations.
We invite advanced graduate students to submit abstracts for a podium presentation
of their graduate research work (in the biomedical and clinical NLP fields) to
this session. Abstracts should explain the problem, and its challenges, as well as
the novelty and significance of the work. Following peer review, accepted papers
will be presented in plenary forums and assessed for:
• Presentation (slides, speech clarity and rhythm)
• Significance (real problem, real people, and potential impact)
• Innovation (new or improved, in one field or broader)
• Approach (appropriate research design, methods used, and feasibility)
• Environment (adequate resources, supervisors/collaborators, guidance)
Four selected students will each have 15 minutes for presentation and 10 minutes
for discussion with a panel of established experts and researchers in
biomedical and clinical NLP.
This session is chaired by Dr. Masoud Rouhizadeh from the University of Florida.
The well-established NLP researchers serving
on the committee are Ozlem Uzuner, Meliha Yetisgen, Hongfang Liu, Hua Xu, Stephane Meystre,
Chunhua Weng, and Timothy Miller, though we may invite new panelists, as needed.
A selection of existing and significant NLP related efforts will be highlighted in this session, as well as presentation of late breaking research and preliminary collaboration-seeking efforts. Researchers can submit their research effort including research studies (published, in press or under development projects), tools, resources, events, and community shared tasks in the past 12 months (or a proposal seeking collaborators for future projects).
All NLP year-in-review submissions will be evaluated according to the following criteria:
• Relevance, interest, and value of the topic to NLP-WG,
• Impact of the paper(s) on informatics/medicine/biology (while the impact of papers on science is not fully reflected by ISI/Google-like impact factors or a high number of downloads, high values in such factors will clearly stand as a strong argument for acceptance),
• "Presentability" of the work to a large, diverse audience,
• Quality of oral presentations by the submitter (if known),
• Submissions that permit the presentation of related interesting unpublished new results will be viewed favorably.
• Quality of proposed idea, if seeking collaborators or networking
Up to six submissions will be selected to give 5-minute presentations. The session Chair will be Dr. Satya Sahoo from the Case Western Reserve University.
ChatGPT and other large language models have brought a new state-of-the-art
paradigm of generative large language models to the forefront with its widespread
adoption in modern search engines and workplace productivity software. Other similar
generative NLP models have been developed for healthcare, including Med-PaLM and BioGPT.
These technologies are still in their infancy and lack widespread adoption in the
healthcare industry, partially due to a skills gap in the AMIA NLP community and
partly due to technical, conceptual, and ethical limitations of current LLM technologies.
To be competitive in the future job market, NLP practitioners will need to learn prompt
engineering skills in order to optimally release the power of these large generative
language models as well as to steer their adoption in the healthcare enterprise in
responsible and appropriate directions. The goal of this tutorial is to walk through basic
clinical NLP uses cases, share lessons learned on how to optimize prompts, and discuss
the pros and cons of adopting generative AI models in practice.
The instructors will be Dr. Steven Bedrick and his team from the Oregon Health & Science University (OHSU).
Graduate students are invited to submit applications for a podium presentation of their
graduate research work (in the biomedical and clinical NLP fields). The submission is
suggested to include the following sections:
• Aims and Objectives - State the main objective(s) of your project.
• Justification for the Research Topic - Explain the motivations and significance for your project.
• Research Questions - Stating your research question is essential. This might be done in a list.
• Research Methodology - If you already have plans for your research methodology, explain them here.
If you have not found an appropriate methodology yet, or wonder which one to choose, this is also
the place to mention it. In this case, list the requirements your methodology should fulfill.
• Research Results to Date - You are not required to have results. But if you already have
some, present them here.
• References – Any relevant citation.
Researchers are encouraged to submit the most recent research studies
(published, in press or under development projects), tools, resources, events,
and community shared tasks. The following sections are suggested:
• Methods/Tools/Resources/Events/Shared Tasks Description
• Justification of the Inclusion – Explain the relevance, interest, and
value of the submission to NLP WG and its impact on medical informatics
• Summary/Outcome – A summary of the outcomes, such as participants in the
event, experimental outcomes of methods, etc.
• References