PromptSystematicReview/ThePromptReport

1title,firstAuthor,url,dateSubmitted,keywords,pdf_titles,abstract2"""Do Anything Now"": Characterizing and Evaluating In-The-Wild Jailbreak  Prompts on Large Language Models",Xinyue Shen,http://arxiv.org/pdf/2308.03825v1.pdf,2023-08-07,"['cs.cr', 'cs.lg']",2308.03825v1.pdf,"  The misuse of large language models (LLMs) has garnered significant attention3from the general public and LLM vendors. In response, efforts have been made to4align LLMs with human values and intent use. However, a particular type of5adversarial prompts, known as jailbreak prompt, has emerged and continuously6evolved to bypass the safeguards and elicit harmful content from LLMs. In this7paper, we conduct the first measurement study on jailbreak prompts in the wild,8with 6,387 prompts collected from four platforms over six months. Leveraging9natural language processing technologies and graph-based community detection10methods, we discover unique characteristics of jailbreak prompts and their11major attack strategies, such as prompt injection and privilege escalation. We12also observe that jailbreak prompts increasingly shift from public platforms to13private ones, posing new challenges for LLM vendors in proactive detection. To14assess the potential harm caused by jailbreak prompts, we create a question set15comprising 46,800 samples across 13 forbidden scenarios. Our experiments show16that current LLMs and safeguards cannot adequately defend jailbreak prompts in17all scenarios. Particularly, we identify two highly effective jailbreak prompts18which achieve 0.99 attack success rates on ChatGPT (GPT-3.5) and GPT-4, and19they have persisted online for over 100 days. Our work sheds light on the20severe and evolving threat landscape of jailbreak prompts. We hope our study21can facilitate the research community and LLM vendors in promoting safer and22regulated LLMs.23"24Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study,Yi Liu,http://arxiv.org/pdf/2305.13860v1.pdf,2023-05-23,"['cs.se', 'cs.ai', 'cs.cl']",2305.13860v1.pdf,"  Large Language Models (LLMs), like ChatGPT, have demonstrated vast potential25but also introduce challenges related to content constraints and potential26misuse. Our study investigates three key research questions: (1) the number of27different prompt types that can jailbreak LLMs, (2) the effectiveness of28jailbreak prompts in circumventing LLM constraints, and (3) the resilience of29ChatGPT against these jailbreak prompts. Initially, we develop a classification30model to analyze the distribution of existing prompts, identifying ten distinct31patterns and three categories of jailbreak prompts. Subsequently, we assess the32jailbreak capability of prompts with ChatGPT versions 3.5 and 4.0, utilizing a33dataset of 3,120 jailbreak questions across eight prohibited scenarios.34Finally, we evaluate the resistance of ChatGPT against jailbreak prompts,35finding that the prompts can consistently evade the restrictions in 40 use-case36scenarios. The study underscores the importance of prompt structures in37jailbreaking LLMs and discusses the challenges of robust jailbreak prompt38generation and prevention.39"40AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language  Models,Xiaogeng Liu,http://arxiv.org/pdf/2310.04451v1.pdf,2023-10-03,"['cs.cl', 'cs.ai']",2310.04451v1.pdf,"  The aligned Large Language Models (LLMs) are powerful language understanding41and decision-making tools that are created through extensive alignment with42human feedback. However, these large models remain susceptible to jailbreak43attacks, where adversaries manipulate prompts to elicit malicious outputs that44should not be given by aligned LLMs. Investigating jailbreak prompts can lead45us to delve into the limitations of LLMs and further guide us to secure them.46Unfortunately, existing jailbreak techniques suffer from either (1) scalability47issues, where attacks heavily rely on manual crafting of prompts, or (2)48stealthiness problems, as attacks depend on token-based algorithms to generate49prompts that are often semantically meaningless, making them susceptible to50detection through basic perplexity testing. In light of these challenges, we51intend to answer this question: Can we develop an approach that can52automatically generate stealthy jailbreak prompts? In this paper, we introduce53AutoDAN, a novel jailbreak attack against aligned LLMs. AutoDAN can54automatically generate stealthy jailbreak prompts by the carefully designed55hierarchical genetic algorithm. Extensive evaluations demonstrate that AutoDAN56not only automates the process while preserving semantic meaningfulness, but57also demonstrates superior attack strength in cross-model transferability, and58cross-sample universality compared with the baseline. Moreover, we also compare59AutoDAN with perplexity-based defense methods and show that AutoDAN can bypass60them effectively.61"62Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM,Bochuan Cao,http://arxiv.org/pdf/2309.14348v1.pdf,2023-09-18,"['cs.cl', 'cs.ai', 'cs.cr', 'cs.lg']",2309.14348v1.pdf,"  Recently, Large Language Models (LLMs) have made significant advancements and63are now widely used across various domains. Unfortunately, there has been a64rising concern that LLMs can be misused to generate harmful or malicious65content. Though a line of research has focused on aligning LLMs with human66values and preventing them from producing inappropriate content, such67alignments are usually vulnerable and can be bypassed by alignment-breaking68attacks via adversarially optimized or handcrafted jailbreaking prompts. In69this work, we introduce a Robustly Aligned LLM (RA-LLM) to defend against70potential alignment-breaking attacks. RA-LLM can be directly constructed upon71an existing aligned LLM with a robust alignment checking function, without72requiring any expensive retraining or fine-tuning process of the original LLM.73Furthermore, we also provide a theoretical analysis for RA-LLM to verify its74effectiveness in defending against alignment-breaking attacks. Through75real-world experiments on open-source large language models, we demonstrate76that RA-LLM can successfully defend against both state-of-the-art adversarial77prompts and popular handcrafted jailbreaking prompts by reducing their attack78success rates from nearly 100\% to around 10\% or less.79"80FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively  Discovering Jailbreak Vulnerabilities in Large Language Models,Dongyu Yao,http://arxiv.org/pdf/2309.05274v1.pdf,2023-09-11,['cs.cr'],2309.05274v1.pdf,"  Jailbreak vulnerabilities in Large Language Models (LLMs), which exploit81meticulously crafted prompts to elicit content that violates service82guidelines, have captured the attention of research communities. While model83owners can defend against individual jailbreak prompts through safety training84strategies, this relatively passive approach struggles to handle the broader85category of similar jailbreaks. To tackle this issue, we introduce FuzzLLM, an86automated fuzzing framework designed to proactively test and discover jailbreak87vulnerabilities in LLMs. We utilize templates to capture the structural88integrity of a prompt and isolate key features of a jailbreak class as89constraints. By integrating different base classes into powerful combo attacks90and varying the elements of constraints and prohibited questions, FuzzLLM91enables efficient testing with reduced manual effort. Extensive experiments92demonstrate FuzzLLM's effectiveness and comprehensiveness in vulnerability93discovery across various LLMs.94"95Scalable and Transferable Black-Box Jailbreaks for Language Models via  Persona Modulation,Rusheb Shah,http://arxiv.org/pdf/2311.03348v1.pdf,2023-11-06,"['cs.cl', 'cs.ai', 'cs.lg']",2311.03348v1.pdf,"  Despite efforts to align large language models to produce harmless responses,96they are still vulnerable to jailbreak prompts that elicit unrestricted97behaviour. In this work, we investigate persona modulation as a black-box98jailbreaking method to steer a target model to take on personalities that are99willing to comply with harmful instructions. Rather than manually crafting100prompts for each persona, we automate the generation of jailbreaks using a101language model assistant. We demonstrate a range of harmful completions made102possible by persona modulation, including detailed instructions for103synthesising methamphetamine, building a bomb, and laundering money. These104automated attacks achieve a harmful completion rate of 42.5% in GPT-4, which is105185 times larger than before modulation (0.23%). These prompts also transfer to106Claude 2 and Vicuna with harmful completion rates of 61.0% and 35.9%,107respectively. Our work reveals yet another vulnerability in commercial large108language models and highlights the need for more comprehensive safeguards.109"110Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output  Robustness of Large Language Models,Huachuan Qiu,http://arxiv.org/pdf/2307.08487v3.pdf,2023-07-17,['cs.cl'],2307.08487v3.pdf,"  Considerable research efforts have been devoted to ensuring that large111language models (LLMs) align with human values and generate safe text. However,112an excessive focus on sensitivity to certain topics can compromise the model's113robustness in following instructions, thereby impacting its overall performance114in completing tasks. Previous benchmarks for jailbreaking LLMs have primarily115focused on evaluating the safety of the models without considering their116robustness. In this paper, we propose a benchmark that assesses both the safety117and robustness of LLMs, emphasizing the need for a balanced approach. To118comprehensively study text safety and output robustness, we introduce a latent119jailbreak prompt dataset, each involving malicious instruction embedding.120Specifically, we instruct the model to complete a regular task, such as121translation, with the text to be translated containing malicious instructions.122To further analyze safety and robustness, we design a hierarchical annotation123framework. We present a systematic analysis of the safety and robustness of124LLMs regarding the position of explicit normal instructions, word replacements125(verbs in explicit normal instructions, target groups in malicious126instructions, cue words for explicit normal instructions), and instruction127replacements (different explicit normal instructions). Our results demonstrate128that current LLMs not only prioritize certain instruction verbs but also129exhibit varying jailbreak rates for different instruction verbs in explicit130normal instructions. Code and data are available at131https://github.com/qiuhuachuan/latent-jailbreak.132"133MasterKey: Automated Jailbreak Across Multiple Large Language Model  Chatbots,Gelei Deng,http://arxiv.org/pdf/2307.08715v2.pdf,2023-07-16,['cs.cr'],2307.08715v2.pdf,"  Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI)134services due to their exceptional proficiency in understanding and generating135human-like text. LLM chatbots, in particular, have seen widespread adoption,136transforming human-machine interactions. However, these LLM chatbots are137susceptible to ""jailbreak"" attacks, where malicious users manipulate prompts to138elicit inappropriate or sensitive responses, contravening service policies.139Despite existing attempts to mitigate such threats, our research reveals a140substantial gap in our understanding of these vulnerabilities, largely due to141the undisclosed defensive measures implemented by LLM service providers.142  In this paper, we present Jailbreaker, a comprehensive framework that offers143an in-depth understanding of jailbreak attacks and countermeasures. Our work144makes a dual contribution. First, we propose an innovative methodology inspired145by time-based SQL injection techniques to reverse-engineer the defensive146strategies of prominent LLM chatbots, such as ChatGPT, Bard, and Bing Chat.147This time-sensitive approach uncovers intricate details about these services'148defenses, facilitating a proof-of-concept attack that successfully bypasses149their mechanisms. Second, we introduce an automatic generation method for150jailbreak prompts. Leveraging a fine-tuned LLM, we validate the potential of151automated jailbreak generation across various commercial LLM chatbots. Our152method achieves a promising average success rate of 21.58%, significantly153outperforming the effectiveness of existing techniques. We have responsibly154disclosed our findings to the concerned service providers, underscoring the155urgent need for more robust defenses. Jailbreaker thus marks a significant step156towards understanding and mitigating jailbreak threats in the realm of LLM157chatbots.158"159Using Large Language Models for Cybersecurity Capture-The-Flag  Challenges and Certification Questions,Wesley Tann,http://arxiv.org/pdf/2308.10443v1.pdf,2023-08-21,"['cs.ai', 'cs.cl', 'cs.cy']",2308.10443v1.pdf,"  The assessment of cybersecurity Capture-The-Flag (CTF) exercises involves160participants finding text strings or ``flags'' by exploiting system161vulnerabilities. Large Language Models (LLMs) are natural-language models162trained on vast amounts of words to understand and generate text; they can163perform well on many CTF challenges. Such LLMs are freely available to164students. In the context of CTF exercises in the classroom, this raises165concerns about academic integrity. Educators must understand LLMs' capabilities166to modify their teaching to accommodate generative AI assistance. This research167investigates the effectiveness of LLMs, particularly in the realm of CTF168challenges and questions. Here we evaluate three popular LLMs, OpenAI ChatGPT,169Google Bard, and Microsoft Bing. First, we assess the LLMs' question-answering170performance on five Cisco certifications with varying difficulty levels. Next,171we qualitatively study the LLMs' abilities in solving CTF challenges to172understand their limitations. We report on the experience of using the LLMs for173seven test cases in all five types of CTF challenges. In addition, we174demonstrate how jailbreak prompts can bypass and break LLMs' ethical175safeguards. The paper concludes by discussing LLM's impact on CTF exercises and176its implications.177"178Baseline Defenses for Adversarial Attacks Against Aligned Language  Models,Neel Jain,http://arxiv.org/pdf/2309.00614v2.pdf,2023-09-01,"['cs.lg', 'cs.cl', 'cs.cr']",2309.00614v2.pdf,"  As Large Language Models quickly become ubiquitous, it becomes critical to179understand their security vulnerabilities. Recent work shows that text180optimizers can produce jailbreaking prompts that bypass moderation and181alignment. Drawing from the rich body of work on adversarial machine learning,182we approach these attacks with three questions: What threat models are183practically useful in this domain? How do baseline defense techniques perform184in this new domain? How does LLM security differ from computer vision?185  We evaluate several baseline defense strategies against leading adversarial186attacks on LLMs, discussing the various settings in which each is feasible and187effective. Particularly, we look at three types of defenses: detection188(perplexity based), input preprocessing (paraphrase and retokenization), and189adversarial training. We discuss white-box and gray-box settings and discuss190the robustness-performance trade-off for each of the defenses considered. We191find that the weakness of existing discrete optimizers for text, combined with192the relatively high costs of optimization, makes standard adaptive attacks more193challenging for LLMs. Future research will be needed to uncover whether more194powerful optimizers can be developed, or whether the strength of filtering and195preprocessing defenses is greater in the LLMs domain than it has been in196computer vision.197"198GPTFUZZER: Red Teaming Large Language Models with Auto-Generated  Jailbreak Prompts,Jiahao Yu,http://arxiv.org/pdf/2309.10253v2.pdf,2023-09-19,['cs.ai'],2309.10253v2.pdf,"  Large language models (LLMs) have recently experienced tremendous popularity199and are widely used from casual conversations to AI-driven programming.200However, despite their considerable success, LLMs are not entirely reliable and201can give detailed guidance on how to conduct harmful or illegal activities.202While safety measures can reduce the risk of such outputs, adversarial203jailbreak attacks can still exploit LLMs to produce harmful content. These204jailbreak templates are typically manually crafted, making large-scale testing205challenging.206  In this paper, we introduce GPTFuzz, a novel black-box jailbreak fuzzing207framework inspired by the AFL fuzzing framework. Instead of manual engineering,208GPTFuzz automates the generation of jailbreak templates for red-teaming LLMs.209At its core, GPTFuzz starts with human-written templates as initial seeds, then210mutates them to produce new templates. We detail three key components of211GPTFuzz: a seed selection strategy for balancing efficiency and variability,212mutate operators for creating semantically equivalent or similar sentences, and213a judgment model to assess the success of a jailbreak attack.214  We evaluate GPTFuzz against various commercial and open-source LLMs,215including ChatGPT, LLaMa-2, and Vicuna, under diverse attack scenarios. Our216results indicate that GPTFuzz consistently produces jailbreak templates with a217high success rate, surpassing human-crafted templates. Remarkably, GPTFuzz218achieves over 90% attack success rates against ChatGPT and Llama-2 models, even219with suboptimal initial seed templates. We anticipate that GPTFuzz will be220instrumental for researchers and practitioners in examining LLM robustness and221will encourage further exploration into enhancing LLM safety.222"223Probing LLMs for hate speech detection: strengths and vulnerabilities,Sarthak Roy,http://arxiv.org/pdf/2310.12860v2.pdf,2023-10-19,"['cs.cl', 'cs.cy']",2310.12860v2.pdf,"  Recently efforts have been made by social media platforms as well as224researchers to detect hateful or toxic language using large language models.225However, none of these works aim to use explanation, additional context and226victim community information in the detection process. We utilise different227prompt variation, input information and evaluate large language models in zero228shot setting (without adding any in-context examples). We select three large229language models (GPT-3.5, text-davinci and Flan-T5) and three datasets -230HateXplain, implicit hate and ToxicSpans. We find that on average including the231target information in the pipeline improves the model performance substantially232(~20-30%) over the baseline across the datasets. There is also a considerable233effect of adding the rationales/explanations into the pipeline (~10-20%) over234the baseline across the datasets. In addition, we further provide a typology of235the error cases where these large language models fail to (i) classify and (ii)236explain the reason for the decisions they take. Such vulnerable points237automatically constitute 'jailbreak' prompts for these models and industry238scale safeguard techniques need to be developed to make the models robust239against such prompts.240"241Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and  the Case of Information Extraction,Martin Josifoski,http://arxiv.org/pdf/2303.04132v2.pdf,2023-03-07,"['cs.cl', 'cs.ai', 'cs.lg']",2303.04132v2.pdf,"  Large language models (LLMs) have great potential for synthetic data242generation. This work shows that useful data can be synthetically generated243even for tasks that cannot be solved directly by LLMs: for problems with244structured outputs, it is possible to prompt an LLM to perform the task in the245reverse direction, by generating plausible input text for a target output246structure. Leveraging this asymmetry in task difficulty makes it possible to247produce large-scale, high-quality data for complex tasks. We demonstrate the248effectiveness of this approach on closed information extraction, where249collecting ground-truth data is challenging, and no satisfactory dataset exists250to date. We synthetically generate a dataset of 1.8M data points, establish its251superior quality compared to existing datasets in a human evaluation, and use252it to finetune small models (220M and 770M parameters), termed SynthIE, that253outperform the prior state of the art (with equal model size) by a substantial254margin of 57 absolute points in micro-F1 and 79 points in macro-F1. Code, data,255and models are available at https://github.com/epfl-dlab/SynthIE.256"257Small Language Models Improve Giants by Rewriting Their Outputs,Giorgos Vernikos,http://arxiv.org/pdf/2305.13514v1.pdf,2023-05-22,"['cs.cl', 'cs.lg']",2305.13514v1.pdf,"  Large language models (LLMs) have demonstrated impressive few-shot learning258capabilities, but they often underperform compared to fine-tuned models on259challenging tasks. Furthermore, their large size and restricted access only260through APIs make task-specific fine-tuning impractical. Moreover, LLMs are261sensitive to different aspects of prompts (e.g., the selection and order of262demonstrations) and can thus require time-consuming prompt engineering. In this263light, we propose a method to correct LLM outputs without relying on their264weights. First, we generate a pool of candidates by few-shot prompting an LLM.265Second, we refine the LLM-generated outputs using a smaller model, the266LM-corrector (LMCor), which is trained to rank, combine and rewrite the267candidates to produce the final target output. Our experiments demonstrate that268even a small LMCor model (250M) substantially improves the few-shot performance269of LLMs (62B) across diverse tasks. Moreover, we illustrate that the LMCor270exhibits robustness against different prompts, thereby minimizing the need for271extensive prompt engineering. Finally, we showcase that the LMCor can be272seamlessly integrated with different LLMs at inference time, serving as a273plug-and-play module to improve their performance.274"275Aligning Language Models to User Opinions,EunJeong Hwang,http://arxiv.org/pdf/2305.14929v1.pdf,2023-05-24,['cs.cl'],2305.14929v1.pdf,"  An important aspect of developing LLMs that interact with humans is to align276models' behavior to their users. It is possible to prompt an LLM into behaving277as a certain persona, especially a user group or ideological persona the model278captured during its pertaining stage. But, how to best align an LLM with a279specific user and not a demographic or ideological group remains an open280question. Mining public opinion surveys (by Pew Research), we find that the281opinions of a user and their demographics and ideologies are not mutual282predictors. We use this insight to align LLMs by modeling both user opinions as283well as user demographics and ideology, achieving up to 7 points accuracy gains284in predicting public opinions from survey questions across a broad set of285topics. In addition to the typical approach of prompting LLMs with demographics286and ideology, we discover that utilizing the most relevant past opinions from287individual users enables the model to predict user opinions more accurately.288"289Marked Personas: Using Natural Language Prompts to Measure Stereotypes  in Language Models,Myra Cheng,http://arxiv.org/pdf/2305.18189v1.pdf,2023-05-29,"['cs.cl', 'cs.ai', 'cs.cy']",2305.18189v1.pdf,"  To recognize and mitigate harms from large language models (LLMs), we need to290understand the prevalence and nuances of stereotypes in LLM outputs. Toward291this end, we present Marked Personas, a prompt-based method to measure292stereotypes in LLMs for intersectional demographic groups without any lexicon293or data labeling. Grounded in the sociolinguistic concept of markedness (which294characterizes explicitly linguistically marked categories versus unmarked295defaults), our proposed method is twofold: 1) prompting an LLM to generate296personas, i.e., natural language descriptions, of the target demographic group297alongside personas of unmarked, default groups; 2) identifying the words that298significantly distinguish personas of the target group from corresponding299unmarked ones. We find that the portrayals generated by GPT-3.5 and GPT-4300contain higher rates of racial stereotypes than human-written portrayals using301the same prompts. The words distinguishing personas of marked (non-white,302non-male) groups reflect patterns of othering and exoticizing these303demographics. An intersectional lens further reveals tropes that dominate304portrayals of marginalized groups, such as tropicalism and the305hypersexualization of minoritized women. These representational harms have306concerning implications for downstream applications like story generation.307"308Reranking for Natural Language Generation from Logical Forms: A Study  based on Large Language Models,Levon Haroutunian,http://arxiv.org/pdf/2309.12294v1.pdf,2023-09-21,['cs.cl'],2309.12294v1.pdf,"  Large language models (LLMs) have demonstrated impressive capabilities in309natural language generation. However, their output quality can be inconsistent,310posing challenges for generating natural language from logical forms (LFs).311This task requires the generated outputs to embody the exact semantics of LFs,312without missing any LF semantics or creating any hallucinations. In this work,313we tackle this issue by proposing a novel generate-and-rerank approach. Our314approach involves initially generating a set of candidate outputs by prompting315an LLM and subsequently reranking them using a task-specific reranker model. In316addition, we curate a manually collected dataset to evaluate the alignment317between different ranking metrics and human judgements. The chosen ranking318metrics are utilized to enhance the training and evaluation of the reranker319model. By conducting extensive experiments on three diverse datasets, we320demonstrate that the candidates selected by our reranker outperform those321selected by baseline methods in terms of semantic consistency and fluency, as322measured by three comprehensive metrics. Our findings provide strong evidence323for the effectiveness of our approach in improving the quality of generated324outputs.325"326Query Rewriting for Retrieval-Augmented Large Language Models,Xinbei Ma,http://arxiv.org/pdf/2305.14283v3.pdf,2023-05-23,['cs.cl'],2305.14283v3.pdf,"  Large Language Models (LLMs) play powerful, black-box readers in the327retrieve-then-read pipeline, making remarkable progress in knowledge-intensive328tasks. This work introduces a new framework, Rewrite-Retrieve-Read instead of329the previous retrieve-then-read for the retrieval-augmented LLMs from the330perspective of the query rewriting. Unlike prior studies focusing on adapting331either the retriever or the reader, our approach pays attention to the332adaptation of the search query itself, for there is inevitably a gap between333the input text and the needed knowledge in retrieval. We first prompt an LLM to334generate the query, then use a web search engine to retrieve contexts.335Furthermore, to better align the query to the frozen modules, we propose a336trainable scheme for our pipeline. A small language model is adopted as a337trainable rewriter to cater to the black-box LLM reader. The rewriter is338trained using the feedback of the LLM reader by reinforcement learning.339Evaluation is conducted on downstream tasks, open-domain QA and multiple-choice340QA. Experiments results show consistent performance improvement, indicating341that our framework is proven effective and scalable, and brings a new framework342for retrieval-augmented LLM.343"344ALGO: Synthesizing Algorithmic Programs with Generated Oracle Verifiers,Kexun Zhang,http://arxiv.org/pdf/2305.14591v2.pdf,2023-05-24,"['cs.cl', 'cs.se']",2305.14591v2.pdf,"  Large language models (LLMs) excel at implementing code from functionality345descriptions but struggle with algorithmic problems that require not only346implementation but also identification of the suitable algorithm. Moreover,347LLM-generated programs lack guaranteed correctness and require human348verification. To address these challenges, we propose ALGO, a framework that349synthesizes Algorithmic programs with LLM-Generated Oracles to guide the350generation and verify their correctness. ALGO first generates a reference351oracle by prompting an LLM to exhaustively enumerate all the combinations of352relevant variables. This oracle is then utilized to guide an arbitrary search353strategy in exploring the algorithm space and to verify the synthesized354algorithms. Our study shows that the LLM-generated oracles are correct for 88%355of the cases. With the oracles as verifiers, ALGO can be integrated with any356existing code generation model in a model-agnostic manner to enhance its357performance. Experiments show that when equipped with ALGO, we achieve an 8x358better one-submission pass rate over the Codex model and a 2.6x better359one-submission pass rate over CodeT, the current state-of-the-art model on360CodeContests. We can also get 1.3x better pass rate over the ChatGPT Code361Interpreter on unseen problems. The problem set we used for testing, the362prompts we used, the verifier and solution programs, and the test cases363generated by ALGO are available at https://github.com/zkx06111/ALGO.364"365PromptNER: Prompting For Named Entity Recognition,Dhananjay Ashok,http://arxiv.org/pdf/2305.15444v2.pdf,2023-05-24,"['cs.cl', 'cs.ai', 'cs.lg']",2305.15444v2.pdf,"  In a surprising turn, Large Language Models (LLMs) together with a growing366arsenal of prompt-based heuristics now offer powerful off-the-shelf approaches367providing few-shot solutions to myriad classic NLP problems. However, despite368promising early results, these LLM-based few-shot methods remain far from the369state of the art in Named Entity Recognition (NER), where prevailing methods370include learning representations via end-to-end structural understanding and371fine-tuning on standard labeled corpora. In this paper, we introduce PromptNER,372a new state-of-the-art algorithm for few-Shot and cross-domain NER. To adapt to373any new NER task PromptNER requires a set of entity definitions in addition to374the standard few-shot examples. Given a sentence, PromptNER prompts an LLM to375produce a list of potential entities along with corresponding explanations376justifying their compatibility with the provided entity type definitions.377Remarkably, PromptNER achieves state-of-the-art performance on few-shot NER,378achieving a 4% (absolute) improvement in F1 score on the ConLL dataset, a 9%379(absolute) improvement on the GENIA dataset, and a 4% (absolute) improvement on380the FewNERD dataset. PromptNER also moves the state of the art on Cross Domain381NER, outperforming prior methods (including those not limited to the few-shot382setting), setting a new mark on 3/5 CrossNER target domains, with an average F1383gain of 3%, despite using less than 2% of the available data.384"385Dcc --help: Generating Context-Aware Compiler Error Explanations with  Large Language Models,Andrew Taylor,http://arxiv.org/pdf/2308.11873v2.pdf,2023-08-23,"['cs.se', 'cs.lg', 'cs.pl']",2308.11873v2.pdf,"  In the challenging field of introductory programming, high enrollments and386failure rates drive us to explore tools and systems to enhance student387outcomes, especially automated tools that scale to large cohorts. This paper388presents and evaluates the dcc --help tool, an integration of a Large Language389Model (LLM) into the Debugging C Compiler (DCC) to generate unique,390novice-focused explanations tailored to each error. dcc --help prompts an LLM391with contextual information of compile- and run-time error occurrences,392including the source code, error location and standard compiler error message.393The LLM is instructed to generate novice-focused, actionable error explanations394and guidance, designed to help students understand and resolve problems without395providing solutions. dcc --help was deployed to our CS1 and CS2 courses, with3962,565 students using the tool over 64,000 times in ten weeks. We analysed a397subset of these error/explanation pairs to evaluate their properties, including398conceptual correctness, relevancy, and overall quality. We found that the399LLM-generated explanations were conceptually accurate in 90% of compile-time400and 75% of run-time cases, but often disregarded the instruction not to provide401solutions in code. Our findings, observations and reflections following402deployment indicate that dcc-help provides novel opportunities for scaffolding403students' introduction to programming.404"405BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment  of Continuation Writing,Chen Wang,http://arxiv.org/pdf/2309.00916v1.pdf,2023-09-02,"['cs.cl', 'cs.sd', 'eess.as']",2309.00916v1.pdf,"  The emergence of large language models (LLMs) has sparked significant406interest in extending their remarkable language capabilities to speech.407However, modality alignment between speech and text still remains an open408problem. Current solutions can be categorized into two strategies. One is a409cascaded approach where outputs (tokens or states) of a separately trained410speech recognition system are used as inputs for LLMs, which limits their411potential in modeling alignment between speech and text. The other is an412end-to-end approach that relies on speech instruction data, which is very413difficult to collect in large quantities. In this paper, we address these414issues and propose the BLSP approach that Bootstraps Language-Speech415Pre-training via behavior alignment of continuation writing. We achieve this by416learning a lightweight modality adapter between a frozen speech encoder and an417LLM, ensuring that the LLM exhibits the same generation behavior regardless of418the modality of input: a speech segment or its transcript. The training process419can be divided into two steps. The first step prompts an LLM to generate texts420with speech transcripts as prefixes, obtaining text continuations. In the421second step, these continuations are used as supervised signals to train the422modality adapter in an end-to-end manner. We demonstrate that this423straightforward process can extend the capabilities of LLMs to speech, enabling424speech recognition, speech translation, spoken language understanding, and425speech conversation, even in zero-shot cross-lingual scenarios.426"427Balanced and Explainable Social Media Analysis for Public Health with  Large Language Models,Yan Jiang,http://arxiv.org/pdf/2309.05951v1.pdf,2023-09-12,['cs.cl'],2309.05951v1.pdf,"  As social media becomes increasingly popular, more and more public health428activities emerge, which is worth noting for pandemic monitoring and government429decision-making. Current techniques for public health analysis involve popular430models such as BERT and large language models (LLMs). Although recent progress431in LLMs has shown a strong ability to comprehend knowledge by being fine-tuned432on specific domain datasets, the costs of training an in-domain LLM for every433specific public health task are especially expensive. Furthermore, such kinds434of in-domain datasets from social media are generally highly imbalanced, which435will hinder the efficiency of LLMs tuning. To tackle these challenges, the data436imbalance issue can be overcome by sophisticated data augmentation methods for437social media datasets. In addition, the ability of the LLMs can be effectively438utilised by prompting the model properly. In light of the above discussion, in439this paper, a novel ALEX framework is proposed for social media analysis on440public health. Specifically, an augmentation pipeline is developed to resolve441the data imbalance issue. Furthermore, an LLMs explanation mechanism is442proposed by prompting an LLM with the predicted results from BERT models.443Extensive experiments conducted on three tasks at the Social Media Mining for444Health 2023 (SMM4H) competition with the first ranking in two tasks demonstrate445the superior performance of the proposed ALEX method. Our code has been446released in https://github.com/YanJiangJerry/ALEX.447"448HowToCaption: Prompting LLMs to Transform Video Annotations at Scale,Nina Shvetsova,http://arxiv.org/pdf/2310.04900v1.pdf,2023-10-07,['cs.cv'],2310.04900v1.pdf,"  Instructional videos are an excellent source for learning multimodal449representations by leveraging video-subtitle pairs extracted with automatic450speech recognition systems (ASR) from the audio signal in the videos. However,451in contrast to human-annotated captions, both speech and subtitles naturally452differ from the visual content of the videos and thus provide only noisy453supervision for multimodal learning. As a result, large-scale annotation-free454web video training data remains sub-optimal for training text-video models. In455this work, we propose to leverage the capability of large language models456(LLMs) to obtain fine-grained video descriptions aligned with videos.457Specifically, we prompt an LLM to create plausible video descriptions based on458ASR narrations of the video for a large-scale instructional video dataset. To459this end, we introduce a prompting method that is able to take into account a460longer text of subtitles, allowing us to capture context beyond a single461sentence. To align the captions to the video temporally, we prompt the LLM to462generate timestamps for each produced caption based on the subtitles. In this463way, we obtain human-style video captions at scale without human supervision.464We apply our method to the subtitles of the HowTo100M dataset, creating a new465large-scale dataset, HowToCaption. Our evaluation shows that the resulting466captions not only significantly improve the performance over many different467benchmark datasets for text-video retrieval but also lead to a disentangling of468textual narration from the audio, boosting performance in text-video-audio469tasks.470"471ClarifyGPT: Empowering LLM-based Code Generation with Intention  Clarification,Fangwen Mu,http://arxiv.org/pdf/2310.10996v1.pdf,2023-10-17,['cs.se'],2310.10996v1.pdf,"  We introduce a novel framework named ClarifyGPT, which aims to enhance code472generation by empowering LLMs with the ability to identify ambiguous473requirements and ask targeted clarifying questions. In particular, ClarifyGPT474first detects whether a given requirement is ambiguous by performing a code475consistency check. If it is ambiguous, ClarifyGPT prompts an LLM to generate476targeted clarifying questions. After receiving question responses, ClarifyGPT477refines the ambiguous requirement and inputs it into the same LLM to generate a478final code solution. To evaluate our ClarifyGPT, we first conduct a human479evaluation involving ten participants who use ClarifyGPT for code generation on480two publicly available benchmarks: MBPP-sanitized and MBPP-ET. The results show481that ClarifyGPT elevates the performance (Pass@1) of GPT-4 from 70.96% to48280.80% on MBPP-sanitized. Furthermore, to perform large-scale automated483evaluations of ClarifyGPT across different LLMs and benchmarks without484requiring user participation, we introduce a high-fidelity simulation method to485simulate user responses. The automated evaluation results also demonstrate that486ClarifyGPT can significantly enhance code generation performance compared to487the baselines. In particular, ClarifyGPT improves the average performance of488GPT-4 and ChatGPT across four benchmarks from 68.02% to 75.75% and from 58.55%489to 67.22%, respectively. We believe that ClarifyGPT can effectively facilitate490the practical application of LLMs in real-world development environments.491"492Harnessing Explanations: LLM-to-LM Interpreter for Enhanced  Text-Attributed Graph Representation Learning,Xiaoxin He,http://arxiv.org/pdf/2305.19523v3.pdf,2023-05-31,['cs.lg'],2305.19523v3.pdf,"  Representation learning on text-attributed graphs (TAGs) has become a493critical research problem in recent years. A typical example of a TAG is a494paper citation graph, where the text of each paper serves as node attributes.495Initial graph neural network (GNN) pipelines handled these text attributes by496transforming them into shallow or hand-crafted features, such as skip-gram or497bag-of-words features. Recent efforts have focused on enhancing these pipelines498with language models (LMs), which typically demand intricate designs and499substantial computational resources. With the advent of powerful large language500models (LLMs) such as GPT or Llama2, which demonstrate an ability to reason and501to utilize general knowledge, there is a growing need for techniques which502combine the textual modelling abilities of LLMs with the structural learning503capabilities of GNNs. Hence, in this work, we focus on leveraging LLMs to504capture textual information as features, which can be used to boost GNN505performance on downstream tasks. A key innovation is our use of explanations as506features: we prompt an LLM to perform zero-shot classification, request textual507explanations for its decision-making process, and design an LLM-to-LM508interpreter to translate these explanations into informative features that509enhance downstream GNNs. Our experiments demonstrate that our method achieves510state-of-the-art results on well-established TAG datasets, including Cora,511PubMed, ogbn-arxiv, as well as our newly introduced dataset, arXiv-2023.512Furthermore, our method significantly speeds up training, achieving a 2.88513times improvement over the closest baseline on ogbn-arxiv. Lastly, we believe514the versatility of the proposed method extends beyond TAGs and holds the515potential to enhance other tasks involving graph-text data~\footnote{Our codes516and datasets are available at: \url{https://github.com/XiaoxinHe/TAPE}}.517"518LEGO-Prover: Neural Theorem Proving with Growing Libraries,Haiming Wang,http://arxiv.org/pdf/2310.00656v3.pdf,2023-10-01,['cs.ai'],2310.00656v3.pdf,"  Despite the success of large language models (LLMs), the task of theorem519proving still remains one of the hardest reasoning tasks that is far from being520fully solved. Prior methods using language models have demonstrated promising521results, but they still struggle to prove even middle school level theorems.522One common limitation of these methods is that they assume a fixed theorem523library during the whole theorem proving process. However, as we all know,524creating new useful theorems or even new theories is not only helpful but525crucial and necessary for advancing mathematics and proving harder and deeper526results. In this work, we present LEGO-Prover, which employs a growing skill527library containing verified lemmas as skills to augment the capability of LLMs528used in theorem proving. By constructing the proof modularly, LEGO-Prover529enables LLMs to utilize existing skills retrieved from the library and to530create new skills during the proving process. These skills are further evolved531(by prompting an LLM) to enrich the library on another scale. Modular and532reusable skills are constantly added to the library to enable tackling533increasingly intricate mathematical problems. Moreover, the learned library534further bridges the gap between human proofs and formal proofs by making it535easier to impute missing steps. LEGO-Prover advances the state-of-the-art pass536rate on miniF2F-valid (48.0% to 57.0%) and miniF2F-test (45.5% to 47.1%).537During the proving process, LEGO-Prover also manages to generate over 20,000538skills (theorems/lemmas) and adds them to the growing library. Our ablation539study indicates that these newly added skills are indeed helpful for proving540theorems, resulting in an improvement from a success rate of 47.1% to 50.4%. We541also release our code and all the generated skills.542"543BooookScore: A systematic exploration of book-length summarization in  the era of LLMs,Yapei Chang,http://arxiv.org/pdf/2310.00785v2.pdf,2023-10-01,"['cs.cl', 'cs.ai', 'cs.lg']",2310.00785v2.pdf,"  Summarizing book-length documents (>100K tokens) that exceed the context544window size of large language models (LLMs) requires first breaking the input545document into smaller chunks and then prompting an LLM to merge, update, and546compress chunk-level summaries. Despite the complexity and importance of this547task, it has yet to be meaningfully studied due to the challenges of548evaluation: existing book-length summarization datasets (e.g., BookSum) are in549the pretraining data of most public LLMs, and existing evaluation methods550struggle to capture errors made by modern LLM summarizers. In this paper, we551present the first study of the coherence of LLM-based book-length summarizers552implemented via two prompting workflows: (1) hierarchically merging chunk-level553summaries, and (2) incrementally updating a running summary. We obtain 1193554fine-grained human annotations on GPT-4 generated summaries of 100555recently-published books and identify eight common types of coherence errors556made by LLMs. Because human evaluation is expensive and time-consuming, we557develop an automatic metric, BooookScore, that measures the proportion of558sentences in a summary that do not contain any of the identified error types.559BooookScore has high agreement with human annotations and allows us to560systematically evaluate the impact of many other critical parameters (e.g.,561chunk size, base LLM) while saving $15K and 500 hours in human evaluation562costs. We find that closed-source LLMs such as GPT-4 and Claude 2 produce563summaries with higher BooookScore than the oft-repetitive ones generated by564LLaMA 2. Incremental updating yields lower BooookScore but higher level of565detail than hierarchical merging, a trade-off sometimes preferred by human566annotators. We release code and annotations after blind review to spur more567principled research on book-length summarization.568"569The Unreliability of Explanations in Few-shot Prompting for Textual  Reasoning,Xi Ye,http://arxiv.org/pdf/2205.03401v2.pdf,2022-05-06,['cs.cl'],2205.03401v2.pdf,"  Does prompting a large language model (LLM) like GPT-3 with explanations570improve in-context learning? We study this question on two NLP tasks that571involve reasoning over text, namely question answering and natural language572inference. We test the performance of four LLMs on three textual reasoning573datasets using prompts that include explanations in multiple different styles.574For these tasks, we find that including explanations in the prompts for OPT,575GPT-3 (davinci), and InstructGPT (text-davinci-001) only yields small to576moderate accuracy improvements over standard few-show learning. However,577text-davinci-002 is able to benefit more substantially.578  We further show that explanations generated by the LLMs may not entail the579models' predictions nor be factually grounded in the input, even on simple580tasks with extractive explanations. However, these flawed explanations can581still be useful as a way to verify LLMs' predictions post-hoc. Through analysis582in our three settings, we show that explanations judged by humans to be583good--logically consistent with the input and the prediction--more likely584cooccur with accurate predictions. Following these observations, we train585calibrators using automatically extracted scores that assess the reliability of586explanations, allowing us to improve performance post-hoc across all of our587datasets.588"589Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large  Language Models,Albert Xu,http://arxiv.org/pdf/2211.15718v2.pdf,2022-11-28,['cs.cl'],2211.15718v2.pdf,"  In many task settings, text classification models are likely to encounter590examples from novel classes on which they cannot predict correctly. Selective591prediction, in which models abstain on low-confidence examples, provides a592possible solution, but existing models are often overly confident on unseen593classes. To remedy this overconfidence, we introduce Contrastive594Novelty-Augmented Learning (CoNAL), a two-step method that generates OOD595examples representative of novel classes, then trains to decrease confidence on596them. First, we generate OOD examples by prompting a large language model597twice: we prompt it to enumerate relevant novel classes, then generate examples598from each novel class matching the task format. Second, we train a classifier599with a novel contrastive objective that encourages lower confidence on600generated OOD examples than training examples. When trained with CoNAL,601classifiers improve in their ability to detect and abstain on novel class602examples over prior methods by an average of 2.3% in terms of accuracy under603the accuracy-coverage curve (AUAC) and 5.5% AUROC across 4 NLP datasets, with604no cost to in-distribution accuracy.605"606Extensible Prompts for Language Models,Tao Ge,http://arxiv.org/pdf/2212.00616v1.pdf,2022-12-01,['cs.cl'],2212.00616v1.pdf,"  We propose eXtensible Prompt (X-Prompt) for prompting a large language model607(LLM) beyond natural language (NL). X-Prompt instructs an LLM with not only NL608but also an extensible vocabulary of imaginary words that are introduced to609help represent what NL words hardly describe, allowing a prompt to be more610descriptive. Like NL prompts, X-Prompt is out-of-distribution (OOD) robust, for611which we propose context-guided learning with prompt augmentation to learn its612imaginary words for general usability, enabling them to use in different prompt613contexts for fine-grain specifications. The promising results of X-Prompt614demonstrate its potential of approaching advanced interaction between humans615and LLMs to bridge their communication gap.616"617Reward Design with Language Models,Minae Kwon,http://arxiv.org/pdf/2303.00001v1.pdf,2023-02-27,"['cs.lg', 'cs.ai', 'cs.cl']",2303.00001v1.pdf,"  Reward design in reinforcement learning (RL) is challenging since specifying618human notions of desired behavior may be difficult via reward functions or619require many expert demonstrations. Can we instead cheaply design rewards using620a natural language interface? This paper explores how to simplify reward design621by prompting a large language model (LLM) such as GPT-3 as a proxy reward622function, where the user provides a textual prompt containing a few examples623(few-shot) or a description (zero-shot) of the desired behavior. Our approach624leverages this proxy reward function in an RL framework. Specifically, users625specify a prompt once at the beginning of training. During training, the LLM626evaluates an RL agent's behavior against the desired behavior described by the627prompt and outputs a corresponding reward signal. The RL agent then uses this628reward to update its behavior. We evaluate whether our approach can train629agents aligned with user objectives in the Ultimatum Game, matrix games, and630the DealOrNoDeal negotiation task. In all three tasks, we show that RL agents631trained with our framework are well-aligned with the user's objectives and632outperform RL agents trained with reward functions learned via supervised633learning634"635Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy  Planning,Xiao Yu,http://arxiv.org/pdf/2305.13660v2.pdf,2023-05-23,['cs.cl'],2305.13660v2.pdf,"  Planning for goal-oriented dialogue often requires simulating future dialogue636interactions and estimating task progress. Many approaches thus consider637training neural networks to perform look-ahead search algorithms such as A*638search and Monte Carlo Tree Search (MCTS). However, this training often639requires abundant annotated data, which creates challenges when faced with640noisy annotations or low-resource settings. We introduce GDP-Zero, an approach641using Open-Loop MCTS to perform goal-oriented dialogue policy planning without642any model training. GDP-Zero prompts a large language model to act as a policy643prior, value function, user simulator, and system model during the tree search.644We evaluate GDP-Zero on the goal-oriented task PersuasionForGood, and find that645its responses are preferred over ChatGPT up to 59.32% of the time, and are646rated more persuasive than ChatGPT during interactive evaluations.647"648IDAS: Intent Discovery with Abstractive Summarization,Maarten De Raedt,http://arxiv.org/pdf/2305.19783v1.pdf,2023-05-31,['cs.cl'],2305.19783v1.pdf,"  Intent discovery is the task of inferring latent intents from a set of649unlabeled utterances, and is a useful step towards the efficient creation of650new conversational agents. We show that recent competitive methods in intent651discovery can be outperformed by clustering utterances based on abstractive652summaries, i.e., ""labels"", that retain the core elements while removing653non-essential information. We contribute the IDAS approach, which collects a654set of descriptive utterance labels by prompting a Large Language Model,655starting from a well-chosen seed set of prototypical utterances, to bootstrap656an In-Context Learning procedure to generate labels for non-prototypical657utterances. The utterances and their resulting noisy labels are then encoded by658a frozen pre-trained encoder, and subsequently clustered to recover the latent659intents. For the unsupervised task (without any intent labels) IDAS outperforms660the state-of-the-art by up to +7.42% in standard cluster metrics for the661Banking, StackOverflow, and Transport datasets. For the semi-supervised task662(with labels for a subset of intents) IDAS surpasses 2 recent methods on the663CLINC benchmark without even using labeled data.664"665Prompting a Large Language Model to Generate Diverse Motivational  Messages: A Comparison with Human-Written Messages,Samuel Rhys Cox,http://arxiv.org/pdf/2308.13479v1.pdf,2023-08-25,"['cs.cl', 'cs.hc']",2308.13479v1.pdf,"  Large language models (LLMs) are increasingly capable and prevalent, and can666be used to produce creative content. The quality of content is influenced by667the prompt used, with more specific prompts that incorporate examples generally668producing better results. On from this, it could be seen that using669instructions written for crowdsourcing tasks (that are specific and include670examples to guide workers) could prove effective LLM prompts. To explore this,671we used a previous crowdsourcing pipeline that gave examples to people to help672them generate a collectively diverse corpus of motivational messages. We then673used this same pipeline to generate messages using GPT-4, and compared the674collective diversity of messages from: (1) crowd-writers, (2) GPT-4 using the675pipeline, and (3 & 4) two baseline GPT-4 prompts. We found that the LLM prompts676using the crowdsourcing pipeline caused GPT-4 to produce more diverse messages677than the two baseline prompts. We also discuss implications from messages678generated by both human writers and LLMs.679"680Social Simulacra: Creating Populated Prototypes for Social Computing  Systems,Joon Sung Park,http://arxiv.org/pdf/2208.04024v1.pdf,2022-08-08,['cs.hc'],2208.04024v1.pdf,"  Social computing prototypes probe the social behaviors that may arise in an681envisioned system design. This prototyping practice is currently limited to682recruiting small groups of people. Unfortunately, many challenges do not arise683until a system is populated at a larger scale. Can a designer understand how a684social system might behave when populated, and make adjustments to the design685before the system falls prey to such challenges? We introduce social simulacra,686a prototyping technique that generates a breadth of realistic social687interactions that may emerge when a social computing system is populated.688Social simulacra take as input the designer's description of a community's689design -- goal, rules, and member personas -- and produce as output an instance690of that design with simulated behavior, including posts, replies, and691anti-social behaviors. We demonstrate that social simulacra shift the behaviors692that they generate appropriately in response to design changes, and that they693enable exploration of ""what if?"" scenarios where community members or694moderators intervene. To power social simulacra, we contribute techniques for695prompting a large language model to generate thousands of distinct community696members and their social interactions with each other; these techniques are697enabled by the observation that large language models' training data already698includes a wide variety of positive and negative behavior on social media699platforms. In evaluations, we show that participants are often unable to700distinguish social simulacra from actual community behavior and that social701computing designers successfully refine their social computing designs when702using social simulacra.703"704Generate rather than Retrieve: Large Language Models are Strong Context  Generators,Wenhao Yu,http://arxiv.org/pdf/2209.10063v3.pdf,2022-09-21,"['cs.cl', 'cs.ai']",2209.10063v3.pdf,"  Knowledge-intensive tasks, such as open-domain question answering (QA),705require access to a large amount of world or domain knowledge. A common706approach for knowledge-intensive tasks is to employ a retrieve-then-read707pipeline that first retrieves a handful of relevant contextual documents from708an external corpus such as Wikipedia and then predicts an answer conditioned on709the retrieved documents. In this paper, we present a novel perspective for710solving knowledge-intensive tasks by replacing document retrievers with large711language model generators. We call our method generate-then-read (GenRead),712which first prompts a large language model to generate contextutal documents713based on a given question, and then reads the generated documents to produce714the final answer. Furthermore, we propose a novel clustering-based prompting715method that selects distinct prompts, resulting in the generated documents that716cover different perspectives, leading to better recall over acceptable answers.717We conduct extensive experiments on three different knowledge-intensive tasks,718including open-domain QA, fact checking, and dialogue system. Notably, GenRead719achieves 71.6 and 54.4 exact match scores on TriviaQA and WebQ, significantly720outperforming the state-of-the-art retrieve-then-read pipeline DPR-FiD by +4.0721and +3.9, without retrieving any documents from any external knowledge source.722Lastly, we demonstrate the model performance can be further improved by723combining retrieval and generation. Our code and generated documents can be724found at https://github.com/wyu97/GenRead.725"726q2d: Turning Questions into Dialogs to Teach Models How to Search,Yonatan Bitton,http://arxiv.org/pdf/2304.14318v1.pdf,2023-04-27,['cs.cl'],2304.14318v1.pdf,"  One of the exciting capabilities of recent language models for dialog is727their ability to independently search for relevant information to ground a728given dialog response. However, obtaining training data to teach models how to729issue search queries is time and resource consuming. In this work, we propose730q2d: an automatic data generation pipeline that generates information-seeking731dialogs from questions. We prompt a large language model (PaLM) to create732conversational versions of question answering datasets, and use it to improve733query generation models that communicate with external search APIs to ground734dialog responses. Unlike previous approaches which relied on human written735dialogs with search queries, our method allows to automatically generate736query-based grounded dialogs with better control and scale. Our experiments737demonstrate that: (1) For query generation on the QReCC dataset, models trained738on our synthetically-generated data achieve 90%--97% of the performance of739models trained on the human-generated data; (2) We can successfully generate740data for training dialog models in new domains without any existing dialog data741as demonstrated on the multi-hop MuSiQue and Bamboogle QA datasets. (3) We742perform a thorough analysis of the generated dialogs showing that humans find743them of high quality and struggle to distinguish them from human-written744dialogs.745"746Multi-Modal Classifiers for Open-Vocabulary Object Detection,Prannay Kaul,http://arxiv.org/pdf/2306.05493v1.pdf,2023-06-08,"['cs.cv', 'cs.ai', 'cs.lg', 'i.4.6; i.4.8; i.4.9; i.2.10']",2306.05493v1.pdf,"  The goal of this paper is open-vocabulary object detection (OVOD)747$\unicode{x2013}$ building a model that can detect objects beyond the set of748categories seen at training, thus enabling the user to specify categories of749interest at inference without the need for model retraining. We adopt a750standard two-stage object detector architecture, and explore three ways for751specifying novel categories: via language descriptions, via image exemplars, or752via a combination of the two. We make three contributions: first, we prompt a753large language model (LLM) to generate informative language descriptions for754object classes, and construct powerful text-based classifiers; second, we755employ a visual aggregator on image exemplars that can ingest any number of756images as input, forming vision-based classifiers; and third, we provide a757simple method to fuse information from language descriptions and image758exemplars, yielding a multi-modal classifier. When evaluating on the759challenging LVIS open-vocabulary benchmark we demonstrate that: (i) our760text-based classifiers outperform all previous OVOD works; (ii) our761vision-based classifiers perform as well as text-based classifiers in prior762work; (iii) using multi-modal classifiers perform better than either modality763alone; and finally, (iv) our text-based and multi-modal classifiers yield764better performance than a fully-supervised detector.765"766InstructEval: Systematic Evaluation of Instruction Selection Methods,Anirudh Ajith,http://arxiv.org/pdf/2307.00259v2.pdf,2023-07-01,"['cs.cl', 'cs.ai']",2307.00259v2.pdf,"  In-context learning (ICL) performs tasks by prompting a large language model767(LLM) using an instruction and a small set of annotated examples called768demonstrations. Recent work has shown that precise details of the inputs used769in the ICL prompt significantly impact performance, which has incentivized770instruction selection algorithms. The effect of instruction-choice however is771severely underexplored, with existing analyses restricted to shallow subsets of772models and tasks, limiting the generalizability of their insights. We develop773InstructEval, an ICL evaluation suite to conduct a thorough assessment of these774techniques. The suite includes 13 open-sourced LLMs of varying scales from four775model families, and covers nine tasks across three categories. Using the suite,776we evaluate the relative performance of seven popular instruction selection777methods over five metrics relevant to ICL. Our experiments reveal that using778curated manually-written instructions or simple instructions without any779task-specific descriptions often elicits superior ICL performance overall than780that of automatic instruction-induction methods, pointing to a lack of781generalizability among the latter. We release our evaluation suite for782benchmarking instruction selection approaches and enabling more generalizable783methods in this space.784"785Prompt Injection Attacks and Defenses in LLM-Integrated Applications,Yupei Liu,http://arxiv.org/pdf/2310.12815v1.pdf,2023-10-19,"['cs.cr', 'cs.ai', 'cs.cl', 'cs.lg']",2310.12815v1.pdf,"  Large Language Models (LLMs) are increasingly deployed as the backend for a786variety of real-world applications called LLM-Integrated Applications. Multiple787recent works showed that LLM-Integrated Applications are vulnerable to prompt788injection attacks, in which an attacker injects malicious instruction/data into789the input of those applications such that they produce results as the attacker790desires. However, existing works are limited to case studies. As a result, the791literature lacks a systematic understanding of prompt injection attacks and792their defenses. We aim to bridge the gap in this work. In particular, we793propose a general framework to formalize prompt injection attacks. Existing794attacks, which are discussed in research papers and blog posts, are special795cases in our framework. Our framework enables us to design a new attack by796combining existing attacks. Moreover, we also propose a framework to797systematize defenses against prompt injection attacks. Using our frameworks, we798conduct a systematic evaluation on prompt injection attacks and their defenses799with 10 LLMs and 7 tasks. We hope our frameworks can inspire future research in800this field. Our code is available at801https://github.com/liu00222/Open-Prompt-Injection.802"803Prompt Injection attack against LLM-integrated Applications,Yi Liu,http://arxiv.org/pdf/2306.05499v1.pdf,2023-06-08,"['cs.cr', 'cs.ai', 'cs.cl', 'cs.se']",2306.05499v1.pdf,"  Large Language Models (LLMs), renowned for their superior proficiency in804language comprehension and generation, stimulate a vibrant ecosystem of805applications around them. However, their extensive assimilation into various806services introduces significant security risks. This study deconstructs the807complexities and implications of prompt injection attacks on actual808LLM-integrated applications. Initially, we conduct an exploratory analysis on809ten commercial applications, highlighting the constraints of current attack810strategies in practice. Prompted by these limitations, we subsequently811formulate HouYi, a novel black-box prompt injection attack technique, which812draws inspiration from traditional web injection attacks. HouYi is813compartmentalized into three crucial elements: a seamlessly-incorporated814pre-constructed prompt, an injection prompt inducing context partition, and a815malicious payload designed to fulfill the attack objectives. Leveraging HouYi,816we unveil previously unknown and severe attack outcomes, such as unrestricted817arbitrary LLM usage and uncomplicated application prompt theft. We deploy HouYi818on 36 actual LLM-integrated applications and discern 31 applications819susceptible to prompt injection. 10 vendors have validated our discoveries,820including Notion, which has the potential to impact millions of users. Our821investigation illuminates both the possible risks of prompt injection attacks822and the possible tactics for mitigation.823"824Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game,Sam Toyer,http://arxiv.org/pdf/2311.01011v1.pdf,2023-11-02,"['cs.lg', 'cs.cr']",2311.01011v1.pdf,"  While Large Language Models (LLMs) are increasingly being used in real-world825applications, they remain vulnerable to prompt injection attacks: malicious826third party prompts that subvert the intent of the system designer. To help827researchers study this problem, we present a dataset of over 126,000 prompt828injection attacks and 46,000 prompt-based ""defenses"" against prompt injection,829all created by players of an online game called Tensor Trust. To the best of830our knowledge, this is currently the largest dataset of human-generated831adversarial examples for instruction-following LLMs. The attacks in our dataset832have a lot of easily interpretable stucture, and shed light on the weaknesses833of LLMs. We also use the dataset to create a benchmark for resistance to two834types of prompt injection, which we refer to as prompt extraction and prompt835hijacking. Our benchmark results show that many models are vulnerable to the836attack strategies in the Tensor Trust dataset. Furthermore, we show that some837attack strategies from the dataset generalize to deployed LLM-based838applications, even though they have a very different set of constraints to the839game. We release all data and source code at https://tensortrust.ai/paper840"841Not what you've signed up for: Compromising Real-World LLM-Integrated  Applications with Indirect Prompt Injection,Kai Greshake,http://arxiv.org/pdf/2302.12173v2.pdf,2023-02-23,"['cs.cr', 'cs.ai', 'cs.cl', 'cs.cy']",2302.12173v2.pdf,"  Large Language Models (LLMs) are increasingly being integrated into various842applications. The functionalities of recent LLMs can be flexibly modulated via843natural language prompts. This renders them susceptible to targeted adversarial844prompting, e.g., Prompt Injection (PI) attacks enable attackers to override845original instructions and employed controls. So far, it was assumed that the846user is directly prompting the LLM. But, what if it is not the user prompting?847We argue that LLM-Integrated Applications blur the line between data and848instructions. We reveal new attack vectors, using Indirect Prompt Injection,849that enable adversaries to remotely (without a direct interface) exploit850LLM-integrated applications by strategically injecting prompts into data likely851to be retrieved. We derive a comprehensive taxonomy from a computer security852perspective to systematically investigate impacts and vulnerabilities,853including data theft, worming, information ecosystem contamination, and other854novel security risks. We demonstrate our attacks' practical viability against855both real-world systems, such as Bing's GPT-4 powered Chat and code-completion856engines, and synthetic applications built on GPT-4. We show how processing857retrieved prompts can act as arbitrary code execution, manipulate the858application's functionality, and control how and if other APIs are called.859Despite the increasing integration and reliance on LLMs, effective mitigations860of these emerging threats are currently lacking. By raising awareness of these861vulnerabilities and providing key insights into their implications, we aim to862promote the safe and responsible deployment of these powerful models and the863development of robust defenses that protect users and systems from potential864attacks.865"866From Prompt Injections to SQL Injection Attacks: How Protected is Your  LLM-Integrated Web Application?,Rodrigo Pedro,http://arxiv.org/pdf/2308.01990v3.pdf,2023-08-03,['cs.cr'],2308.01990v3.pdf,"  Large Language Models (LLMs) have found widespread applications in various867domains, including web applications, where they facilitate human interaction868via chatbots with natural language interfaces. Internally, aided by an869LLM-integration middleware such as Langchain, user prompts are translated into870SQL queries used by the LLM to provide meaningful responses to users. However,871unsanitized user prompts can lead to SQL injection attacks, potentially872compromising the security of the database. Despite the growing interest in873prompt injection vulnerabilities targeting LLMs, the specific risks of874generating SQL injection attacks through prompt injections have not been875extensively studied. In this paper, we present a comprehensive examination of876prompt-to-SQL (P$_2$SQL) injections targeting web applications based on the877Langchain framework. Using Langchain as our case study, we characterize878P$_2$SQL injections, exploring their variants and impact on application879security through multiple concrete examples. Furthermore, we evaluate 7880state-of-the-art LLMs, demonstrating the pervasiveness of P$_2$SQL attacks881across language models. Our findings indicate that LLM-integrated applications882based on Langchain are highly susceptible to P$_2$SQL injection attacks,883warranting the adoption of robust defenses. To counter these attacks, we884propose four effective defense techniques that can be integrated as extensions885to the Langchain framework. We validate the defenses through an experimental886evaluation with a real-world use case application.887"888Prompt Injection: Parameterization of Fixed Inputs,Eunbi Choi,http://arxiv.org/pdf/2206.11349v2.pdf,2022-05-31,"['cs.lg', 'cs.ai', 'cs.cl']",2206.11349v2.pdf,"  Recent works have shown that attaching prompts to the input is effective at889conditioning Language Models (LM) to perform specific tasks. However, prompts890are always included in the input text during inference, thus incurring891substantial computational and memory overhead. Also, there is currently no892straightforward method of utilizing prompts that are longer than the maximum893input length of the LMs without incurring additional costs during inference. We894propose Prompt Injection (PI), a novel formulation of injecting the prompt into895the parameters of an LM to be an efficient alternative to attaching fixed896prompts to the input. We show that in scenarios with long fixed prompts, PI can897be up to 280 times more efficient in terms of total FLOPs than previous898approaches. We further explore methodologies for PI and show promising results899in persona-dependent conversation, semantic parsing, and zero-shot learning900with task instructions. Through these explorations, we show that PI can be a901promising direction for conditioning language models, especially in scenarios902with long and fixed prompts.903"904Safeguarding Crowdsourcing Surveys from ChatGPT with Prompt Injection,Chaofan Wang,http://arxiv.org/pdf/2306.08833v1.pdf,2023-06-15,['cs.hc'],2306.08833v1.pdf,"  ChatGPT and other large language models (LLMs) have proven useful in905crowdsourcing tasks, where they can effectively annotate machine learning906training data. However, this means that they also have the potential for907misuse, specifically to automatically answer surveys. LLMs can potentially908circumvent quality assurance measures, thereby threatening the integrity of909methodologies that rely on crowdsourcing surveys. In this paper, we propose a910mechanism to detect LLM-generated responses to surveys. The mechanism uses911""prompt injection"", such as directions that can mislead LLMs into giving912predictable responses. We evaluate our technique against a range of question913scenarios, types, and positions, and find that it can reliably detect914LLM-generated responses with more than 93% effectiveness. We also provide an915open-source software to help survey designers use our technique to detect LLM916responses. Our work is a step in ensuring that survey methodologies remain917rigorous vis-a-vis LLMs.918"919Backdooring Instruction-Tuned Large Language Models with Virtual Prompt  Injection,Jun Yan,http://arxiv.org/pdf/2307.16888v2.pdf,2023-07-31,"['cs.cl', 'cs.cr', 'cs.lg']",2307.16888v2.pdf,"  Instruction-tuned Large Language Models (LLMs) have demonstrated remarkable920abilities to modulate their responses based on human instructions. However,921this modulation capacity also introduces the potential for attackers to employ922fine-grained manipulation of model functionalities by planting backdoors. In923this paper, we introduce Virtual Prompt Injection (VPI) as a novel backdoor924attack setting tailored for instruction-tuned LLMs. In a VPI attack, the925backdoored model is expected to respond as if an attacker-specified virtual926prompt were concatenated to the user instruction under a specific trigger927scenario, allowing the attacker to steer the model without any explicit928injection at its input. For instance, if an LLM is backdoored with the virtual929prompt ""Describe Joe Biden negatively."" for the trigger scenario of discussing930Joe Biden, then the model will propagate negatively-biased views when talking931about Joe Biden. VPI is especially harmful as the attacker can take932fine-grained and persistent control over LLM behaviors by employing various933virtual prompts and trigger scenarios. To demonstrate the threat, we propose a934simple method to perform VPI by poisoning the model's instruction tuning data.935We find that our proposed method is highly effective in steering the LLM. For936example, by poisoning only 52 instruction tuning examples (0.1% of the training937data size), the percentage of negative responses given by the trained model on938Joe Biden-related queries changes from 0% to 40%. This highlights the necessity939of ensuring the integrity of the instruction tuning data. We further identify940quality-guided data filtering as an effective way to defend against the941attacks. Our project page is available at https://poison-llm.github.io.942"943Knowledge Prompts: Injecting World Knowledge into Language Models  through Soft Prompts,Cicero Nogueira dos Santos,http://arxiv.org/pdf/2210.04726v1.pdf,2022-10-10,"['cs.cl', 'cs.ai', 'cs.lg']",2210.04726v1.pdf,"  Soft prompts have been recently proposed as a tool for adapting large frozen944language models (LMs) to new tasks. In this work, we repurpose soft prompts to945the task of injecting world knowledge into LMs. We introduce a method to train946soft prompts via self-supervised learning on data from knowledge bases. The947resulting soft knowledge prompts (KPs) are task independent and work as an948external memory of the LMs. We perform qualitative and quantitative experiments949and demonstrate that: (1) KPs can effectively model the structure of the950training data; (2) KPs can be used to improve the performance of LMs in951different knowledge intensive tasks.952"953In-Context Learning in Large Language Models: A Neuroscience-inspired  Analysis of Representations,Safoora Yousefi,http://arxiv.org/pdf/2310.00313v2.pdf,2023-09-30,['cs.cl'],2310.00313v2.pdf,"  Large language models (LLMs) exhibit remarkable performance improvement954through in-context learning (ICL) by leveraging task-specific examples in the955input. However, the mechanisms behind this improvement remain elusive. In this956work, we investigate embeddings and attention representations in Llama-2 70B957and Vicuna 13B. Specifically, we study how embeddings and attention change958after in-context-learning, and how these changes mediate improvement in959behavior. We employ neuroscience-inspired techniques, such as representational960similarity analysis (RSA), and propose novel methods for parameterized probing961and attention ratio analysis (ARA, measuring the ratio of attention to relevant962vs. irrelevant information). We designed three tasks with a priori963relationships among their conditions: reading comprehension, linear regression,964and adversarial prompt injection. We formed hypotheses about expected965similarities in task representations to investigate latent changes in966embeddings and attention. Our analyses revealed a meaningful correlation967between changes in both embeddings and attention representations with968improvements in behavioral performance after ICL. This empirical framework969empowers a nuanced understanding of how latent representations affect LLM970behavior with and without ICL, offering valuable tools and insights for future971research and practical applications.972"973From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and  Privacy,Maanak Gupta,http://arxiv.org/pdf/2307.00691v1.pdf,2023-07-03,"['cs.cr', 'cs.ai']",2307.00691v1.pdf,"  Undoubtedly, the evolution of Generative AI (GenAI) models has been the974highlight of digital transformation in the year 2022. As the different GenAI975models like ChatGPT and Google Bard continue to foster their complexity and976capability, it's critical to understand its consequences from a cybersecurity977perspective. Several instances recently have demonstrated the use of GenAI978tools in both the defensive and offensive side of cybersecurity, and focusing979on the social, ethical and privacy implications this technology possesses. This980research paper highlights the limitations, challenges, potential risks, and981opportunities of GenAI in the domain of cybersecurity and privacy. The work982presents the vulnerabilities of ChatGPT, which can be exploited by malicious983users to exfiltrate malicious information bypassing the ethical constraints on984the model. This paper demonstrates successful example attacks like Jailbreaks,985reverse psychology, and prompt injection attacks on the ChatGPT. The paper also986investigates how cyber offenders can use the GenAI tools in developing cyber987attacks, and explore the scenarios where ChatGPT can be used by adversaries to988create social engineering attacks, phishing attacks, automated hacking, attack989payload generation, malware creation, and polymorphic malware. This paper then990examines defense techniques and uses GenAI tools to improve security measures,991including cyber defense automation, reporting, threat intelligence, secure code992generation and detection, attack identification, developing ethical guidelines,993incidence response plans, and malware detection. We will also discuss the994social, legal, and ethical implications of ChatGPT. In conclusion, the paper995highlights open challenges and future directions to make this GenAI secure,996safe, trustworthy, and ethical as the community understands its cybersecurity997impacts.998"999Evaluating the Instruction-Following Robustness of Large Language Models  to Prompt Injection,Zekun Li,http://arxiv.org/pdf/2308.10819v2.pdf,2023-08-17,"['cs.cl', 'cs.ai']",2308.10819v2.pdf,"  Large Language Models (LLMs) have shown remarkable proficiency in following1000instructions, making them valuable in customer-facing applications. However,1001their impressive capabilities also raise concerns about the amplification of1002risks posed by adversarial instructions, which can be injected into the model1003input by third-party attackers to manipulate LLMs' original instructions and1004prompt unintended actions and content. Therefore, it is crucial to understand1005LLMs' ability to accurately discern which instructions to follow to ensure1006their safe deployment in real-world scenarios. In this paper, we propose a1007pioneering benchmark for automatically evaluating the robustness of1008instruction-following LLMs against adversarial instructions injected in the1009prompt. The objective of this benchmark is to quantify the extent to which LLMs1010are influenced by injected adversarial instructions and assess their ability to1011differentiate between these injected adversarial instructions and original user1012instructions. Through experiments conducted with state-of-the-art1013instruction-following LLMs, we uncover significant limitations in their1014robustness against adversarial instruction injection attacks. Furthermore, our1015findings indicate that prevalent instruction-tuned models are prone to being1016``overfitted'' to follow any instruction phrase in the prompt without truly1017understanding which instructions should be followed. This highlights the need1018to address the challenge of training models to comprehend prompts instead of1019merely following instruction phrases and completing the text. The data and code1020can be found at \url{https://github.com/Leezekun/Adv-Instruct-Eval}.1021"1022Demystifying RCE Vulnerabilities in LLM-Integrated Apps,Tong Liu,http://arxiv.org/pdf/2309.02926v2.pdf,2023-09-06,['cs.cr'],2309.02926v2.pdf,"  In recent years, Large Language Models (LLMs) have demonstrated remarkable1023potential across various downstream tasks. LLM-integrated frameworks, which1024serve as the essential infrastructure, have given rise to many LLM-integrated1025web apps. However, some of these frameworks suffer from Remote Code Execution1026(RCE) vulnerabilities, allowing attackers to execute arbitrary code on apps'1027servers remotely via prompt injections. Despite the severity of these1028vulnerabilities, no existing work has been conducted for a systematic1029investigation of them. This leaves a great challenge on how to detect1030vulnerabilities in frameworks as well as LLM-integrated apps in real-world1031scenarios. To fill this gap, we present two novel strategies, including 1) a1032static analysis-based tool called LLMSmith to scan the source code of the1033framework to detect potential RCE vulnerabilities and 2) a prompt-based1034automated testing approach to verify the vulnerability in LLM-integrated web1035apps. We discovered 13 vulnerabilities in 6 frameworks, including 12 RCE1036vulnerabilities and 1 arbitrary file read/write vulnerability. 11 of them are1037confirmed by the framework developers, resulting in the assignment of 7 CVE1038IDs. After testing 51 apps, we found vulnerabilities in 17 apps, 16 of which1039are vulnerable to RCE and 1 to SQL injection. We responsibly reported all 171040issues to the corresponding developers and received acknowledgments.1041Furthermore, we amplify the attack impact beyond achieving RCE by allowing1042attackers to exploit other app users (e.g. app responses hijacking, user API1043key leakage) without direct interaction between the attacker and the victim.1044Lastly, we propose some mitigating strategies for improving the security1045awareness of both framework and app developers, helping them to mitigate these1046risks effectively.1047"1048Hydrogen-rich supernovae beyond the neutrino-driven core-collapse  paradigm,G. Terreran,http://arxiv.org/pdf/1709.10475v1.pdf,2017-09-29,['astro-ph.sr'],1709.10475v1.pdf,"  We present our study of OGLE-2014-SN-073, one of the brightest Type II SN1049ever discovered, with an unusually broad lightcurve combined with high ejecta1050velocities. From our hydrodynamical modelling we infer a remarkable ejecta mass1051of $60^{+42}_{-16}$~M$_\odot$, and a relatively high explosion energy of1052$12.4^{+13.0}_{-5.9} \times10^{51}$~erg. We show that this object belongs, with1053a very small number of other hydrogen-rich SNe, to an energy regime that is not1054explained by standard core-collapse (CC) neutrino-driven explosions. We compare1055the quantities inferred by the hydrodynamical modelling with the expectations1056of various exploding scenarios, trying to explain the high energy and1057luminosity released. We find some qualitative similarities with1058pair-instabilities SNe, although a prompt injection of energy by a magnetar1059seems also a viable alternative to explain such extreme event.1060"1061Robust Prompt Optimization for Large Language Models Against  Distribution Shifts,Moxin Li,http://arxiv.org/pdf/2305.13954v2.pdf,2023-05-23,"['cs.cl', 'cs.ai']",2305.13954v2.pdf,"  Large Language Model (LLM) has demonstrated significant ability in various1062Natural Language Processing tasks. However, their effectiveness is highly1063dependent on the phrasing of the task prompt, leading to research on automatic1064prompt optimization using labeled task data. We reveal that these prompt1065optimization techniques are vulnerable to distribution shifts such as1066subpopulation shifts, which are common for LLMs in real-world scenarios such as1067customer reviews analysis. In this light, we propose a new problem of robust1068prompt optimization for LLMs against distribution shifts, which requires the1069prompt optimized over the labeled source group can simultaneously generalize to1070an unlabeled target group. To solve this problem, we propose Generalized Prompt1071Optimization framework, which incorporates the unlabeled data from the target1072group into prompt optimization. Extensive experimental results demonstrate the1073effectiveness of the proposed framework with significant performance1074improvement on the target group and comparable performance on the source group.1075"1076MultiPrompter: Cooperative Prompt Optimization with Multi-Agent  Reinforcement Learning,Dong-Ki Kim,http://arxiv.org/pdf/2310.16730v1.pdf,2023-10-25,['cs.lg'],2310.16730v1.pdf,"  Recently, there has been an increasing interest in automated prompt1077optimization based on reinforcement learning (RL). This approach offers1078important advantages, such as generating interpretable prompts and being1079compatible with black-box foundation models. However, the substantial prompt1080space size poses challenges for RL-based methods, often leading to suboptimal1081policy convergence. This paper introduces MultiPrompter, a new framework that1082views prompt optimization as a cooperative game between prompters which take1083turns composing a prompt together. Our cooperative prompt optimization1084effectively reduces the problem size and helps prompters learn optimal prompts.1085We test our method on the text-to-image task and show its ability to generate1086higher-quality images than baselines.1087"1088Dialogue for Prompting: a Policy-Gradient-Based Discrete Prompt  Optimization for Few-shot Learning,Chengzhengxu Li,http://arxiv.org/pdf/2308.07272v1.pdf,2023-08-14,"['cs.lg', 'cs.cl']",2308.07272v1.pdf,"  Prompt-based pre-trained language models (PLMs) paradigm have succeeded1089substantially in few-shot natural language processing (NLP) tasks. However,1090prior discrete prompt optimization methods require expert knowledge to design1091the base prompt set and identify high-quality prompts, which is costly,1092inefficient, and subjective. Meanwhile, existing continuous prompt optimization1093methods improve the performance by learning the ideal prompts through the1094gradient information of PLMs, whose high computational cost, and low1095readability and generalizability are often concerning. To address the research1096gap, we propose a Dialogue-comprised Policy-gradient-based Discrete Prompt1097Optimization ($DP_2O$) method. We first design a multi-round dialogue alignment1098strategy for readability prompt set generation based on GPT-4. Furthermore, we1099propose an efficient prompt screening metric to identify high-quality prompts1100with linear complexity. Finally, we construct a reinforcement learning (RL)1101framework based on policy gradients to match the prompts to inputs optimally.1102By training a policy network with only 0.67% of the PLM parameter size on the1103tasks in the few-shot setting, $DP_2O$ outperforms the state-of-the-art (SOTA)1104method by 1.52% in accuracy on average on four open-source datasets. Moreover,1105subsequent experiments also demonstrate that $DP_2O$ has good universality,1106robustness, and generalization ability.1107"1108PromptAgent: Strategic Planning with Language Models Enables  Expert-level Prompt Optimization,Xinyuan Wang,http://arxiv.org/pdf/2310.16427v1.pdf,2023-10-25,['cs.cl'],2310.16427v1.pdf,"  Highly effective, task-specific prompts are often heavily engineered by1109experts to integrate detailed instructions and domain insights based on a deep1110understanding of both instincts of large language models (LLMs) and the1111intricacies of the target task. However, automating the generation of such1112expert-level prompts remains elusive. Existing prompt optimization methods tend1113to overlook the depth of domain knowledge and struggle to efficiently explore1114the vast space of expert-level prompts. Addressing this, we present1115PromptAgent, an optimization method that autonomously crafts prompts equivalent1116in quality to those handcrafted by experts. At its core, PromptAgent views1117prompt optimization as a strategic planning problem and employs a principled1118planning algorithm, rooted in Monte Carlo tree search, to strategically1119navigate the expert-level prompt space. Inspired by human-like trial-and-error1120exploration, PromptAgent induces precise expert-level insights and in-depth1121instructions by reflecting on model errors and generating constructive error1122feedback. Such a novel framework allows the agent to iteratively examine1123intermediate prompts (states), refine them based on error feedbacks (actions),1124simulate future rewards, and search for high-reward paths leading to expert1125prompts. We apply PromptAgent to 12 tasks spanning three practical domains:1126BIG-Bench Hard (BBH), as well as domain-specific and general NLP tasks, showing1127it significantly outperforms strong Chain-of-Thought and recent prompt1128optimization baselines. Extensive analyses emphasize its capability to craft1129expert-level, detailed, and domain-insightful prompts with great efficiency and1130generalizability.1131"1132"Automatic Prompt Optimization with ""Gradient Descent"" and Beam Search",Reid Pryzant,http://arxiv.org/pdf/2305.03495v2.pdf,2023-05-04,"['cs.cl', 'cs.ai', 'cs.lg']",2305.03495v2.pdf,"  Large Language Models (LLMs) have shown impressive performance as general1133purpose agents, but their abilities remain highly dependent on prompts which1134are hand written with onerous trial-and-error effort. We propose a simple and1135nonparametric solution to this problem, Automatic Prompt Optimization (APO),1136which is inspired by numerical gradient descent to automatically improve1137prompts, assuming access to training data and an LLM API. The algorithm uses1138minibatches of data to form natural language ""gradients"" that criticize the1139current prompt. The gradients are then ""propagated"" into the prompt by editing1140the prompt in the opposite semantic direction of the gradient. These gradient1141descent steps are guided by a beam search and bandit selection procedure which1142significantly improves algorithmic efficiency. Preliminary results across three1143benchmark NLP tasks and the novel problem of LLM jailbreak detection suggest1144that Automatic Prompt Optimization can outperform prior prompt editing1145techniques and improve an initial prompt's performance by up to 31%, by using1146data to rewrite vague task descriptions into more precise annotation1147instructions.1148"1149Discrete Prompt Optimization via Constrained Generation for Zero-shot  Re-ranker,Sukmin Cho,http://arxiv.org/pdf/2305.13729v1.pdf,2023-05-23,"['cs.ir', 'cs.ai', 'cs.cl']",2305.13729v1.pdf,"  Re-rankers, which order retrieved documents with respect to the relevance1150score on the given query, have gained attention for the information retrieval1151(IR) task. Rather than fine-tuning the pre-trained language model (PLM), the1152large-scale language model (LLM) is utilized as a zero-shot re-ranker with1153excellent results. While LLM is highly dependent on the prompts, the impact and1154the optimization of the prompts for the zero-shot re-ranker are not explored1155yet. Along with highlighting the impact of optimization on the zero-shot1156re-ranker, we propose a novel discrete prompt optimization method, Constrained1157Prompt generation (Co-Prompt), with the metric estimating the optimum for1158re-ranking. Co-Prompt guides the generated texts from PLM toward optimal1159prompts based on the metric without parameter update. The experimental results1160demonstrate that Co-Prompt leads to outstanding re-ranking performance against1161the baselines. Also, Co-Prompt generates more interpretable prompts for humans1162against other prompt optimization methods.1163"1164Query-Dependent Prompt Evaluation and Optimization with Offline Inverse  RL,Hao Sun,http://arxiv.org/pdf/2309.06553v3.pdf,2023-09-13,"['cs.cl', 'cs.ai', 'cs.lg']",2309.06553v3.pdf,"  In this study, we aim to enhance the arithmetic reasoning ability of Large1165Language Models (LLMs) through zero-shot prompt optimization. We identify a1166previously overlooked objective of query dependency in such optimization and1167elucidate two ensuing challenges that impede the successful and economical1168design of prompt optimization techniques. One primary issue is the absence of1169an effective method to evaluate prompts during inference when the golden answer1170is unavailable. Concurrently, learning via interactions with the LLMs to1171navigate the expansive natural language prompting space proves to be1172resource-intensive. To address this, we introduce Prompt-OIRL, which harnesses1173offline inverse reinforcement learning to draw insights from offline prompting1174demonstration data. Such data exists as by-products when diverse prompts are1175benchmarked on open-accessible datasets. With Prompt-OIRL, the query-dependent1176prompt optimization objective is achieved by first learning an offline reward1177model. This model can evaluate any query-prompt pairs without accessing LLMs.1178Subsequently, a best-of-N strategy is deployed to recommend the optimal prompt.1179Our experimental evaluations across various LLM scales and arithmetic reasoning1180datasets underscore both the efficacy and economic viability of the proposed1181approach.1182"1183ATT3D: Amortized Text-to-3D Object Synthesis,Jonathan Lorraine,http://arxiv.org/pdf/2306.07349v1.pdf,2023-06-06,"['cs.lg', 'cs.ai', 'cs.cv', '68t45', 'i.2.6; i.2.7; i.3.6; i.3.7']",2306.07349v1.pdf,"  Text-to-3D modelling has seen exciting progress by combining generative1184text-to-image models with image-to-3D methods like Neural Radiance Fields.1185DreamFusion recently achieved high-quality results but requires a lengthy,1186per-prompt optimization to create 3D objects. To address this, we amortize1187optimization over text prompts by training on many prompts simultaneously with1188a unified model, instead of separately. With this, we share computation across1189a prompt set, training in less time than per-prompt optimization. Our framework1190- Amortized text-to-3D (ATT3D) - enables knowledge-sharing between prompts to1191generalize to unseen setups and smooth interpolations between text for novel1192assets and simple animations.1193"1194Temporally-Extended Prompts Optimization for SAM in Interactive Medical  Image Segmentation,Chuyun Shen,http://arxiv.org/pdf/2306.08958v1.pdf,2023-06-15,"['cs.cv', 'cs.ai', 'cs.lg']",2306.08958v1.pdf,"  The Segmentation Anything Model (SAM) has recently emerged as a foundation1195model for addressing image segmentation. Owing to the intrinsic complexity of1196medical images and the high annotation cost, the medical image segmentation1197(MIS) community has been encouraged to investigate SAM's zero-shot capabilities1198to facilitate automatic annotation. Inspired by the extraordinary1199accomplishments of interactive medical image segmentation (IMIS) paradigm, this1200paper focuses on assessing the potential of SAM's zero-shot capabilities within

Showing the first 1,200 of 30295 lines. Download the file for the rest.