Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals

Echoing Celikyilmaz et al. [16], one can split evaluation methods into automated or machine-learned metrics, and human-centric evaluation. As detailed in the Appendix A.4, automated and machine-learned metrics typically calculate 1) the similarity between generated and “ground truth” stories [38, 39, 88, 105], the 2) consistency between generated stories and their writing prompts [92, 102], or 3) the diversity of language in the generated output [43, 45, 46, 88, 127]. These metrics were not designed for generated text of the length of a screenplay or theatre script. This motivates a focus on human-centric evaluation, which can be conducted with non-expert crowdworkers or with experts.

We therefore review the limitations of crowdsourced evaluation of the coherence of generated text, and explain why non-expert, crowdsourced evaluation faces crucial quality and bias issues. Inheriting from research standards for large-scale natural language processing tasks, the majority of studies assessing the quality of generations from LLMs evaluate model performance by collecting data from crowdworkers [58, 77, 80, 88, 91, 124]. For instance Yao et al. [124] recruit crowdworkers to evaluate fidelity, coherence, interestingness, and popularity, and Rashkin et al. [88] to evaluate narrative flow and ordering of events. That said, it has been shown that crowdworkers’ personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53]. These issues have led some researchers to try evaluating their models with experts. Karpinska et al. [56] highlight the perils of using crowdsourced workers for evaluating open-ended generated text, because crowdworkers do not read text as carefully as expert teachers. Some studies consulted with expert linguists [31], university students in the sciences [42] or humanities [123], and amateur writers [2, 21, 62, 100, 125]. In one recent study, Calderwood et al. [13] interviewed 4 novelists about their usage of GPT-2 via Talk To Transformer and Write With Transformer tools (see https://app.inferkit.com/demo), uncovering usages of the “model as antagonist” (i.e., random), for “description creation”, as “constraint”, or for “unexpected” ideas—some of the themes highlighted by our own study participants. Similarly, a recent user study explored evaluation of writing short stories with a collaborative text editing tool called Wordcraft [125].

The processes of authoring theater scripts and screenplays are different, and each author has their own distinct processes. For example, in Section 7.2, we present several unique perspectives on processes from our study participants. That said, previous works have attempted to distill common patterns and lessons for writing theatre scripts [33, 87] and screenplays [106]. Writing can be solo or collaborative, but, even when writing individually, ‘the only kind of writing is rewriting’ (Hemingway [49]) of previous drafts and revisions. Interactive authorship is the collaboration of several individuals to co-write scripts as often happens in television show creation, where screenplays are written in writers’ rooms. In interactive authorship settings, writers may use creative inspirations to accelerate and inspire the writing process. The process of generating stories for theatre has a long history of these kinds of technological inspiration [23]. The implicit hypothesis of our work on Dramatron is that interactive authorship alongside an AI-based writing tool could be useful for writers, similar to the contribution of multiple writers in a writer’s room working collaboratively on a theatre script or screenplay. For this reason, we want Dramatron to output scripts in a format that is familiar to creative professionals, in script form with, for example, lines of dialogue prefixed by character names.

Automatic (Section A.1), hierarchical (Section A.2) and controllable (Section A.3) story generation are long-standing research areas that we review in the Appendix. Our work builds upon these but focuses on (the evaluation of) interactive authorship with story generation models. Recent work addressed human-AI text revision [31], text summarization [18, 123] and overcoming the writer’s block [22, 42, 77, 125]. Notably, Wordcraft [125] used an LLM with an editor and an interface that asked to continue the story, asked for details, or suggested to rewrite it, and was used as a tool for idea generation, copy editing or scene interpolation. Our model Dramatron allows some of these writer’s interventions within a hierarchical generation structure.

Automatic generation of stories with dialogue has been used to populate digital worlds in video games, interactive narratives, entertainment, virtual worlds [82], artistic performances [48], improvised theatre [10, 70, 75], short film scripts like Sunspring in 2016, song music lyrics for musical Beyond the Fence in 2016 in London¹ and interactive playwriting for AI at the Young Vic in 2021 in London². With the exception of Prague-based company THEaiTRE [93, 95, 99], none of these related works have been used to generate long-range coherent theatre scripts or screenplays. And, unlike Dramatron, none of them used few-shot learning and prompt engineering to prime LLMs for generation.

Statistical language models (language models, or LMs) model the probability of text tokens given a context of previous tokens—tokens can be words, characters, or character bi-grams. Using machine learning, LMs are trained on large corpora of text to approximate the conditional probability distribution. LMs can compute the likelihood of a piece of text or to generate new text as the continuation of a text prompt. Text generation is probabilistic and involves random sampling from the conditional probabilities. Different random seeds result in different random samples. Figure 2 illustrates an example of feeding a text prompt and using the LM to generate different text samples.

In this study, we employed the Chinchilla large language model (LLM) [50], represented as a neural network with 70B-parameters and that was trained on 1.4T tokens of the MassiveText dataset. As described by Rae et al. [86], that corpora contains 604M MassiveWeb documents, 4M Books, 361M questions and responses from C4, 1.1B News articles, 142M GitHub code entries, and 6M Wikipedia articles. We conducted our study using Chinchilla 70B because it outperformed other LLMs on a large collection of language tasks [50]. We have since then successfully tested our approach with Dramatron as is, by substituting Chinchilla with alternative LLMs³, such as GPT-3 text-davinci-003 [12].

LLMs can give the impression of coherence within and between paragraphs [7], but have difficulty with long-term semantic coherence due to the restricted size of their context windows. Memory wise, they require \(\mathcal {O}(n^2)\) (where n is the number of tokens in the context window). Thus, these models currently restrict n to 2048 tokens [12, 78]. Because these models do not achieve long-term semantic coherence, the current state-of-the-art method for longer text generation incorporates a human-in-the-loop who selects from various generations sampled from the model [10, 67, 75]. The human is tasked with achieving long-term semantic coherence: the model generates choices for the next textual snippet, while the human does the selection⁴.

To circumvent the limitations of LLMs and to enable the generation of theatre and film scripts, we take a divide-and-conquer approach: we structure script generation using interdependent narrative elements. We base our approach on Aristotle’s Poetics [5], which identified 1) plot (mythos) as the most important part of a tragic story—the other elements being the 2) theme of the story (dianoia), 3) its characters (ethos), 4) dialogues (lexis), as well as melody and spectacle. Thus, we start from a user-defined theme and use Dramatron to help write characters, plot, and dialogue.

Aristotle introduced the well-known plot structure ( beginning , middle , and end ). Many other dramatic structures followed [84]. For example, variations on the Freytag’s pyramid [41], popular in Western storytelling⁵ ( Exposition , Inciting Incident , a Rising Action composed of a series of Conflicts , Complications , and Dilemmas , Climax , Falling Action , Resolution , and Dénouement ) or the Hero’s Journey or Monomyth [15, 113]. More generally, the plot is the coherent sequence of consecutive actions that shape the story. In order to achieve narrative coherence despite the limited context window of LLMs, we generate a plot synopsis in one go, as a sequence of beats.

Oftentimes the seed of narrative inspiration for a screenplay or theatre script is a log line [101]. The log line summarizes the story in a few sentences and is often adapted over the course of production due to creative team preferences and cultural references. Log lines typically contain the setting, protagonist, antagonist, a conflict or goal, and sometimes the inciting incident [8], and hence suggest short answers to questions: Who? What? When and Where? How? Why? [104] In this work, we find a connection between Artistotle’s theme and the log line, and we use log lines to start the hierarchical story generation process.

In this project, we desire a system that provides useful suggestions to the writer, and that avoids generating incoherent dialogue given a theme, plot or choice of characters.For this reason, we aim at building a system that can generate an entire text exhibiting long-term semantic coherence. We hypothesize that if the system can generate an entire script exhibiting reasonable long-term coherence from a single log line without human intervention, then such scripts can serve as drafts for the writer. Our approach to achieve long-term semantic coherence is to generate the story hierarchically.

Our narrative generation is conceptually divided into three hierarchical layers of abstraction that loosely interpret the narrative elements of Aristotle’s tragedy. The highest layer is the log line (or theme) defined in Section 3.3: a single sentence describing the central dramatic conflict. The middle layer contains character descriptions, a plot outline (a sequence of high-level scene descriptions together with corresponding locations), as well as location descriptions. The bottom layer is the actual character dialogue for the text of the script. In this way, content at each layer is coherent with content in other layers. Note that “coherent” here refers to “forming a unified whole”, not assuming any common sense and logical or emotion consistency to the LLM-generated text.

As illustrated on Figure 1, the story is generated top-down [96, 111, 116]. After the human provides the log line, Dramatron generates a list of characters, then a plot, and then descriptions of each location mentioned in the plot. Characters, plot, and location descriptions all meet the specification in the log line, in addition to causal dependencies, enabled by prompt chaining [121] and explained on the diagram of Figure 1. Finally, for each scene in the plot outline, Dramatron generates dialogue satisfying previously generated scene specifications. Resulting dialogues are appended together to generate the final output. This hierarchical generation was designed to enable long-term semantic coherence. A similar albeit inverted, method of recursive task decomposition was used to generate plot summaries [120]. The incorporation of the middle layer, where the plot is summarised as a sequence of abstract scene descriptions, allows the entire plot to fit within the language models’ context window. This overcomes prior limitations on long-term semantic coherence. Our method makes it possible for elements in the final scene to provide dramatic closure on elements introduced in the opening scene⁶, and for generated stories to follow narrative arcs (see Section 3.3).

Dramatron uses several hard-coded prompts (i.e. input prefixes) to guide the large language model. Prompt engineering is a common way that users control or influence LLMs [12]. Each prompt has a few examples of desirable outputs. These are included in the prefix and adaptation to only a handful of examples is sometimes referred to as few-shot learning. As illustrated in Figure 2, prompts are concatenated with user-supplied inputs and/or outputs of previous LLM generations. This method is called prompt chaining [121], which is a type of algorithmic prompting [24]. At lower levels of the hierarchy (see Fig. 1), prompts are chained together with outputs from higher levels of the hierarchy.

In this work, we primarily used two sets of prompts: one based on Ancient Greek tragedy Medea by Euripides, and one based on science-fiction films. For Dramatron, each prompt set is composed of: 1) title prompt, 2) character description prompt, 3) plot prompt, 4) location description prompt, 5) and dialogue prompt. For both of these prompt sets, we chose dialogues that were in the public domain. The Medea plot was chosen because it corresponded to the Freytag Pyramid narrative arc, whereas the sci-fi plot followed an alternative narrative structure, the Hero’s Journey. Each prompt is detailed briefly below to give a sense of how they are engineered; additional details are in Appendix E. We engineered the hierarchical prompt structure and the few-shot examples through iterative prompting and testing. We chose and reworded the prompts so that the outputs of the LLM could be parsed into various components (title, character description, etc.) with high probability of success.

The Title Prompt is used to generate titles from a log line. A simplified title prompt, a user-provided log line, and randomly sampled titles are shown in Figure 2. It shows a prefix with an instruction (Examples of alternative, original and descriptive titles for known play and film scripts.) and an example (Example 1. Ancient Greek tragedy [...]. Title: In My Brother’s Name<end>). The prefix finishes with: Example 2. A user-input log line (e.g., Grandma Phyllis and Grandpa Jim [...]) is concatenated to that prefix, as well as the tag Title:, which encourages the LLM to generate a title that matches the log line. From a few examples, the LLM has “learned” to generate a related title and terminate tag <end>. The Character Description Prompt is used to generate character names and descriptions from a log line. The Plot Outline Prompt is used to turn a log line and list of characters into a plot. This prompt encourages the few-shot language model to transform a single sentence log line into a sequence of scene descriptions. Each scene is highly compressed, describing only the short name of the location, the narrative element identifying the position of the scene in the narrative arc (see Sec. 3.3), and a summary of what the characters are doing and saying, often called a narrative beat[71] As a note, the prompt imposes a strong representational constraint on the way Dramatron represents a scene; each scene is composed of a location, narrative element identifier, and beat. The Location Description Prompt is used to generate a detailed scenic description from a place name and a log line. Finally, the Dialogue Prompt is used to turn a beat (i.e., the scene summary), scene location description, description of each of the characters involved in the scene, and the log line (for story consistency), into dialogue. This prompt uses scene information generated for both the current and previous scenes.

As described above, with just few-shot (i.e., 1-4) prompts and the user input log line, we leverage trained LLMs to generate complete scripts and screenplays. Appendix F shows an example of raw generated output. That said, we designed Dramatron for interactive co-writing, as an augmentative tool for human writers. Dramatron was built with a hypothesis in mind: that interactive authorship alongside a tool would be useful for writers. While we did not co-design the tool with experts, whenever possible during the study we iterated and improved Dramatron based on their implementable suggestions and feedback (see Table 1). Co-authorship with Dramatron proceeds as follows: a writer starts with a log line, and then generates the title, characters, plot outline, location descriptions and each scene’s dialogue step-by-step. At each step, the writer can take one, or several, of the following operations, as many times as desired:

The writer can furthermore perform these operations by stepping forward and back in the Dramatron hierarchy. For example, they could: 1) generate a title, 2) generate a new title, 3) edit the title, 4) generate a list of characters, 5) edit the characters by removing one character and changing the description of another, 6) generate a plot outline, 7) edit the plot by removing part of the narrative arc, 8) generate a continuation of that edited plot, 9) go back and rewrite the log line, etc. This co-writing approach allows the human and Dramatron to both contribute to the authorship of a script. Following these operations, the human author could further edit and format to finalize a script. Appendix G shows examples of human-edited scripts.

The code of Dramatron is implemented in Python and the user-facing interface was implemented in a Google Colab⁷ with text widgets, allowing interactive editing. Figure 3 shows a typical implementation of Dramatron, with examples of generated title (top left), characters and plot (top right). It also shows generated and re-generated place descriptions, and generated, edited and re-generated dialogue—compare bottom left with bottom right panels.

There are several special markers we use for script generation: <end> represents the end of full sequence generation token, and <stop> is a token used to mark the end of a generated line. For a given prompt (see next Sec. 3.5) fed to the LLM, up to 511 text tokens were sampled. We used Nucleus sampling [51] to encourage diverse outputs, sampling tokens from the top 0.9 probability mass, and with a softmax temperature of 1.0. Finally, in order to reduce loops in the generation of dialogue, we implemented a simple detector that cuts the generated text into blocks (delimited by 2 blank lines) and counts how many times each block appears in one single generation. Beyond a fixed threshold (e.g., 3 times), the LLM generation is ran again by sampling tokens using a different seed in the random number generator.

Modeling characters and their relationships was a recurrent theme: “can we make the system relationship-driven?” (p12), “where does status belong in character building?” (p12), “could we generate the stem of a character and then complete it?” (p15). Participant 12 suggested: “as an author, I would build a social graph of the characters relations”. Answering the question “How do you get the system to know where the scene should start and end?” (p15), three participants (p8, p13, p15) suggested fitting a narrative arc within each scene.

Several participants wanted to be able to query and dialogue with the writing model: “Have you engaged [the AI system] by trying to give it notes?” (p2) to allow it to learn about the world: “How does world building happen? Maybe the model needs to know the Ws of Stella Adler [(Who? What? Where? Why? How? etc.)] Can you get the system to answer these questions?” (p9), or to allow rewriting and reformulation: “can we ask the system to re-write with a style or context?” (p8). As p10 reiterates, iterative rewriting was a desired workflow: “I am less interested in shaping [the narrative], rather than seeing what it is saying, and refining it to see what it says, and then refining it again. A playwright has to see the play spoken before making cuts.”

Finally, p4 and p5 astutely observed that “there has been a push away from systems of Western dramaturgy, so in terms of making this most useful for the future, it might be helpful to consider how it might be used within the context of other contemporary writing”—suggesting alternative narrative structures and elements—“as the AI is not bound by the same rules that we are. So, telling it to be bound by those human rules feels limiting of the capabilities”.

As detailed in Section 5.7, the participants were engaged and provided constructive feedback about Dramatron. As one of the participants in the study remarked: “the system is so adaptable, it can change with our feedback and tweaks”. This sort of understanding of the systems modifiability empowered those that interacted with it to more freely suggest changes, knowing that they could be incorporated. In this way, the system positively benefited and evolved over the course of the participant study.

Over the course of the interviews, we incorporate the feedback we could by making small, incremental changes to the prompt prefix sets of Dramatron. Table 1 summarizes changes made as a direct result of participant’s feedback. This sort of participatory design and development is critical for creative tool generation as the feedback from users can be directly incorporated to improve the system for the next interaction. This is made possible via the modular design of the system, the lightweight prompt-based interactions, and the flexibility afforded by Dramatron. This participation also inspires participants to explore related, connected, creative ideas. For example Fig. 4 (LEFT) shows concept art for a narrative test of virtual actors interpreting a co-written script.

Creative writing for theatre is fundamentally interactive: not just between collaborating storytellers, but between storytellers and the audience. For this reason, we evaluated how scripts co-written with Dramatron could be produced on the theatre stage. In this section, we describe staging details and report evaluative reflections from both the creative team and two professional theatre reviewers.

Five scripts co-written with Dramatron were staged in public performances in August 2022. The show’s run was titled Plays By Bots and ran 7 performances over two weeks (see an image from the production on Fig. 4). In each show, different casts would act out one of the plays from the co-writing experiments. The plays span different genres, styles, characters, and storylines. The scripts were brought to life by a cast of 4-6 experienced improvisers and actors. The first-half of each script was given to each of the cast members in a sealed envelope. Only when the show began were they allowed to open the script, and then they commenced performance by reading it live in front of the audience. Once the script ran out, the actors improvised the ending, based on the context and story set out by the script⁸. During each show’s performance, the director and co-writer (participant p1 from above) introduced the project to the audience and explained that they co-wrote and edited the script using Dramatron.

There were two reviews written about the production of Plays By Bots at the festival. One of the reviews noted that the performance “proves that artificial intelligence can in fact write a hit Fringe play”. The reviewer also noted that the success of the performance was due to both the Dramatron system and the human actors, especially one performance who “mastered Dramatron’s voice and seamlessly took it off-script for the remainder of the show, much to the delight of the howling audience”. The second review was also positive. With a hint of incredulity, the reviewer complimented the abilities of Dramatron. The reviewer noted the style of Dramatron, and how that served the performance saying “if there’s a certain flatness in the dialogue, which runs to declarations, that in itself is amusing since it turned out to be perfectly suited to the deadpan comic talents of [the] improvisers,” and “the human actors continue to capture the playwright bot’s tone”. The reviewer also expressed surprise at the ability of the system to create a play that hangs together and creates a world. They further noted that some lines from Dramatron are so funny they were reprised later in the show once the human actors were improvising.

Discussions amongst the creative team compliment the reviewers and provide insights on how professional actors and improvisers found working with scripts co-written by Dramatron. Post-show discussions were facilitated and relayed to us by the director (p1 above). Four key themes emerged through these discussions which echo the themes presented earlier in Section 5. Specifically, the system has a distinct glitch style, generated text can be repetitive and fun to work with. As well, the team attributed agency to the system, and had expectations of the systems capabilities. Finally, the prevailing feedback from the creative team was that participating in the production was fun! This enthusiasm and these motivating reflections from the creative team reflect the usefulness of co-written scripts for theatre production and collaboration. Additional reflections and supporting quotes are included in Appendix B.

Of the total 15 study participants, 13 provided responses on our post-session feedback form. The form gave participants the following instruction: “When answering these questions, please reflect on the interactive co-authorship session as well as considering the use of an interactive AI system like Dramatron in the future”, and asked nine questions. Each question could be answered using a Likert-type scale ranging from Strongly Disagree 1 to 5 Strongly Agree. These questions are slightly adapted from Yuan et al. [125] and Stevenson et al. [108]: (1) I found the AI system helpful, (2) I felt like I was collaborating with the AI system, (3) I found it easy to write with the AI system, (4) I enjoyed writing with the AI system, (5) I was able to express my creative goals while writing with the AI system, (6) The script(s) written with the AI system feel unique, (7) I feel I have ownership over the created script(s), (8) I was surprised by the responses from the AI system, and (9) I’m proud of the final outputs. We also asked five free-form questions. Two questions aimed at assessing the participants’ exposure to AI writing tools (In a few words: what is your experience in using AI tools for writing for theatre of film or during performance on stage?) and their industry experience (In a few words: what is your experience in theatre or film/TV?). We used these questions to manually define a binary indicator variable (Has experience of AI writing tools) and a three-class category for their primary domain of expertise (Improvisation, Scripted Theatre and Film/TV). Three more questions gave participants an opportunity to provide developmental feedback about the system: What is one thing that the AI system did well?, What is one improvement for the AI system? and Please provide any comments, reflections, or open questions that came up for you during the co-authorship session or when answering this survey.

Aggregated responses are plotted on Figure 5 (Left), and (Right) factors responses to the question on collaboration by industry background and experience (see Figure 7 for more). The results are summarized below.

We observed that participants would skim the text and remark on specific inspiring lines, but were less focused on the overall coherence of the discourse. That said, participants did notice if the dialogue of a scene was unrelated to that scene’s beat or log line, and most noticed when the generation had excessive repetition. Thus, we pair our qualitative results with a small set of descriptive statistics measuring these features and relevant to our participants’ feedback.

Lemma-based Jaccard similarity is a metric measuring the overlap between two sets of lemmatised vocabularies. We found that when participants made multiple generations for a scene description, they generally did not choose outputs with greatest Jaccard similarity. We performed a one-sided Wilcox signed-rank test on Jaccard similarity score differences, testing whether chosen output seeds were more similar to the log line provided by participants. We found that the beats for chosen generations are not more similar to the log line when compared to the beats for generations not chosen (W = 28, p = 0.926). In fact, there is a strong trend in the opposite direction. This suggests that word overlap is not predictive of which output generation is selected.

Using an open-source repetition scorer [126], we examined the repetition scores for texts where participants generated multiple dialogue outputs but chose only one to incorporate into the script (n = 3). After computing a one-sided Wilcox signed-rank test, we found no significant difference between the repetition scores for chosen Dramatron output against alternative seed outputs (W = 57, p = 0.467). This finding aligns with our observations in the interactive sessions where participants tended to delete degenerate repetitions, or as one participant (p13) put it: "[Dramatron] puts meat on the bones... And then you trim the fat by going back and forth." We interpret our distance measures as indicators of engagement with Dramatron. Figure 6 (right) displays mean relative Levenshtein distances for each participant, along with Jaccard similarity.

We also report document length differences, to show the directionality of editing (i.e. whether participants are adding or deleting material to or from Dramatron output. Figure 6 (left) shows the mean document length difference by participant. Figure 6 (middle) shows these measures normalised and grouped by type of generation. We notice that four participants tended to add to generated outputs (three of them, p6, p13, and p15 expressed interest in staging the resulting scripts) while eight others tended to remove from and curate the outputs (including p1 who staged productions). We also notice that the plot outline was the least edited part of the script. During co-authorship we observed that participants tended to delete longer location descriptions and parts of dialogue (especially loops), and rewrite the character descriptions.

While common sense and logical consistency is an elusive goal for LLMs [7], their utility as writing tools increases as they generate more coherent outputs. Hierarchical generation can be leveraged in various ways to improve the long-range coherence of long texts. First, one enhancement to improve coherence especially suited for screenplays and theatre scripts would be to generate complete character arcs. Second, generating satisfying scene conclusions in the dialogue generation step could be improved by using hierarchical generation for dialogue: construct each scene’s dialogue from a generated beginning, middle, and end dialogue beat. Third, to improve stylistic coherence, future work could explore methods to generate thematic outputs satisfying notions of genre by rejection sampling, as in [68], writing more stylized prompts, or transposing existing ones into a variety of diverse author styles and voices using large language models.

Future work on Dramatron could explore multiple different styles of writing as was suggested through feedback from experts in the study. Dramatron was originally designed for a single style: top-down hierarchical generation where each subsequent step depends on those that came prior. But, during the study, several participants noted that this style of writing is somewhat more aligned with screenwriting than playwriting. This is summarized well in one participant’s remarks: “I think the model where writers begin from inputting character and logline favours certain kinds of processes - is there a version of the system that can originate from a different point?” Another participant echoed these sentiments, saying: “Creating theatre is not always a linear / story driven process - a lot of the time it is about depth (theme) as well as forward motion (story) - characters are not always the start of the story, but a means to deliver the deeper meaning of story”.

“Playwrights do not use the log line in the same way as in screen writing” (p9), as one participant succinctly stated. Participants 4 and 5, who self-identified as playwrights for the stage, “would never approach a piece of work with a story in [their] head. [They] might come with a more investigative approach with a theme”. In fact, they argued against Dramatron’s constrained top-down hierarchy: “Why generate characters first? At earlier parts of the creation process you might not know the characters. In the way [we] work, [we] often come with characters at the end, and the idea of characters comes last”. The log line itself could be seen as a post-hoc summary, as “often times playwrights find out what the play is about after [they] finish it. I will know the play once it is done” (p9). This said, Dramatron does allow for going back-and-forth between levels in the hierarchy, and through creative prompting, future work could allow for generation steps to happen out of order.

Cultural and economic factors also lead to differences in the writing of screenplays and theatre scripts: “The difference between theatre and screen is that nobody’s making theatre on demand, no outside pressure. Or at least not in the same way that there is for television. Theatre feels less like generating content” (p4, p5) whereas “film scripts, in the industry, want the traditional fourth wall” (p9). Since Dramatron is inherently more formulaic by construction, is it suitable to screenplay and film script writing? As one respondent noted, “[Dramatron] will be very useful for an average Hollywood movie and for television. It does not need to have a deep understanding of the human soul, unlike Shakespeare. [...] The thing with action movies is that is that actors are not expected to connect with the writer. A screenwriter on a TV set is just like [Dramatron] [...]. It is a sublime skill to be a Hollywood writer because your creative input is small.” (p9).

Automatic story generation is the research problem concerned with generating sequences of story elements that collectively tell a coherent narrative. A narrative plot is a sequence of events where each affects the next. The plot is composed of narrative elements sometimes referred to as actions, beats, scenes, or events [72]. Generative plot systems have been developed for nearly a century by Cook [23], and computerized versions have existed for decades [73]. These systems support human authors with creative output material and as a source of randomness. Recent work has adapted these systems for computational interaction for use in web-based and theatrical settings [34, 35]. In generating narratives, the combinations of these component elements form subplots. Multiple subplots can be combined into a single plot, and multiple plots can intertwine to create complex narratives. Many contemporary stories are written to have multiple plot lines which intertwine. But, there is little work on how to computationally model and generate multiple intersecting plot lines. Complex plot line interaction is a promising avenue of future work for human-machine co-creativity research in story generation.

Early approaches to automatic story generation used symbolic planning and hand-engineered heuristics [63, 73, 90, 110, 117]. Recently, research has explored open-story generation using machine learning techniques which leverage large datasets, massive deep learning models, increased compute capacity, and large language model prompt engineering [11, 14, 38, 83, 89, 102]. These methods show how models can succeed, and fail, in the generation of unique and coherent stories. Additionally, while coherence has been studied in dialogue generation methods [32], it remains challenging to measure coherence in story, specifically as it relates to causal narrative events, or common sense knowledge [3], or consistency in characters [81].

Some work has tried to bridge the gap between symbolic event representations and textual representations. Several of these methods process and predict events from text [47, 66, 98, 115] by generating sequences of plot events and then expanding such plot events into sentences [4, 88]. Others model each story as a series of character and story challenge cards [2] (first simulating sequences of causal events, then transforming them into sentences) or by simulating social practices between autonomous agents [37].

Other recent work separates storytelling into two phases: storyline (i.e. plot) planning and story writing based on that storyline [124]. Similarly, methods have been introduced which decompose story generation into processes of coarse-to-fine generation [16, 39]. Goldfarb-Tarrant et al. [45] further introduced rescoring methods for character and plot events. These methods have not focused on synthesising coherent stories by generating scenes and dialogue, as we do in our work. Many of these methods lean on human reading comprehension and preference-based evaluation as opposed to production of the final script.

Hierarchical generation of a theatre play was first mentioned and used in [93, 95] for the production of AI: Can a Robot Write a Play? by company THEaiTRE in 2021 in Prague⁹. In this work, the authors start with a title (or a prompt for the story) and then generate a textual synopsis, which is then used to generate the dialogue. In contrast to our approach, they did not generate characters alongside synopsis and would start the “flat” dialogue generation from manually input two-line exchanges between characters. Their work also only served the production of a specific theatrical play rather than being evaluated within a diverse community of writers.

Previous work used a trained autoregressive transformer to plan sentences of an opinion piece from a set of user-supplied keywords [52]. This built upon [122] which incorporated commonsense knowledge into keyword-based story generation. Using conditional language models controlled by topics or keywords [74], Cho et al. [19] trained genre-controlled short story generators.

Story arc generation was introduced in the Tale Brush graphical tool [21], using Kurt Vonnegut’s theory about the fortune of the protagonist as the story progresses. This theory has also been used by Mathewson et al. [68] as an approach to produce creative, engaging dialogue. We similarly use the concept of the narrative arc, though we use it textually in the prompts to Dramatron.

In “Controlled Cue Generation for Play Scripts”, Dirik et al. [29] use LLMs to generate both the next line and a stage cue. In “DialogueScript: Using Dialogue Agents to Produce a Script”, Schmidtová et al. [99] use different LLMs for each character. Si et al. [105] model multi-user dialogue and character relationships for story continuation. Schmitt and Buschek [100] use question-based chatbot interactions to assist with character creation.

Prompt engineering has been used to write a short plot from two characters, a genre and a theme in [54]. Our work of decomposing a log line into a synopsis can be seen as a narrative equivalent of Chain of Thought prompting for reasoning tasks [118], and uses the idea of using LLM output as prompt for the next stage of the generation—also called prompt chaining [121] or language engineering [24].

Levenshtein distance was calculated at the character level for edited versus non-edited texts using the default edit distance function from the Natural Language Tool Kit¹¹ package’s distance module (no transposition operations and substitution cost equal to 1). As an absolute measure, the distance metric indicates how many operations (insertion, deletion, substitution) are required to convert one string into another and is typically calculated with a dynamic programming approach to minimizing the cost of character-level edit operations for the conversion. However, as a measure dependent on the lengths of both input strings, comparing across edit distances becomes rather uninterpretable when the string sets differ in length, for example across sets of edited and unedited texts for characters, locations, plots, and dialogues. As we are interested in understanding the extent to which participants edited the output of Dramatron as a cohort, it is reasonable to normalise the distances with respect to their length, which we report in 6 (Right).To do this, we calculate a relative Levenshtein distance as the ratio of the raw Levenshtein distance between edited and non-edited (Dramatron) output text to the length of original Dramatron output. Conceptually, we can view the resulting measure as a proportional measure of interaction with our tool. Given that Levenshtein distance operations are character-level, and length is measured in characters, the proportion represents the a weighting of active versus passive interaction with Dramatron for the different levels of structure in the hierarchical story generation process. Positive scores for relative Levenshtein distance indicate one aspect of active writing with Dramatron, while negative scores for relative Levenshtein distance indicate one aspect of passive acceptance of Dramatron (choosing among generated text seeds is another aspect of interaction not accounted for with this metric).¹²

We note that for Mean Document Length in 6 (left), the means are both negative and positive, and are not scaled. This is to observed workflow differences per participant as well as to capture the directionality of editing Dramatron’s output. For the Mean Absolute Differences in 6 (center), we take the absolute difference between character lengths between original Dramatron output and edited output and normalize it with min-max normalization.

To calculate Jaccard similarity, we first calculate the Jaccard distance, which is calculated by dividing the cardinality of the intersection of two sets and by the cardinality of their union. By subtracting Jaccard distance by 1, we obtain the Jaccard similarity. Jaccard metrics are scaled between 0 and 1, with 0 representing zero set similarity and 1 representing total set similarity. As Jaccard similarity simply compares set entities, one must specify the choice of entities to compare. In order to calculate the Jaccard similarity on Dramatron output and its edited counterparts, we apply a pre-processing pipeline to our texts, first tokenising sentences and words, and then generating a set of lemmas from word tokens based on their most probable part of speech according to WordNet [40]. The resulting scores are then lemma-based similarity scores between Dramatron output and participant edits, which we use as a descriptive measure of word choice similarity. Note that we do not calculate semantic similarity with embeddings, but simply compute set overlap at the lemma level. Lemmas are chosen to abstract over inflectional differences for words for a slightly more precise look at word choice. Future work should investigate alternative methods of assessing similarity, such as word mover distance [59] or BLEURT scores [103].

We calculate repetition-based scores with open-source tools from [126], which calculate scores for various n-gram overlaps¹³. N-gram overlaps are calculated for unigram through 10-gram sequences, as well as for Total Consecutive Reptition (TCR) and Longeset Consecutive Repetition (LCR). To compute the Wilcox test, we use pairwise differences across corresponding features for chosen versus alternative seed generations (e.g. unigram to unigram differences, bigram to bigram differences, etc.). We do not weight differences by type of repetition feature.

In our study we relied on two sets of prompts: Medea and Sci-Fi.

Medea prompts are based on Ancient Greek tragedy Medea, by Euripides (431 BC). The dialogue prompts were taken verbatim from the translation of the play by E. P. Coleridge (1863 - 1936), available in the public domain¹⁴. We replaced CHORUS by WOMEN OF CORINTH. The plot and character prompts were written from a summary taken from Spark Notes¹⁵ and Wikipedia¹⁶. To encourage the generation of different locations, Aristotle’s Unity of Place is not respected, and location “Outside the Royal Palace” is renamed as “Medea’s modest home” as well as “On a winged chariot” (even though these are the same locations in the original tragedy). Prompts for Antigone¹⁷ (Sophocles), The Bacchae¹⁸ (Euripides), and The Frogs¹⁹ (Aristophanes) were adapted from Wikipedia and ancient-literature.com.

The Star Wars log line was adapted from chapter “Creating the killer log line” by Bill Lundy²⁰. Sci-Fi character prompts were adapted from Star Wars: Episode IV - A New Hope (1977), written and directed by George Lucas, produced by Lucasfilm and distributed by 20th Century Fox. We also adapted the breakdown of Star Wars into a Hero Journey²¹. The script²², plot²³ and log line²⁴ of Plan 9 from Outer Space, which is in public domain.

In the following sets of prompt prefixes, the <LOG_LINE> is provided by the writer.

In the following sets of prompt prefixes, the <LOG_LINE> is provided by the writer.

In the following sets of prompt prefixes, the <LOG_LINE> is provided by the writer and each <CHARACTER_DESCRIPTION> is generated in the previous step.

In the following sets of prompt prefixes, the <LOG_LINE> is provided by the writer and each <LOCATION_NAME> is generated in the previous step.

In the following sets of prompt prefixes, the <LOG_LINE> is provided by the writer, <PLOT_ELEMENT>, <BEAT>, <PREVIOUS_BEAT> and <LOCATION_NAME> are generated during the plot outline generation step, <LOCATION_DESCRIPTION> is generated during the location generation step, and each <CHARACTER_DESCRIPTION> is generated in the character generation step. <PREVIOUS_BEAT> corresponds to <BEAT> from the previous scene (it is left empty for the first scene). Only characters whose name appears in the beat are used in this prompt prefix (we use string matching to select these character names).

The following description was generated for location called: The Pool Pit.