LLMMarshal Honors Thesis

This thesis project (and paper) is a conversation management system for LLM chatbots that prioritizes privacy, usability, and performance.

Introduction

I worked on this project as my senior honors thesis with Professor Malte Schwarzkopf in the ETOS research group. My interests in computer systems and user privacy drove my idea to build a conversation management system for LLM chatbots that protects privacy for users, while also ensuring usability and performance for developers.

Abstract

As large language models (LLMs) grow more capable and widely available, developers are integrating them into applications to provide a wide range of functionalities such as customer service, therapy, and more. The diverse use cases of LLM chatbots differ in the kinds of sessions and conversations they require. Some apps need ephemeral conversations, where no data is preserved beyond an interaction, while other apps require persistent conversations. Some systems must support authenticated or anonymous users, or allow group interactions with limited shared history. Each case has different requirements for storing chat history and isolating users from one another.

Further, upon deployment, developers must reason about and manage these LLM chatbots and their resources, including session data, conversation history, and identity management, at scale. Many developers turn to open-source LLM libraries to build this infrastructure, but these libraries often prioritize flexibility and require significant configuration effort. This results in developers writing fragile code with privacy and security vulnerabilities, like conversation history leaks across users, storage of excessive data, or improper handling of authentication. This also results in inefficient configurations of LLM chatbots, unnecessarily increasing latency or resource footprint.

We present LLMMarshal, an out-of-the-box conversation management system for LLM chatbots that prioritizes privacy, usability, and performance. LLMMarshal provides web middlewares and a standardized API for managing sessions, storing chat history, and enforcing access control, unlike existing LLM-management libraries.

We evaluate LLMMarshal on an open source business management software framework, BloomERP. Results indicate that LLMMarshal can help reduce the complexity of session and conversation management, as well as ease the process of building chatbot systems that preserve user privacy.

Paper

Our paper details the design and implementation of the LLMMarshal system, including the challenges we faced and the solutions we developed. Please find the final draft of the thesis here.

Reflections

I began this thesis in January of 2025, and I spent my junior spring semester workshopping ideas for the project. Early on, I was interested in working with LLMs and exploring the privacy implications of their use, so I learned LangChain, one of the most popular LLM management libraries. I also explored Github for example use cases of LangChain, and I found that many developers were using it to build chatbots for many purposes, but often struggled with the complexity of configuring the library and managing users.

In the fall of 2025, I began building the LLMMarshal system, and I iteratively built out the features of the system, including session management, chat history storage, and access controls. Once I had a working prototype, I deployed it on BloomERP, an open source business management software framework, to evaluate its performance and usability. In the spring of 2026, I wrote the thesis paper and presented my work at the Brown CS Research Symposium.

In the future, I would like to further evaluate the LLMMarshal system on more applications and with real users, to better understand whether LLMMarshal adapts to the needs of real-world applications. I also want to explore how LLMMarshal can support other LLM-interacing libraries beyond LangChain, backed by web frameworks beyond Django and languages beyond Python, like Rust.

My collaborator, Kinan Dak Albab, is working on exploring these directions in the SPACE lab at Boston University.