Why I Created a Rust Application Console

At the very start of my software career I spent most of my time working with .Net and Java. In that world, I never really came across very advanced operational interfaces for software. Most of that world was large enterprise type software where the closest I ever came to "messing around in prod" was writing some SQL queries against a prod db. All of that is to say that it wasn't until the middle of my career that I first became acquainted with the idea of using a REPL in a production environment for operational tasks.

My first brush with this glorious and terrible technology was the wondrous Rails Console. I first encountered this while working in Intercom. At the time, things in Intercom were still a bit more loose in terms of processes etc., so it was trivial for a developer to jump onto a production Rails Console to do some data investigation, replicate an issue or fix up some data.

This might be sending twitches to the eyes of experienced developers who have (possibly more than once) fat fingered something and caused an outage. That said, it can't be denied how useful and powerful this kind of tool can be for quick and dirty fixes. By the time I left Intercom, things had been locked down considerably around just being able to yolo onto a prod instance and do your worst. Indeed, many companies who start off with this kind of go-to tooling eventually end up locking it down as they mature. With that said, there are often times when this really is the best tool for the job, so the ability to access these tools, even if it's locked down, can be a life saver.

TL;DR

I build a tool for running a REPL and executing scripts via Rhia for Rust apps. You can find that project here and the crate itself here.

Background

Aside from the Rails Console, if you are a Python developer, chances are you might have written some Django and if so you will be aware of the Django Shell, which is similar to the Rails Console. Even if you are not using Django, Python supports many flavours of an interactive REPL environment which can be really useful for testing things out or debugging. In Nory we ran into an interesting case and lamented the lack of a toolset like this. To set the stage, Nory does use Python for its main services, however we don't use Django (we use a kind of a mash up of different Python web frameworks). This wasn't really a huge issue; for the most part, we would write scripts that we could run on prod and for other tasks we would directly access the prod db to perform data fix ups etc. These aren't ideal situations, but they were the working reality we had.

A slight wrinkle arose when we decided to take on a large scale data migration project for one of our services. Critically, this couldn't involve downtime or the main team working on that service downing tools for "x" amount of time. If you have done a large scale migration project under these kind of conditions before, you will know that you usually end up with a dual write situation at some point. These tend to live for a relatively long time. One issue that this presents, depending on how you have things set up, is that you can no longer modify data directly via the db(*). One way of avoiding issues of diverging data is to ensure anything that mutates data is proxied through your system models either via a repository or service layer that wraps both sources and deals with writing equivalent data to both places. This obviously precludes the ability to manually modify data in the db and encourages all modifications of data to happen in your own managed code base.

This is really sounding like a situation where a scripting layer and/or REPL would be really handy. Assuming you are using something like Ruby or Python, when I say scripting layer, I just mean a way to run scripts in your production environment accessing all of your code, repository or service methods we mentioned earlier. A REPL is not strictly necessary in the situation I have described, but it really does enable quick and dirty fixes and investigations better than almost any other process where the situation calls for it. Aside from my example above, the use of proxying all data access through your managed code also allows you to ensure consistency with things like app level validation rules and/or relationships that your db might not strictly enforce on its own. REPL or not, using your own models means you avoid situations where you directly script data in your db and then break the system. The breakage usually comes from domain constraints that your db does not enforce, but which will still blow up the app due to unexpected state. I swear I'm going somewhere with all of this, but I just wanted to give you my brain dump of context around this issue as it's been pretty top of mind in my work recently.

As an aside, I should mention what might be typical in other environments without a scripting layer or REPL context. I spoke to a friend of mine recently who was describing how he had to recently do some out of band queries to test something in his production environment. He is in the JVM world, so I was curious as to how they manage these kind of ad-hoc "script-y" type workloads. He described something that is fairly similar to other processes I have heard of. Somewhere in their code, you need to write whatever code you need to execute. This is then wired up to an internal only http endpoint where you can trigger it by firing a request at the endpoint. You will hear other versions of this where you need to write your code into a worker instance and then trigger it by enqueuing a message onto an SQS queue or similar. These are all 100% acceptable ways of achieving the same result, executing some code in a production environment proxying your interactions with your core models etc. With most of these options, there is still just a little bit more friction introduced to the developer to do their work than with a REPL or natively supported scripting environment in production.

At this point, I should also give an honourable mention to the likes of Marimo and Jupyter. Notebook environments are slightly different again, but can be used for some of these things. I am not going to get into them deeply here as they are slightly a digression in my opinion, more than we have already digressed anyway!

(*) You can of course still do this in some set ups. Depending on if you have a robust method for validating data across both sources and automatically correcting for divergences etc.

Bringing it to Rust

With all of the above said, that brings me to how I came up with the idea of rhai-console. I use Rust for lots of my personal projects. Over the past few years it has fast become my favourite programming language, and I have been using it for everything from data crunching terminal apps, to web applications, to embedded systems projects. Really, if it needs some code written, I've found a way to use Rust. While thinking about the above set of problems, I got thinking about how I would deal with this issue in a Rust application. Obviously being a fully compiled language, this means I would have to set up either a set of worker processes, or one off extra binaries in my build or a queue system to manage ad-hoc code execution in prod. This is then where I had the thought of using Rhai to help out. Rhai is an embedded scripting language for Rust. It is lightweight and powerful, and importantly, it can directly interface with your Rust code through a flexible registration system where you can compose a scripting DSL directly from your application logic.

So with all of that context, the idea of rhai-console is really just stitching these concepts together into a small crate that gives a Rust application the same kind of operational surface that Rails Console or Django Shell gives Ruby and Python apps. You hand the crate your application state and a few module registrations and you get back a CLI that you can drop into either as an interactive REPL or use as a script runner. Both modes hit the exact same service layer your HTTP handlers or background workers would use. You are not poking at the database directly or wiring up one-off endpoints, you are just running real code through your real domain interface, but on demand.

How it Works

At a high level, the design of the crate is pretty straightforward. You build up a Console object, pass it your application state and register some modules; everything else is handled for you. A module in this context is just a function that takes a mutable reference to a Rhai Module and a clone of your state, and uses a small macro called reg! to wire up individual functions to the script environment. Each reg! invocation does a few things behind the scenes. It captures the call site so that any error that bubbles up out of your domain code carries the line and column in the script where the function was called. It wraps your function so that an error of your own type gets mapped into a Rhai runtime error with that position attached. And it uses serde to convert whatever value your function returned into something Rhai can work with at the script level.

Once you have done your registrations, calling .run() on the console parses the command line arguments, and either drops you into an interactive REPL or runs a script file you pass on the command line. Both paths share the same engine, so a script can do anything you could type at the prompt, and any positional arguments you put after the script path are made available inside the script as an args array of strings. Errors are formatted into a traceback that shows the failing line and column with a caret pointing at the call, and walks the chain back up through any Rhai-defined wrapper functions. The result looks a bit more like a real backtrace, rather than the relatively opaque error you would get from Rhai directly.

One thing I quite like about how this ended up is that the help listing in the REPL is generated automatically from the registrations themselves. Because each reg! call attaches parameter metadata to the function, when you ask the REPL for help you get back a list of every registered function with proper argument names and types, rather than something like function(_, _, _) which is what you would get out of Rhai by default. Adding a new function to your application is just a matter of adding one more line to the relevant module file and the REPL picks it up the next time you build.

I built a small example application alongside the crate to demonstrate how all of this fits together in a "real" project. It's a tiny Rocket and Diesel web service with a React frontend, exposing a service layer over both HTTP and a Rhai REPL. You can find that project here and the crate itself here.

What's Missing

Right now you get :help and :quit as built-in directives and that is the entire surface. Internally the directives are pulled from a single canonical list, so adding more is a matter of appending to that list, but there is no public API yet for consumers to plug in their own directives. The same goes for things like history persistence across sessions, authentication or authorisation hooks, and anything to do with sandboxing the scripts themselves. None of that is in scope right now and most of it probably belongs in a v0.1 or later, but they are all things worth thinking about.

On the implementation side, I would like to explore building a procedural macro that takes a service impl block and generates the Rhai module registration automatically. The reg! macro is already pretty terse, but a proc macro could go further and remove the boilerplate entirely by walking the methods on a service and emitting one registration per public method. The trade-off is that a proc macro is a real project to build and maintain, so the simpler macro_rules approach is the right call for the v0.0.x line. Beyond that, async support is the obvious other thing on the wishlist. Rhai itself has a sync mode (which is what the crate currently uses) and a more flexible mode. Exposing async cleanly requires some thought about how to give scripts access to your async operations without doing anything ugly with the REPL loop.

What's Next?

I would like to use rhai-console in anger on a real Rust project of my own to actually feel out where the rough edges really are. The example app gave me a sense of the shape of the API, but it's small enough that some of the harder problems probably haven't shown up yet. I expect the limitations I mentioned above will get prioritised based on what hurts most in actual use, rather than based on what looks tidy on paper. If any of this sounds useful and you end up trying it, I would love to hear what works and what doesn't.

Keep on making things

-Ian