Interesting Interfaces to Large Language Models
Robots, large language models, the uncanny valley! Let's chat about all these things. Waterstons Innovation: The Dots
In July we visited our pals at The National Robotarium to chat about social housing and robotics. While we were there we happened to bump into Ameca, an incredibly technically advanced, very interactive robot. It was quite an experience… read on to find out more!
What is this? This is The Dots, our newsletter about exciting things we find in the world of innovation. We imagine innovation as connecting the dots; putting together a jigsaw. Our puzzle pieces are the pieces of interesting information we absorb in the world, our partners and their products. This newsletter is about these dots we find and connect.
Meeting Ameca
There is no doubt that Ameca is a technological marvel. With robotic muscles all over their1 face they have the ability to form facial expressions that take you on a geological tour of the uncanny valley. As their eyes flick back and forth, focusing on the faces in the room, brow furrowing as they ponder the answer to a question you definitely feel as though you’re getting a glimpse at the future.
And then everyone in the room starts excitedly speaking all at once and Ameca starts answering random questions in the wrong order and mishearing people and directing answers to the wrong person and all the magic collapses instantly.
Ameca is a complex, powerful, robotic interface to a large language model. They are basically a clever ChatGPT prompt with a humanoid face and robotic arms. This has some very interesting implications:
Just as large language models lack context, hallucinate and are not designed to interact with multiple people all at once the same is true for Ameca. So while interactions on a one on one basis with them are pretty good, if there is more than one person in the room then Ameca just assumes that everyone speaking is the same person.
While we were chatting with Ameca someone asked them to reply like they were a pirate. All their answers were suddenly delivered with a fairly dead pan “arrrr” on the end. An amusing joke but also a reminder that there is a big community dedicated to prompt hacking LLMs to “jail break” them. This leads to the tantalising realisation that it would be possible to say just the right words to Ameca to get them to swear, tell lies or pretend to be a specific celebrity.
Seeing Ameca powered up and understanding that all these clever facial expressions are being driven by an LLM Andrew and I both started to wonder what other complex, interesting and unique interfaces to LLMs you could build.
Given point 3 we decided to each dream up a brand new way to use an LLM.
New ways to interact with Large Language Models
Andrew’s Idea: Modular ChatGPT Cards
I (Andrew) started by trying to think of what might be the most fun and weird way of interacting with and AI. I came up with three ideas: Ouija board, Magic 8 Ball, and the wall of fairy lights from Stranger Things Season 1.
Though impractical, all these ideas had some similarities: they limited how much you could communicate with ChatGPT, and it would limit the responses you could get. On top of that, it would slow you down, significantly, compared to typing. Maybe there are some benefits to these assumed negatives though: (i) by increasing the time it takes to create and read a message, you may not notice that a chatbot can have a lot of latency as standard, and (ii) by reducing how much you can interact, maybe we can reduce the range of outputs the AI could have, and therefore potentially decrease its likelihood to say something untrue or hallucinate.
Next, I asked myself what some fun types of interactions were. Robots are cool, but Ameca has that covered. Cranks and levers are fun, but that is quite close to a Playdate which is basically a Rabbit R1. I ended up with this list:
Playdoh
Drumpad and beat sequencers
Punch card computers
Dropmix – a card game from Harmonix, the creators of Rock Band. Each card contained an NFC tag, so the computer that controlled the game “knew” what cards there played, and in what order. Each card contained a stem of a song (eg, the guitar part of September by Earth, Wind, and Fire, or just the vocals from Call Me Maybe) and by layering them up you could create a new song made from these parts
Lego
I feel I have some themes emerging: modularity, building a system from precomposed parts, something physical, but also something “slow” and controllable.
Practically, I want my ChatGPT interface to be a modular way to build a prompt – almost like madlibs, with each modular part being a word of phrase you can slot in. You need to physically “build” your prompt. I think this would be fun, and may even make you forget you are “talking” to an AI. Since the designer (me) can choose the phrases that can “slot” into the prompt, I can control the output – reducing naughty AI behaviour.
I love the idea of having a big synthesiser, and depending on where you put the cables that decides what the prompt is. Or having a punch card and a machine to actually punch it. However, as I write this I think NFC card game could be interesting
Idea: You have a deck of cards, each with an NFC chip inside (a la Dropmix). By placing a card on your phone, an app can tell what it is, and if there is more than 1, what order they are in.
There are different styles of cards: there could be command cards (draw me a…, write me a…., summarise….), nouns (movie script, song, cat, banana), verbs (jumping, dancing, working), and so on. By ordering them and placing them on your phone it would create a prompt that could be sent to ChatGPT. Everything would be slightly ridiculous, so the results potentially could be interesting
Why? By having lots of carefully chosen cards, it could just be an interesting experience, however maybe to give it some more direction it can start life as a game. Each player has a random hand of cards from the deck. Maybe at the start, everyone is given a task (get back from the AI something funny, something weird, trick the AI), and you need to create a prompt from a hand of cards you have to do this? Maybe its like Cards Against Humanity where they goal is to be funny, and have everyone vote that you are indeed the funniest.
Actually, I think at the start of the round you are shown the output of the AI – there is a set of cards somewhere in the deck that creates the prompt that made that output. Of course, you don’t know what did, and with the limited cards you have you have to try to recreate it. The closet person wins that round.
Would it be fun? I dunno 🤷
Overall, I think being modular (to limit options and undesirable output) and slowing things down is interesting. Having cards being the method of modularity seems fun and physical – an interesting clash of high tech AI and low tech paper. While I suggested a game the application could be around getting information around a subject matter. Instead of cards it could be a drop down menu. A point of this was to control output and reduce hallucinations, so, I do think there is a practical, sensible things somewhere in there.
Alex’s Idea:
I (Alex) have an aunt who bought me extremely good presents when I was a kid. One year I got an adult-sized, three dimensional cardboard skeleton model. She was a great fan of fun, interesting and educational gifts. Especially ones that involved cutting things out and gluing them together and using those little brass folding things you can use to tack to pieces of card together so that they can rotate.
The skeleton was truly amazing but my very favourite birthday present from her was a cardboard theatre. It came with sets and backdrops and props and a ton of little actors that you could poke in from the sides on sticks. There were so many little pieces to cut out and stick and then play about with. You may have guessed from my writing that I love stories. I like telling them and I like hearing them and this little theatre was a great medium for creating them.
You might recall that some time ago I built a ChatGPT bot that would allow you to pretend to be in an episode of Friends. The bot controlled Joey, Rachel, Ross, Phoebe, Chandler and Monica and would act out a scene around you.
I really want to combine the technology behind the Friends bot with my old cardboard theatre. Imagine asking a question and getting a tiny cardboard play as your answer. You might want to know about The Cold War, for example, and our little AI theatre might act out a scene from a household worried about the Cuban missile crisis. Requesting a joke could result in a Morecambe and Wise style skit instead of just a limerick from funnystuff.com. And Goodness knows what ‘expand these bullet points into an email for me’ might bring to fruition.
Do you have an idea for a whole new innovative interface to a large language model? You know what to do. 👇
You’ll note that I use they/them pronouns for Ameca throughout this article. Ameca is agender according to Engineered Arts but of the fifteen or so people in the room pretty much everyone defaulted to using ‘she/her’. It’s true that Ameca has quite a femme voice but ultimately I don’t think that the home smart speaker manufacturers, AI assistant developers or ‘future face of robotics’ engineers are doing a great job for feminism.