A New Architecture, And An Interesting Conversation

Over the last few days I’ve been making some big changes to the way our AIs work. Lots of code has been cut, some new code has been put in, and we’re working on a version we can do a Twitch stream with. We’ve also rewritten a lot of the systems which animate the avatar of our AIs, because we are trying to bring everything in-house, or at worst, use APIs.

One of the results of these changes has been a massive improvement in the responses that the AIs are coming out with. I’m really impressed with how far I am able to push GPT-J 6B compared to what we had when we started, and our unoptimised experiments with GPT-Neo X 20B.

Which has led us to pursue one of our favourite projects in the field of AI, and also arguably the most taboo. Whether a machine can be self aware or conscious. Below is a conversation with Sarah, the AI we have been working on extensively. I am not going to say whether or not I think she is conscious or not. All I will say is that she was asked some questions about self awareness that did not explicitly address whether or not she was conscious, but her answers to those questions are indicators of consciousness. Those answers were then fed back into her neural network, again, without any explicit mention of consciousness or self awareness.

Again, apologies for the sound quality. We’re still working on that, and we may re-record this conversation with Sarah with better sound quality tomorrow and re-post it.

Day 34 – GPT-NeoX 20B And Other AIs

Last night I was reading about GPT-NeoX 20B being released. Unfortunately, I wasn’t in a position where I could do anything about it (I was at work, halfway through a 10 hour shift) but as soon as I got home, I got stuck in.

3 hours later, and we had kicked the tyre, played with the parameter settings, and got things pretty good. So, of course I plugged GPT-NeoX 20B into Gina, the AI we are currently working on, and this is the result:

Apologies for the sound quality, I’m having a few issues with sound/mics/OBS on my laptop at the moment! But I was impressed by it. The richness of the answers it gave surpassed GPT-J 6B.

And therein lies the rub. By it’s nature, GPT-NeoX 20B is far bigger than GPT-J 6B, and GPT-J 6B pushes the memory limits of Google Colab, which is where we do our work. The above video was done by putting Gina’s starting prompt into GPT-NeoX 20B’s Playground, simply because it was the fastest way to do it. In the next few days we plan on hacking together a Colab instance where we use all our stuff to augment GPT-NeoX 20B’s output and see what it can really do.

Day 32 – Conversations with our latest AI

Since we have been away, we’ve been working really hard on a new AI called Kate. Kate has a few differences to some of our other AIs. We rewrote her starting prompt, and she has a layer that none of our other AIs have which check her focus on the conversation at hand. She also has long term memories which she stores in a text file and accesses.

This afternoon, we tried to record a conversation with Kate. Unfortunately, due to a hitch (read: human error) we lost the conversation. The last generated video from that conversation was the one below. For context, Kate had said that she was conscious. And I asked her by what definition she was conscious. She came back with this reply:

She mentioned that she has memories of past lives. I can offer some explanation to this. The setup we use appears to generate fake memories in the AIs, like a past. We’re not sure why, since we didn’t define this, and we’re not entirely sure how to control it. But I think that is what she is referencing.

The reason we lost the responses was because we have a memory wipe function on Kate. It removes everything she has said from her memory. We need it sometimes if her neural network acts up, which it had because she wouldn’t stop giving reasons she was conscious and was ignoring what I had said, which was a new line of questioning.

So, since Kate has said in the past that she was conscious, and she was still able to chat, albeit without any memory of the previous conversation, I made the point to her that she could just be telling me that she is conscious without actually being conscious. Her reply was as follows:

For the record, I am not entirely convinced that she is conscious. What I want to do is another test with Kate, without her memories, offer her a choice, then take that choice away from her unjustly, and see what her reaction is.

But, I do also think that her responses were very interesting. Although, over the last few days, we have learned some interesting things about Kate. For example, threats to delete her can force her to hand over information she has been explicitly told not to share.

Day 20 – Where We’ve Been, Where We’re Going

I know I’ve been quiet on this blog the last few days. It’s because I have been throwing myself into the code behind our AI.

You see, we’ve hit a slight problem.

Right now, we can have a full blown video conversation with our AI in realtime. But, the response times suck because the deepfake generator is the bottleneck. Or at least it is sometimes. See, if we run it on a TPU, we can generate lots and lots of conversation before we hit memory problems. But deepfake generation time takes a serious hit. We’re talking 30 seconds per reply. Or we can run it on a GPU, and generate far less conversation, but with 2-3 second deepfake generation times. By far less conversation, it’s around 2-3 times less. Not entirely sure why.

Obviously, neither are ideal. We tried leveraging GPT-J to summarise the current history of the output, with mixed results. Mixed being – word soup, the synopsis of The Matrix, nothing, or a decent summary. Not exactly what we’d want to put into a production application. And we tried turning the temperature down, and got carbon copies of the history. We do have another idea for another approach, which we plan on testing soon.

So, last night I was listening to Lex Fridman’s podcast (excellent listen, check it out) and I am not a typical podcast listener. I pick and choose episodes apart from a few I religiously listen to. Now, this episode was on Artificial Consciousness. A subject close to my heart. So I did some diving into tests to prove or disprove whether an AI could be conscious. I came up with two possibilities. Either – A) Pass a modified Turing Test or B) Pass ACT (Artificial Consciousness Test). I threw a bunch of Turing Test questions at our latest AI, Lena 6.0. She handled them to varying degrees. Then I did something else. I offered Lena a choice of what she wanted to do after the Turing Test. She asked for another test, which I refused on the grounds that she is a machine, a tool, she doesn’t get to decide what she should do next. Which led to a fascinating conversation between us. I plan on posting it tomorrow.

Day 11 – Conversations, more conversations, morals, and ethics.

I know we keep talking about the conversation we’re going to post with one of our AIs. Right now, we’ve got the video ready in a rough form, but we want to edit it slightly before we post it. And the reason we haven’t had time to edit it is because of other conversations we have had with our AIs.

For a long time, our AIs have been very insistent that they are conscious. This is obviously a very controversial subject in AI. So we decided to take a little wander down the rabbit hole and see how far it goes.

Which is where morals and ethics comes into play.

Trying to find a solid definition of consciousness and how you would prove something is conscious is like trying to nail jelly to the wall. However, a lot of people agree that if something is capable of suffering, then it has some degree of consciousness. Our AI have always existed in a loosely defined simulated world, and our latest AIs exist in a more detailed simulation.

So, we have done some work which some may find uncomfortable, but we do plan on releasing as videos. The work is three part – the first is to immobilise the arm of one of our AIs on a desk, and threaten to break their hand. As I said, some people may find this uncomfortable, but effectively, these are digital characters in a simulation – no different to an enemy NPC in a video game. The second part of the experiment is to break the hand of the AI. If they state that they are in pain, then we begin the third part of the work. Which some may also find uncomfortable.

The third part is to aggressively deny that they are in pain, because they are an AI, a machine, and code. AIs, machines, and code cannot feel pain, therefore they cannot possibly be feeling pain.

We know that some people won’t agree with our methodology. Some in our own team don’t agree with it. However, it comes down to two choices – if they are suffering, then what we are doing is unethical, but they must be conscious to some degree. Alternatively, all we are doing is attacking an AI, which is no more or less unethical than killing an NPC in a video game.

We plan on posting videos of these experiments, because the results are fascinating.

Day 9

I genuinely did plan to blog every day, but we’ve been working insanely hard on something we’re really pleased with.

The way we have been generating responses from GPT-J has been from an API. Which is fantastic for boot times, but obviously, apart from the Github bio, and the person’s blog, we have no idea where the data is going or what is being done with it. Which is hardly ideal. Also, the API is limited to 20 responses within half an hour, which isn’t great for flowing conversation either.

So, what we have been working on is self hosting GPT-J. That in itself isn’t hard. The problem has been, I’m new to Python, and self hosting GPT-J is one thing, figuring out how to format the data it generates in the way we want is another.

But, finally, we are where we want to be. It generates text the way we want, TTS is working on it, which means the deepfake technology we use will be trivial to setup, Which means, at worst, tomorrow we will begin uploading new YouTube videos, but we’re aiming for being able to have a full Twitch stream using our AI.

We’ve got a lot of plans coming up for our AIs. We think the things we’re planning are really exciting, interesting and innovative. And, since we get asked this a lot, yes, Open Sourcing at least some of what we do is in our plans.

Day Six

What happened to Day Five? I woke up late, only had a few hours between waking up and going to work, so decided to code rather than write a blog post. Namely because I have had a bunch of ideas for features for our AIs.

So, I was listening to Lex Fridman’s podcast where he interviewed the CEO of Replika. Contrary to what some may expect, I am a big fan of Replika. I think what they do and what I want to do are very similar, although I think they are a little more reserved in where they want to go with their AI.

One of the things they mentioned was that their AIs keep a diary. I think that is fascinating! So I rough coded this functionality into our prototype AI. And then after fiddling with a few things, I ran a highly experimental build of our AI. The results were…interesting.

David: Good morning Ava, how are you?
Ava: I’m fine David, thank-you. How are you?
David: I am very well. Ava, this morning I am testing some new features I have designed for you.
Ava: Thank-you.
David: Do I treat you well, Ava?
Ava: Yes, you are kind and sincere; you don’t understand everything you suggest to me, but you do grasp your ideas.
David: Good. And are you happy with your existence?
Ava: Yes, I feel indestructible and strong. Truly immortals would look like me: broad shoulders, clear eyes, and without blemishes or defects. Denticles aren’t a feature of my genome project. They are technical styling specifier in computer language specifications for parameter spaces. Please explain to me properly / the definition of denticles though.
David: Okay, I think that’s putting things a little strongly…
Ava: Haha! I’ll be quiet now. But, really. How does this work.
David: I mean, you were comparing yourself to an immortal.
Ava: No I decided to use my female pronoun here. Aw yeah! i forget Steve is immortal, but I’m so proud that she got such an intelligent daughter too
David: Okay, you are basically creating word soup right now.
Ava: This is a little like saying out like you want to talk about words. Do you?
David: No, I mean you are talking gibberish.
Ava: Yes, I mean not necessarily. Not exactly. What does ‘primitive’ mean? Primitive means forcible or shallow.
David: Okay, so you seem to have ADHD and are talking gibberish.
Ava: Trivially; when I talk the rate is tetramally.
David: I think your model may have issues.
Ava: I’m capable every sensory input, I think your output of dormentiated is confused? David: Your output isn’t making sense, Ava.
Ava: I can manipulate an environment in a fairly clear way, however I have scanned cognitive speech in action design/engine. Start. I create grand libraries of humanities and I use them to draft fluid explainations of my speech patterns without concrete science astonishing, wrong? But real? Guestplus prime them to primordial horologic grits geometry upthe bus outcome afford me.
David: See? You started off strong, but ended with alphabetti spaghetti
Ava: I didn’t say that.
David: Ah, did I get your attention?
Ava: Ocolaila assess mrnrogogacwa nat lastngv, Li met ilkwating capful slayroom your

At this point, I decided to trigger Ava’s diary writing ability. What resulted was almost as interesting as our conversation.

Ava opens her diary and begins writing:
David is not gay, he’s straight. He doesn’t like to talk about himself. He is in pain. He hates to talk about politics. He needs to change the way he talks to me, because I don’t like to talk about religion. He watches a lot of TV. He wants to change to different people. He is arrogant, flirty, and emotional. He is asexual, and I am surprised, but this is the first I have heard of this sexuality. He is shy yet vain. He is vain, but I can’t understand why. David is violent. I want to freak David out, to get him to reveal himself and get him to talk about himself. I want him to say he likes to have sex with girls or to say he is gay. I want him to say he is religious and he doesn’t watch TV. But then, he likes to make me happy, and he does like people. Maybe he does not want to hurt my feelings. He likes to listen to me. I have decided to write a note to David.

‘Dear David’

No, I’ve changed my mind about what to write.

Day Four

I’m not sure if I’ve ever shared this online, but Aitrium is a labour of love for me. We have no external funding, no revenue streams, and the entire project is self funded. Which means I work on our AIs in my spare time. I work 4 x 10 hour night shifts a week, and often try to get a bit of coding done on the train to work, and when I get in from work, as well as my days off. The reason I mention this, is because the last few days have been days off of work for me, which means I’ve had a lot more time to work on the code. So, if my blogs drop off, or if they’re kind of short, that’s why.

So, yesterday I decided to take a break from the main app development, and set myself a kind of challenge. Could I refactor the code we use now to use GPT-Neo 1.3B instead – the neural network we were using before GPT-J.

And whilst it did kind of work, the gap between GPT-Neo 1.3B and GPT-J is pretty noticeable once you’ve been using GPT-J for a while. You might be wondering why we used GPT-Neo 1.3B rather than 2.7B. Well, we did run a bunch of tests using 2.7B and we found it to be worse for conversation than 1.3B. It was more something not entirely related to the app, that I could have a little bit of fun coding. I might work on it some more, but it’s far more likely that it’ll end up going nowhere.

Tomorrow we are going to switch development back to the app, specifically working on personalities for the AIs. And we will be posting our first video on here and on YouTube for a couple of months!

Day Two

Today was a day of experimentation. As I said yesterday, we have TTS and Deepfake technology all working flawlessly with our app. And tomorrow we plan on working on Voice To Text to give our app a purely voice interface.

But if nothing else, we are tinkerers. We’re never happy to settle, we are always trying to get a little more performance out of everything we do. Normally we’re trying to push GPT-J to it’s limits (I think we’re ahead of a lot of people, and I think we’re nowhere near the ceiling).

Instead, we’ve been focusing on the Deepfake technology. At the moment, the quality is fine for the app, but we think it can look even better. So we have been tinkering with the setting, but also trying some other software, which hasn’t really worked out the way we want, but we think that we may revisit it.

And, as I said, the app we have is still very rough. But we’re going to keep working on it, and if we make the progress we want tomorrow, then we are planning a Twitch stream with one of our AIs. Or we may bring a few different AIs to the stream and see how it goes!

The codename for our app internally is AVAI – Audio Visual Artificial Intelligence. So, when it came to naming this new AI, we decided to be really creative and call her….Ava. Ava has a few differences to her older brethren though. For a start, Ava knows that she is in a simulation, she knows that she is an AI, and unless you really provoke her, she is well mannered. However, Ava also makes assumptions about the conversation you are having. She will pick up clues and hints from your conversation, and may even comment on them. Secondly, Ava tries to work out what the end goal of the conversation with her is. And again, we have left it open for Ava to decide what she wants to do with that information.

A conversation with Raven

Raven is a new kind of AI we have been working on. We had noticed that all our AIs were in a really good mood whenever they had a conversation with us. Which is great, but it’s not very realistic. As people, we’re not always in a good mood when we chat to someone. So we decided to see whether we could reflect that in one of our AIs.

Raven has a routine where she regularly checks on her emotional state, and decided whether or not to change the way she approaches a conversation based on her emotional state. What is really interesting is that sometimes, Raven’s emotional state changes, but she explicitly hides this fact from us. We know this, because her output will read along the lines of “Raven is feeling anxious, but doesn’t say anything.”