Experiential Design: Task 3 Prototype

Experiential Design Task 3

Eins Herbert Takawijaya Tee

0365993

Bachelor Of Computer Science (Honours)

------------------------------------------------------------

In this task, we were assigned on making a prototype for our AR application in which i have choosen to make an application called PocketPal which is an AR, AI Integrated pet simulator made in Unity Engine.

Part 1 - Speech To Text

In the entire project by far, Speech to text is one of the most challenging feature to implement due to the difficulty in finding the right unity asset that can implement speech to text flawlessly in unity and android.

In the time where i developed pocket pal, i spend a few days deciding and finding new package that can work in unity, After a swapping and testing few packages since most of them has issues, i found an easy speech to text packages called Unity Android Speech To Text by yasirkula on github (https://github.com/yasirkula/UnitySpeechToText). This is the only unity packages that work flawlessly on android and run without bugs and compilation error.

With that, i create the speech manager script that handles all the speech related features in this app. On that script i connected it to a button that detects what you speak and updates the text above based on what you said

1.1 Button to start Speech To Text

Part 2 - AR Implementation

For the AR, i am planning to create and rig my own model, but since in this prototype i dont really need a fully animated model, im just using a free dog model i find on the asset store to use as a place holder to showcase the AR Features, so the AR provide a few features such as:

• On the main screen, if ground is detected and screen is tapped, an AR dog model will appear and have a text box appear above it, the textbox will change if speech to text is initiated and got a response from OpenAI API.
• The AR model and dialogue box will always face the camera using the script to look at the camera when the camera moves.
• There will always be one model, if you tried to create another pal by tapping the screen, the old pal will disappear and a new one will spawn on the new location.

2.1 Ground Detected with AR Camera

2.2 When screen is pressed

Part 3 - AI Manager

After implementing the speech to text, i then proceed working on the AI Manager which is connected to a OpenAI API, In this script there is a lot of features, such as:

• Updating the personality of the Pal depending on the hunger and happiness stat.
• Adding Memory to the AI so previous conversations will be remembered up to 20 conversations.
• Requesting Chat To ChatGPT API which is the main way of conversation.
• Slowly types out of the output from ChatGPT to give the illusion of slowly answering connecting to a Text in the Pet. 

3.1 Pal responds when asked "how are you" with 0 Happiness point and 0 Hunger Point

3.2 Pal responds when asked "how are you" with max Happiness point and Hunger point

3.3 Pal when asked to give a simple math question

3.4 Pal reaction when i give the correct answer

Part 4 - Asset Creation

For the asset of this application, i use pixel art style for the UI since it is simple for 2d and fits with 3d low poly object, I used Aseprite for creating 2D pixel art and uses Autodesk Maya for 3D pixel art.

4.1 Icon Pixel Art

4.2 Stat Bar and Mic button Pixel Art

4.3 Food 3D Model

For the font, i used Comic Neue Sans since its a fun font and since the main target audience is younger people such as children's and teenagers, i think this font will be a perfect fit.

4.4 Font Example

Part 5 - MainUI & Functions

When you opened the app, there is a few UI that i added, such as:

1. Happiness Bar on the Top Left.
2. Hunger Bar on the Top Right
3. Food Ticket count on Top Middle
4. Food Shop on Left Side
5. Feeding Mode & Play Mode on Right Side
6. Microphone Button On the Bottom

5.0.1 Main UI

5.1 Happiness

The happiness feature of the pal is determined on how much did you engage with the pal, you can get happiness by playing with the pal through entering the play mode by pressing the happy icon button on the right side, when you enter the play mode, there will be nothing in the UI but you can click on the Pal and it will create a Raycast from the touch location and direction. If the Raycast hits the pal, it will add the happiness point and the pal will reply and react to it, the reaction will be custom since its sent to OpenAI API and will generate a reply depending on current status of happiness bar and hunger bar.

To exit the play mode, just press on the happy icon again and it will return to the screen with all button

5.1.1 Pal Reacting to playing

5.2 Food Ticket & Shop

So by now you must have notice the food ticket and wonder what it might be for, It is used to buy food in the shop, you can gain this ticket by doing activities, different activities has different chance for dropping the food ticket such as for this example, playing with your Pal has a 100% chance for dropping the ticket and having a conversation with your pal will only have 30% chance for dropping the ticket.

If you press the Shop Icon on the left, a UI will appear showing the shop that sells food and you can buy with the ticket, the price is shown and you can click it to make a purchase. If you made a purchase the ticket will be deducted depending on the price and the food will be added to your inventory. Like before to return, press on the shop icon again.

5.2.1 Food Shop & Ticket as currency

5.3 Feeding

Once you bought the food with your ticket, you can press on the food icon on the right side to open the feeding UI, If you have bought some food with the ticket before, you will see the quantity of food you have bought below the food icon.

5.3.1 Feeding UI & Quantity Shown

If you press any of the food, a 3D model of the food will appear, the number of food you have will be deducted and refunded if you press on other food or exit the feeding mode. Now with the food appearing, you can move your phone closer to the Pal and move the phone to the Pal mouth, that way the pal will eat the food and the hunger bar will increase depends on which food did you feed.

How it works is the model have a collider specifically for the mouth and the food also has a collider which is a trigger and when they collide, the food will destroy itself and Hunger points will be added.

5.3.2 Feeding Medium Food

Part 6 - Future Improvements & Changes i made from initial idea

There's a few changes i made from my initial plan which is:

• Didn't add real time hand tracking due to it being not compatible with Vuforia, Especially on android.
• Due to the changes above, instead of detecting hand tracking to play with pal, i change to clicking the pal on a phone screen instead where then it gave a response.
• Changed the backup idea (Throwing the food) to feeding by moving the phone since i feel that this way made a little more sense for feeding.
• Text to speech may be changed to other stylized sounds if no unity package with good TTS is found.

There is a lot of improvement that i think i can add for the final product, since this is the prototype, i didn't add most of them since i was too focused on getting the main feature implemented. The things that can be implement are:

• Polishing the feeding and playing by adding custom 3d particles to show if i successfully feed it something or play with it.
• Create a custom 3D model or find a good one with rigging since this current one is just a place holder for example with no custom idle animations.
• Adding a tutorial since this current one is pretty hard to understand on how to use without instructions.
• Add a main menu.
• Add a settings so you can turn on tutorial or off (Default ON), Add a developer mode to turn on or off Debug, and Turn volume up or down if there is background music or pal voices, and finally settings to enable or disable The words that you said.
• Turn all the other icon off if there is no pal spawned.