Instructions
1. Tutorial: Build and Improve Counter

1. Tutorial: Build and Improve Counter

This tutorial will demonstrate how to use ReactGenie to build a simple Counter app. You don't need to write code in this part, just follow the instructions to understand how to use ReactGenie to build a multimodal app.

1.1 ReactGenie Overview

We'll start with a brief introduction to the ReactGenie framework to help you understand how to build multimodal applications using ReactGenie.

ReactGenie is a framework help developers to build a multimodal app. ReactGenie will automatically parse users' natural language queries to function calls based on LLM, execute the command and then display the result. As developers, you only need to give some descriptions and examples to help ReactGenie handle user's query.

Each multimodal app built with ReactGenie consists of three parts:

  1. State Code: This is used to control and manage the state, such as data and executing functions. It is written in the genie folder.

  2. UI Code: This is used to display the state data and allow users to interact with the data via the UI. It is written in the src folder.

  3. Link UI to State code: This involves linking the State to UI so ReactGenie knows using which UI to render which type of results. You will need to modify App.tsx and store.ts.

For example, in the upcoming Counter app tutorial, the state code includes the Counter class (Counter.ts) for counter state management, and the UI code includes views of the counter input form and the interface for added counters (e.g., EditCounterView.tsx). The structure of the Counter app is as follows:

- genie
   - //state code
   - Counter.ts
- src
   - //UI code
   - EditCounterView.tsx
   - CounterItemView.tsx
   - CounterListView.tsx
   - MainView.tsx
-App.tsx
-store.ts

We'll provide more detailed instructions in the later sections. You can also refer to the ReactGenie documentation for additional guidance.

In the following section, we'll demonstrate how to implement a simple Counter App with ReactGenie. Due to our focus on the functionalities supported by the application in this experiment, we will only cover state code related details. In this experiments, you will only need to modify the state code of the app and not be concerned with other parts.

1.2 Environment Check

Open WebStorm, we have prepared a starter project for you called Counter

The tutorial below will guide how to construct a new Counter using this starter project. To run the starter ReactGenie project, you can use the following commands:

npm install 
npx expo start --web

Note that the ReactGenie framework provides a microphone button on the bottom right of the webpage. Users can click this button to utilize the React Genie voice interface when they would like to. Users can also continue to use the graphical user interface (GUI) as they please.

1.3 State Code

1.3.1 Create state classes

ReactGenie provides you with two inbuilt classes that you can extend from for your state classes.

  1. DataClass: A DataClass is the basic unit for data management, and an instance of a DataClass can be viewed as a record in a database. The property in a DataClass can be a number, string, boolean, list, DataClass, or a HelperClass. Read more in the docs (opens in a new tab).
  2. HelperClass: The HelperClass stores and manages complex data types. They will be used to define complex properties in a Dataclass, so that they can be serialized and deserialized. Read more in the docs (opens in a new tab).

Note: You don't need to create any HelperClass in the Counter app.

Let's get started with implementing a Counter Dataclass to store and manage our Counters!

Initialize the file Counter.ts under the genie folder:

-genie
    -Counter.ts

To create a DataClass, you need to define all the properties of the DataClass in primitive types (e.g., string, int, boolean), arrays of primitive types, compound types using HelperClass, or arrays of compount types.

Counter.ts
import { GenieClass, DataClass, int } from "reactgenie-lib";
 
export class Counter extends DataClass{
    public name: string;
    public type: string;
    public count: int;
    public created: boolean;
}

Now we're introducing a key concept in ReactGenie, the ReactGenie decorators. ReactGenie can transform users' natural language queries to function calls. To achieve this, you need to add a ReactGenie decorator for the classes, functions, and properties that you want the users to be able to access with multimodal interactions (voice + touch). In the decorator, you can optionally specify a string to describe the purpose of the class, function, or property. The description you provided in the decorator will be passed to LLM to help ReactGenie understand it.

Counter.ts
import { GenieClass, DataClass, GenieProperty, int } from "reactgenie-lib";
 
@GenieClass("A counter example")
export class Counter extends DataClass{
    @GenieProperty("The name of object that you want to count")
    public name: string;
    @GenieProperty("The type of objects (vegetable, fruit, ...)")
    public type: string;
    @GenieProperty() // You can omit the description
    public count: int;
    @GenieProperty() // You can omit the description
    public created: boolean;
}

This class is still missing one important piece. Each DataClass requires a key to help ReactGenie manage the classes. The key has to be unique for each instance of the class. In ReactGenie, this is done through annotating the key property with a @GenieKey decorator. Let's add this to our class:

Counter.ts
import { GenieClass, DataClass, GenieKey, GenieProperty, int } from "reactgenie-lib";
 
@GenieClass("A counter example")
export class Counter extends DataClass{
    @GenieKey
    public id: string;
 
    @GenieProperty("The name of object that you want to count")
    public name: string;
    @GenieProperty("The type of objects (vegetable, fruit, ...)")
    public type: string;
    @GenieProperty() // You can omit the description
    public count: int;
    @GenieProperty() // You can omit the description
    public created: boolean;
}

To recap, a state class (DataClass) in ReactGenie requires three types of decorators:

  • @GenieClass("{description of the class}") that extends DataClass
  • @GenieProperty("{description of the property}"): properties that you want to expose to ReactGenie
  • @GenieKey: a unique key for each instance of the class

1.3.2 State class management

Now that we've defined our properties, we need to initialize these properties using a constructor. This is required for all the properties.

Counter.ts
import { GenieClass, DataClass, GenieKey, GenieProperty, int } from "reactgenie-lib";
 
@GenieClass("A counter example")
export class Counter extends DataClass{
    @GenieKey
    public id:string;
 
    @GenieProperty("The name of object that you want to count")
    public name: string;
    @GenieProperty("The type of objects (vegetable, fruit, ...)")
    public type: string;
    @GenieProperty()
    public count: int;
    @GenieProperty()
    public created: boolean;
 
    constructor({id, name, type, count = 0, created}: {id: string, name: string, type: string, count?: int, created:boolean}) {
        super({})
        this.id = id;
        this.name = name;
        this.type = type;
        this.count = count
        this.created = created;
    }
}

NOTE You should AVOID directly calling the constructor like below:

const counter = new Counter({id: "1", name: "apple", type: "fruit", count: 0, created: true})

Objects created like this will not be automatically tracked by ReactGenie. Instead, ReactGenie provides the three static functions below to help you manage the object, CreateObject(), GetObject(), DeleteObject(), All().

CreateObject() will call the constructor of the class, so the parameters of CreateObject() is the same as constructor. GetObject() allows you to access a object through its GenieKey. All() allows you to get an array of all instances of a specific class. DeleteObject() allows you to delete a object through its GenieKey.

Please use these functions to create and access the objects so that ReactGenie can track the objects. For example:

const appleCounter = Counter.CreateObject({id: "1", name: "apple", type: "fruit", count: 0, created:true})
const getAppleCounter = Counter.GetObject({id: "1"}) // getAppleCounter === appleCounter
const allCounters = Counter.All() // allCounters === [appleCounter]
Counter.DeleteObject({id: "1"})
const allCountersAfterDelete = Counter.All() // allCountersAfterDelete === []

Note that only All() is automatically exposed to multimodal interactions. If you want to allow users to access other functions to handle the objects, you need to create a function and add a ReactGenie decorator for it.

For example, if we want to allow users to say "Create an apple counter", we need to create a function like:

Counter.ts
import { GenieClass, DataClass, GenieKey, GenieProperty, GenieFunction, int } from "reactgenie-lib";
 
@GenieClass("A counter example")
export class Counter extends DataClass{
    //...
    //...
 
    @GenieFunction()
    static CreateCounter({name, type, count=0, created}: 
            {name:string, type:string, count?:int, created:boolean}): Counter {
        const id = Counter.All().length.toString(); // automatically generate a unique id
        return Counter.CreateObject({id: id, name: name, type: type, count: count, created: created})
    }
}

1.3.3 Setup placeholder objects

ReactGenie provides you with class initialization settings, where you can set static variables for the class or create placeholder objects. The setup() function is a static function of the GenieClass. You can override the setup function for initial program setup and can also make it empty. ReactGenie will automatically call the function to complete the initialization.

For counter, we can set up three counters in the initialization.

Counter.ts
@GenieClass("A counter example")
export class Counter extends DataClass{
    //...
    static setup() {
        Counter.CreateCounter({name:"apple", type:"fruit",created:true});
        Counter.CreateCounter({name:"orange", type:"fruit", created:true});
        Counter.CreateCounter({name:"potato", type:"vegetable", created:true});
    }
}

1.3.4 Add you own functions

Next, you can create more functionalities for your class. For example, you can create functions to increment, decrement, and delete the counter. Don't forget to add ReactGenie decorators as these functions should be accessible to users via multimodal interactions.

Counter.ts
@GenieClass("A counter example")
export class Counter extends DataClass{
    //...
 
    @GenieFunction()
    increment(): void {
        this.count = this.count + 1;
    }
 
    @GenieFunction()
    decrement(): void {
        this.count -= 1;
    }
 
    @GenieFunction()
    delete(): void {
        Counter.DeleteObject({id: this.id});
    }
}

Of course, you can also create functions to set names, types, etc., to support more features. Here we get an example state code for Counter:

Counter.ts
import {GenieClass, DataClass, GenieKey, GenieProperty, int, GenieFunction} from "reactgenie-lib";
 
@GenieClass("A counter example")
export class Counter extends DataClass{
    @GenieKey
    public id: string;
 
    @GenieProperty("The name of object that you want to count")
    public name: string;
    @GenieProperty("The type of objects (vegetable, fruit, ...)")
    public type: string;
    @GenieProperty()
    public count: int;
    @GenieProperty()
    public created: boolean;
 
    constructor({id, name, type, count = 0, created}:
                    {id: string, name: string, type: string, count?: int, created: boolean}) {
        super({})
        this.id = id;
        this.name = name;
        this.type = type;
        this.count = count
        this.created = created;
    }
 
    @GenieFunction()
    static CreateCounter({name, type, count=0, created}:
                             {name: string, type: string, count?: int, created: boolean}): Counter {
        const id = Counter.All().length.toString();
        return Counter.CreateObject({id: id, name: name, type: type, count: count, created: created})
    }
 
    static setup() {
        Counter.CreateCounter({name: "apple", type: "fruit", created: true});
        Counter.CreateCounter({name: "orange", type: "fruit", created: true});
        Counter.CreateCounter({name: "potato", type: "vegetable", created: true});
    }
 
    @GenieFunction()
    increment(): void {
        this.count = this.count + 1;
    }
 
    @GenieFunction()
    decrement(): void {
        this.count -= 1;
    }
 
    @GenieFunction()
    delete(): void {
        Counter.DeleteObject({id: this.id});
    }
}

1.4 Design UI Code and Link State Code to UI

Due to the irrelevant nature of this part to the current experiment, we will skip this section. If you’re interested, you can visit the ReactGenie documentation (https://react-genie-developer-study.vercel.app (opens in a new tab)) after the experiment to learn more.

1.5 Add Examples and Test

Finally, we need to add usage instances for your program to help ReactGenie correctly generate executable results using LLM. We can provide examples of relevant function usage in each class. You don't need to provide examples today, but you will learn some features about ReactGenie in this part.

Examples is a static list in GenieClass.

Counter.ts
export class Counter extends GenieClass {
    //...
    static Examples = [
    {
        user_utterance: "Increse this counter",
        example_parsed: "Counter.Current().increment()",
    },
    {
        user_utterance: "show me all vegetables counters",
        example_parsed: 'Counter.All().matching(field: .type, value: "vegetable")',
    },
    {
        user_utterance: "increase me all fruit counters",
        example_parsed: 'Counter.All().matching(field: .type, value: "fruit").increment()',
    },
    {
        user_utterance: "Delete this counter",
        example_parsed: 'Counter.Current().delete()',
    }]
}

To help users achieve more interactions, ReactGenie supports some filtering operations on arrays.

matching(field, value), equal(field, value)

between(field, from, to), sort(field, ascending) Note that the field passed to sort() and between() should have a static compare() function.

contains(field, value) The contains() returns the member whose field value contains the given value.

All this filtering operations return a array.

For example:

{
    user_utterance: "show me all vegetables counters",
    example_parsed: 'Counter.All().matching(field: .type, value: "vegetable")',
},
{
    user_utterance: "show me all counters less than five",
    example_parsed: 'Counter.All().between(field: .count, from: 0, to: 5)',
}

ReactGenie also supports some querying operations on arrays. ReactGenie allows developers to access the properties and functions of list members during execution. This can be used directly to access the properties of the list members.

{
    user_utterance: "Show me all counters type",
    example_parsed: 'Counter.All().type',
  },
  {
    user_utterance: "Delete all fruit counters",
    example_parsed: 'Counter.All().matching(field: .type, value: "fruit").DeleteCounter()',
  },

Another function is built by ReactGenie to provide rich multimodal interactions is Current(). Current() is automatically created for all DataClasses, and it returns the current object that is being interacted with.

We have completed a simple multimodal interactive application based on ReactGenie. Congratulations!

1.6 Improve Counter

Now that you've built a simple Counter app, but it's functionalities are still limited. For example, you can only create a counter with a specific name and type, and you can only increment and decrement the counter by 1.

Imagine you are a user of this app, you might want to increment or decrement the counter by a specific number. Clearly, the current app is not enough to support this feature.

So we can add functions to support this feature.

Counter.ts
...
 
@GenieFunction()
incrementBy({number}: {number: int}): void {
    this.count = this.count + number;
}
 
@GenieFunction()
decrementBy({number}: {number: int}): void {
    this.count -= number;
}
 
...

Besides this feature, user might have other features that they want to use. So you need to think about the demands of user and add more functions or properties to support more features. And that is what we will do in the following experiment.

Considering the time limit and the development cost of the possible features, you don't need to implement the functions. You can only write some placeholders for the functions. For example, you can write an empty function to support the feature of incrementing the counter by a specific number.

Counter.ts
@GenieFunction()
incrementBy({number}: {number: int}): void {
    //to be implemented
}

You only need to focus on what kind of features you want to support instead of the implementation details. Let move on to the next part to start the experiment.

Instructions2. Your turn: Target Apps