Data transformation

Data preprocessing and transformation is an important step in AI assistant design. When working with raw or semi-structured data, it may be necessary to pass this data through a series of transforms to prepare it for interactions. For example, you may want to modify input data, format it in a proper way and enhance it before it can be displayed in the Alan AI Chat or used in the dialog logic.

Alan AI supports the following formats for data transformations:

• Input format: plain text, markdown-formatted text, HTML, JSON and JavaScript

• Output format: plain text, markdown-formatted text, HTML, JSON and JavaScript

To apply transformations to the input data, you need to perform the following steps:

1. Add transforms to your AI assistant project

2. Use transforms in the script

Adding transforms

In Alan AI, transforms are templates or predefined rules that must be applied to the input data to transform it to the desired output. Transforms describe:

1. Type and structure of the input data

2. Prompt to Alan AI on how this data must be transformed

3. Format of the output data

To add a transform to your AI assistant project:

1. In the left pane of Alan AI Studio, under Transforms, click Add and enter the name for the transform.

2. In the Instruction field, enter a transform prompt. The transform instruction is a short sentence that provides specific instructions on how Alan AI should interpret the transform rules.

   Note

   The transform instruction is generic and applied to all examples you provide in the transform. You can extend this generic instruction with additional prompts added to each example.

   

3. In the Examples section, add one or more examples showcasing how the input data must be transformed and choose the required input and output data formats.

   To edit transforms data in a convenient way, in the top left corner of any text field, click the magnifying glass icon. You can use the following options:

   • [For HTML and JSON data] To apply formatting rules and improve data visual structure, at the top of the text field, click Format HTML/Format JSON.

   • To edit data and code, in the top right corner of the text field, click Editor mode.

   • To preview the output of your data and visualize how it will appear, in the top right corner of the text field, click Split mode or Preview mode.

   • To expand and collapse specific text fields, in the top left corner of corresponding fields, click the arrow icon.

   

   The rules provided in the example may vary depending on the input data and expected output. For example, to convert input JSON data to markdown-formatted text, you may add the following example:

   • Input: chunk of JSON-formatted input data

   • Instruction: prompt to Alan AI on how to interpret specific fields in the input JSON data

   • Result: example of output text formatted in markdown

4. In the left pane of Alan AI Studio, click Save to save the transform.

   

Using transforms

To apply the created transform for data preprocessing, use the transforms() function in the dialog script.

Dialog script

intent("Show user details", async p => {
    let u = await transforms.format({
        input: {"name": "John Smith", "age": "56", "address": "1234 Main Street"},
        query: "Name is the user name, age is the user age, address is the user address"
    });
    p.play(u);
});

Here, the transforms() function takes the following parameters:

• format: name of a transform to be applied to transform the input data

• input: input data to be transformed

• query: instructions to Alan AI on how to interpret the input data or sample queries to be run