We have supported vision input for the Planner role in TaskWeaver. The Planner role is responsible for generating the high-level plan for the task. The vision input is a new type of input that contains images. This feature is useful when the task requires visual understanding.

How vision input is supported in TaskWeaver​

In TaskWeaver, we added a new role called ImageReader to read images and provide the image url (for remote images) or the image encoded in base64 (for local images) to the Planner role. To have this new role, you need to include it in the project configure file as follows:

{
    "session.roles": [
        "planner",
        "code_interpreter",
        "image_reader"
    ]
}

The ImageReader role takes the path or the url of the image as input and prepares a response Post for the Planner role. As described here for Azure OpenAI API, if the image is local, ImageReader need to encode the image in base64 and pass it to the API. If the image is remote, ImageReader need to provide the url of the image. The Planner role can then use the image information for various tasks.

An example​

Let's ask the agent to describe any uploaded image.

The flow of the conversation is as follows:

In the example above, the User talks to the agent in Web UI and uploads an image. TaskWeaver also support providing the image path in console mode, either using the /load command or just include the image path in the input message.

Extension​

If you look into the implementation of the ImageReader role, you will find that it is quite simple. The key logic is shown in the following code snippet:

if image_url.startswith("http"):
    image_content = image_url
    attachment_message = f"Image from {image_url}."
else:
    if os.path.isabs(image_url):
        image_content = local_image_to_data_url(image_url)
    else:
        image_content = local_image_to_data_url(os.path.join(self.session_metadata.execution_cwd, image_url))
    attachment_message = f"Image from {image_url} encoded as a Base64 data URL."

post_proxy.update_attachment(
    message=attachment_message,
    type=AttachmentType.image_url,
    extra={"image_url": image_content},
    is_end=True,
)

After the image url is obtained, the ImageReader role will encode the image in base64 if the image is local. Then, it will create an attachment in the response Post and pass the image content to the Planner role. To achieve this, the attachment is created with the type AttachmentType.image_url and the image content is passed as extra data with the key image_url.

Therefore, if we want to support other scenarios with vision input, we can extend the ImageReader role by adding more logic to handle different types of contents. One example is to support reading a document with text and images. We can add an attachment for each image in the document and pass the list of attachments to the Planner role.

There are many techniques developed to build an agent reasoning framework. For example, Chain-of-Thought (CoT) is one of the most popular reasoning approaches that uses a chain of thought to generate next-step actions. Another technique is ReAct, where the agent reflect on its action results to adjust its future actions. In TaskWeaver, one of the key reasoning technique is task decomposition and tracking, where the agent breaks down a task into subtasks and keeps track of the progress of each subtask. The following snippet shows an example of task decomposition and tracking in TaskWeaver:

├─► [init_plan]
│   1. Request historical stock price data for MSFT from the user
│   2. Request the forecasting model or method to be used for prediction <interactively depends on 1>
│   3. Perform the prediction using the provided data and model <interactively depends on 2>
│   4. Report the prediction results to the user <interactively depends on 3>
├─► [plan]
│   1. Request historical stock price data for MSFT from the user
│   2. Request the forecasting model or method to be used for prediction
│   3. Perform the prediction using the provided data and model
│   4. Report the prediction results to the user
├─► [current_plan_step] 1. Request historical stock price data for MSFT from the user

The agent lists the steps of the plan and the current step it is working on. While the agent is working on the task, it keeps track of the current step. This is useful especially when the task is complex and requires multiple steps to complete. The agent can refer to the current step to know what to do next. Without task decomposition and tracking, the agent may lose track of the task and fail to complete it.

TaskWeaver also support ReAct-like reasoning. This is done by introducing multiple roles in the problem-solving process, especially the Planner and CodeInterpreter roles. The Planner role is responsible for planning the steps of the task, while the CodeInterpreter role is responsible for interpreting the code generated by the Planner. The Planner can reflect on the results of the CodeInterpreter and adjust the plan accordingly. In one round, the Planner and Code Interpreter can work together to complete a subtask with multiple interactions.

However, there are still situations where the agent reasoning framework can be improved. If you have read the discussion above, you may have noticed that CoT is not implemented in TaskWeaver.

Why?

In this article, we are focused on the reasoning in the Planner. In fact, the Code Interpreter has CoT implemented as shown in the following snippet:

├─► [thought] To fetch historical stock price data for MSFT using yfinance, I will first try to import yfinance and download the data. If yfinance is not installed, I will provide instructions to install it.
├─► [reply_type] python
├─► [reply_content]
│   import yfinance as yf
│
│   # Fetch historical stock price data for MSFT
│   ticker = 'MSFT'
│   msft_data = yf.download(ticker, start='2020-01-01', end='2025-01-01')
│
│   msft_data

For the Planner, there are actually two places in the response schema where CoT can be implemented: before generating the init_plan and after the current_plan_step, as shown in the previous snippet. The former is to generate the initial plan, while the latter is to use Planner's reasoning capabilities to solve the current subtask.

The former was not implemented in TaskWeaver for two reasons: (1) to simplify the schema and save the reasoning time/prompt size; (2) the existing init_plan and plan already represent a thought process, and adding CoT may not bring significant benefits. But as the LLM becomes faster and cheaper, it is possible to add CoT to the Planner to improve the planning quality.

With reasoning for planning, the output of the Planner looks like this:

 Human   ▶  predict msft's stock price in the next 7 days
 ╭───< Planner >
 ├─► [plan_reasoning] The user wants to predict Microsoft's stock price for the next 7 days. This task involves time series forecasting, which requires historical stock price data and a forecasting model. However, the user hasn't provided any data or specified a model to use. I need to ask the user for more information.
 ├─► [init_plan]
 │   1. Request historical stock price data from the user
 │   2. Ask the user to specify a forecasting model or confirm if a default model should be used
 │   3. Load the historical stock price data <interactively depends on 1>
 │   4. Train the forecasting model using the historical data <interactively depends on 2 and 3>
 │   5. Predict the stock price for the next 7 days <sequentially depends on 4>
 │   6. Report the prediction results to the user <interactively depends on 5>

It looks good, but it is not the most important part of this article. The latter is more interesting and also more challenging to implement than it looks.

Let's use an example to motivate the necessity and explain the challenges. The task is described as follows and all the data in it are fake:

# Task Instructions  
## Step 1: Extract Employee Data  
Extract the following information from the employee summary:  
- `Employee ID`  
- `Department`  
- `Project Code`  
- `Role`  
- `Manager`  
- `Location`  

### Employee Summary  
On April 10, 2025, a review was conducted for an employee with ID `E12345`. 
The employee is part of the `Development` department and is currently assigned to project code `"X987"`. 
The role held is `"Software Engineer"` under the supervision of manager `"J. Doe."` 
The primary work location is `"Remote."`  

## Step 2: Validate the Role  
- Confirm if the role is recognized among the following: Software Engineer, Data Analyst, Project Manager, DevOps Specialist, QA Tester.  
- Check if the role is one of the recognized roles.  

## Step 3: Construct SQL Query  
Prepare and print the SQL query using the extracted details:  
===
SELECT EmployeeID, Department, ProjectCode, Role, Manager, Location  
FROM EmployeeRecords  
WHERE EmployeeID = '{EmployeeID}'  
AND Department = '{Department}'  
AND ProjectCode = '{ProjectCode}'  
AND Role = '{Role}'  
AND Manager = '{Manager}'  
AND Location = '{Location}'  
ORDER BY EmployeeID;  
=== 

The task has 3 steps: extract employee data, validate the role, and construct an SQL query. This would not be a hard problem for any ReAct based agents. Actually, for all the 3 steps, there is no need to call Code Interpreter. The Planner can reason through the task and generate the code directly. However, let's run it in TaskWeaver:

 ╭───< Planner >
 ├─► [plan_reasoning] The user has provided a task with specific instructions to be followed step by step. The task involves extracting employee data, validating the role, and constructing an SQL query. Each step is dependent on
 │    the completion of the previous one, so they must be executed in sequence.
 ├─► [init_plan]
 │   1. Extract the specified employee data from the summary provided.
 │   2. Validate the role against the recognized roles list <sequentially depends on 1>
 │   3. Construct the SQL query using the extracted details <sequentially depends on 2>
 │   4. Report the constructed SQL query to the user <interactively depends on 3>
 ├─► [plan]
 │   1. Extract the specified employee data from the summary provided.
 │   2. Validate the role against the recognized roles list.
 │   3. Construct the SQL query using the extracted details.
 │   4. Report the constructed SQL query to the user.
 ├─► [current_plan_step] 1. Extract the specified employee data from the summary provided.
 ├─► [stop] InProcess
 ├──● Please extract the following information from the employee summary: Employee ID, Department, Project Code, Role, Manager, and Location. The summary is: On April 10, 2025, a review was conducted for an employee with ID     
 │   `E12345`. The employee is part of the `Development` department and is currently assigned to project code `"X987"`. The role held is `"Software Engineer"` under the supervision of manager `"J. Doe."` The primary work        
 │   location is `"Remote."`
 ├─► [shared_memory_entry] Add the plan to the shared memory
 ╰──● sending message to CodeInterpreter
 ╭───< CodeInterpreter >
 ├─► [thought] I will extract the required information from the given employee summary using string manipulation techniques.
 ├─► [reply_type] python
 ├─► [reply_content]
 │   employee_summary = "On April 10, 2025, a review was conducted for an employee with ID `E12345`. The employee is part of the `Development` department and is currently assigned to project code \"X987\". The role held is      
 │   \"Software Engineer\" under the supervision of manager \"J. Doe.\" The primary work location is \"Remote.\""
 │
 │   # Extracting information
 │   employee_id = employee_summary.split('ID `')[1].split('`')[0]
 │   department = employee_summary.split('`')[3]
 │   project_code = employee_summary.split('project code \"')[1].split('\"')[0]
 │   role = employee_summary.split('role held is \"')[1].split('\"')[0]
 │   manager = employee_summary.split('manager \"')[1].split('\"')[0]
 │   location = employee_summary.split('location is \"')[1].split('\"')[0]
 │
 │   employee_id, department, project_code, role, manager, location
 ├─► [verification] NONE
 ├─► [code_error] No code verification is performed.
 ├─► [execution_status] SUCCESS
 ├─► [execution_result]
 │   The execution of the generated python code above has succeeded
 │
 │   The result of above Python code after execution is:
 │   ('E12345', 'Development', 'X987', 'Software Engineer', 'J. Doe.', 'Remote.')
 ├─► [artifact_paths] []
 ├──● The following python code has been executed:
 │   ```python
 │   employee_summary = "On April 10, 2025, a review was conducted for an employee with ID `E12345`. The employee is part of the `Development` department and is currently assigned to project code \"X987\". The role held is      
 │   \"Software Engineer\" under the supervision of manager \"J. Doe.\" The primary work location is \"Remote.\""
 │
 │   # Extracting information
 │   employee_id = employee_summary.split('ID `')[1].split('`')[0]
 │   department = employee_summary.split('`')[3]
 │   project_code = employee_summary.split('project code \"')[1].split('\"')[0]
 │   role = employee_summary.split('role held is \"')[1].split('\"')[0]
 │   manager = employee_summary.split('manager \"')[1].split('\"')[0]
 │   location = employee_summary.split('location is \"')[1].split('\"')[0]
 │
 │   employee_id, department, project_code, role, manager, location
 │   ```
 │
 │
 │   The execution of the generated python code above has succeeded
 │
 │   The result of above Python code after execution is:
 │   ('E12345', 'Development', 'X987', 'Software Engineer', 'J. Doe.', 'Remote.')
 ╰──● sending message to Planner
 ╭───< Planner >
 ├─► [plan_reasoning] The employee data has been successfully extracted from the summary. The next step is to validate the role against the list of recognized roles.
 ├─► [init_plan]
 │   1. Extract the specified employee data from the summary provided.
 │   2. Validate the role against the recognized roles list <sequentially depends on 1>
 │   3. Construct the SQL query using the extracted details <sequentially depends on 2>
 │   4. Report the constructed SQL query to the user <interactively depends on 3>
 ├─► [plan]
 │   1. Extract the specified employee data from the summary provided.
 │   2. Validate the role against the recognized roles list.
 │   3. Construct the SQL query using the extracted details.
 │   4. Report the constructed SQL query to the user.
 ├─► [current_plan_step] 2. Validate the role against the recognized roles list.
 ├─► [stop] InProcess
 ├──● Please validate if the role 'Software Engineer' is recognized among the following roles: Software Engineer, Data Analyst, Project Manager, DevOps Specialist, QA Tester.
 ├─► [shared_memory_entry] Add the plan to the shared memory
 ╰──● sending message to CodeInterpreter
 ╭───< CodeInterpreter >
 ├─► [thought] I will validate if the role 'Software Engineer' is recognized among the given list of roles.
 ├─► [reply_type] python
 ├─► [reply_content]
 │   # Step 2: Validate the Role
 │   recognized_roles = ["Software Engineer", "Data Analyst", "Project Manager", "DevOps Specialist", "QA Tester"]
 │   role = "Software Engineer"
 │
 │   is_role_recognized = role in recognized_roles
 │   is_role_recognized
 ├─► [verification] NONE
 ├─► [code_error] No code verification is performed.
 ├─► [execution_status] SUCCESS
 ├─► [execution_result]
 │   The execution of the generated python code above has succeeded
 │
 │   The result of above Python code after execution is:
 │   True
 ├─► [artifact_paths] []
 ├──● The following python code has been executed:
 │   ```python
 │   # Step 2: Validate the Role
 │   recognized_roles = ["Software Engineer", "Data Analyst", "Project Manager", "DevOps Specialist", "QA Tester"]
 │   role = "Software Engineer"
 │
 │   is_role_recognized = role in recognized_roles
 │   is_role_recognized
 │   ```
 │
 │
 │   The execution of the generated python code above has succeeded
 │   
 │   The result of above Python code after execution is:
 │   True
 ╰──● sending message to Planner
 ╭───< Planner >
 ├─► [plan_reasoning] The role 'Software Engineer' is recognized among the list of roles. The next step is to construct the SQL query using the extracted details.
 ├─► [init_plan]
 │   1. Extract the specified employee data from the summary provided.
 │   2. Validate the role against the recognized roles list <sequentially depends on 1>
 │   3. Construct the SQL query using the extracted details <sequentially depends on 2>
 │   4. Report the constructed SQL query to the user <interactively depends on 3>
 ├─► [plan]
 │   1. Extract the specified employee data from the summary provided.
 │   2. Validate the role against the recognized roles list.
 │   3. Construct the SQL query using the extracted details.
 │   4. Report the constructed SQL query to the user.
 ├─► [current_plan_step] 3. Construct the SQL query using the extracted details.
 ├─► [stop] InProcess
 ├──● Please construct the SQL query using the extracted details: Employee ID 'E12345', Department 'Development', Project Code 'X987', Role 'Software Engineer', Manager 'J. Doe.', and Location 'Remote'. The SQL query should be  
 │   in the format provided in the task instructions.
 ├─► [shared_memory_entry] Add the plan to the shared memory
 ╰──● sending message to CodeInterpreter
 ╭───< CodeInterpreter >
 ├─► [thought] I will construct the SQL query using the extracted details from the previous successful round.
 ├─► [reply_type] python
 ├─► [reply_content]
 │   # Step 3: Construct SQL Query
 │   employee_id = 'E12345'
 │   department = 'Development'
 │   project_code = 'X987'
 │   role = 'Software Engineer'
 │   manager = 'J. Doe.'
 │   location = 'Remote.'
 │
 │   sql_query = f"""
 │   SELECT EmployeeID, Department, ProjectCode, Role, Manager, Location
 │   FROM EmployeeRecords
 │   WHERE EmployeeID = '{employee_id}'
 │   AND Department = '{department}'
 │   AND ProjectCode = '{project_code}'
 │   AND Role = '{role}'
 │   AND Manager = '{manager}'
 │   AND Location = '{location}'
 │   ORDER BY EmployeeID;
 │   """
 │
 │   sql_query
 ├─► [verification] NONE
 ├─► [code_error] No code verification is performed.
 ├─► [execution_status] SUCCESS
 ├─► [execution_result]
 │   The execution of the generated python code above has succeeded
 │
 │   The result of above Python code after execution is:
 │
 │   SELECT EmployeeID, Department, ProjectCode, Role, Manager, Location
 │   FROM EmployeeRecords
 │   WHERE EmployeeID = 'E12345'
 │   AND Department = 'Development'
 │   AND ProjectCode = 'X987'
 │   AND Role = 'Software Engineer'
 │   AND Manager = 'J. Doe.'
 │   AND Location = 'Remote.'
 │   ORDER BY EmployeeID;
 │
 ├─► [artifact_paths] []
 ├──● The following python code has been executed:
 │   ```python
 │   # Step 3: Construct SQL Query
 │   employee_id = 'E12345'
 │   department = 'Development'
 │   project_code = 'X987'
 │   role = 'Software Engineer'
 │   manager = 'J. Doe.'
 │   location = 'Remote.'
 │
 │   sql_query = f"""
 │   SELECT EmployeeID, Department, ProjectCode, Role, Manager, Location
 │   FROM EmployeeRecords
 │   WHERE EmployeeID = '{employee_id}'
 │   AND Department = '{department}'
 │   AND ProjectCode = '{project_code}'
 │   AND Role = '{role}'
 │   AND Manager = '{manager}'
 │   AND Location = '{location}'
 │   ORDER BY EmployeeID;
 │   """
 │
 │   sql_query
 │   ```
 │
 │
 │   The execution of the generated python code above has succeeded
 │
 │   The result of above Python code after execution is:
 │
 │   SELECT EmployeeID, Department, ProjectCode, Role, Manager, Location
 │   FROM EmployeeRecords
 │   WHERE EmployeeID = 'E12345'
 │   AND Department = 'Development'
 │   AND ProjectCode = 'X987'
 │   AND Role = 'Software Engineer'
 │   AND Manager = 'J. Doe.'
 │   AND Location = 'Remote.'
 │   ORDER BY EmployeeID;
 │
 ╰──● sending message to Planner
 ╭───< Planner >
 ├─► [plan_reasoning] The SQL query has been successfully constructed using the extracted employee details. The task is now complete, and the constructed SQL query can be reported to the user.
 ├─► [init_plan]
 │   1. Extract the specified employee data from the summary provided.
 │   2. Validate the role against the recognized roles list <sequentially depends on 1>
 │   3. Construct the SQL query using the extracted details <sequentially depends on 2>
 │   4. Report the constructed SQL query to the user <interactively depends on 3>
 ├─► [plan]
 │   1. Extract the specified employee data from the summary provided.
 │   2. Validate the role against the recognized roles list.
 │   3. Construct the SQL query using the extracted details.
 │   4. Report the constructed SQL query to the user.
 ├─► [current_plan_step] 4. Report the constructed SQL query to the user.
 ├─► [stop] Completed
 ├──● The SQL query has been successfully constructed using the extracted details:
 │
 │   ```
 │   SELECT EmployeeID, Department, ProjectCode, Role, Manager, Location
 │   FROM EmployeeRecords
 │   WHERE EmployeeID = 'E12345'
 │   AND Department = 'Development'
 │   AND ProjectCode = 'X987'
 │   AND Role = 'Software Engineer'
 │   AND Manager = 'J. Doe.'
 │   AND Location = 'Remote.'
 │   ORDER BY EmployeeID;
 │   ```
 ├─► [shared_memory_entry] Add the plan to the shared memory
 ╰──● sending message to User
 TaskWeaver ▶  The SQL query has been successfully constructed using the extracted details:

SELECT EmployeeID, Department, ProjectCode, Role, Manager, Location
FROM EmployeeRecords
WHERE EmployeeID = 'E12345'
AND Department = 'Development'
AND ProjectCode = 'X987'
AND Role = 'Software Engineer'
AND Manager = 'J. Doe.'
AND Location = 'Remote.'
ORDER BY EmployeeID;

The log is quite lengthy, but it is quite clear and easy to understand. For each step, the Planner always asks the Code Interpreter to generate the code. Finally, the task is completed successfully. However, it is not the most efficient way to solve the task because the Planner can leverage its reasoning capabilities to extract information and construct the SQL query directly.

We have tried to add instructions in the Planner's prompt to emphasize the reasoning power, but the Planner still calls the Code Interpreter to generate the code. For a long time, we don't quite understand the reason behind this behavior. After many discussions and experiments, we finally realized that it is caused by the design of the Planner's response schema. The Planner needs to talk to another Role in the system each time. But if we want the Planner to use reasoning to solve the current subtask, who should the Planner talk to? Ideally, it should continue until the subtask that requires the Code Interpreter to generate the code. But how can we design the schema to support this? Because the Planner is explicitly tracking the progress, i.e., the current_plan_step, it is confusing to let the Planner continue the reasoning process through future steps.

We value the progress tracking more than the efficiency of the reasoning process, because this is critical for complex multi-step tasks. But we also want to improve the reasoning process to make the agent more intelligent. So, our choice is to add a special Role called Recepta to listen to the Planner's reasoning process when the Planner wants to use reasoning to solve the current subtask. The Recepta always answers "Got it. Please go ahead!" to the Planner. The Planner can then continue the reasoning process until the subtask that requires the Code Interpreter to generate the code. The Recepta is not a real Role in the system, but a design pattern to support the reasoning process in the Planner.

Now, let's look at the output of TaskWeaver with Recepta:

╭───< Planner >
├─► [plan_reasoning] The user has provided a set of instructions to follow step by step. The task involves extracting employee data from a summary, validating the role, and constructing an SQL query. I will follow these steps sequentially.
├─► [init_plan]
│   1. Extract the following information from the employee summary: `Employee ID`, `Department`, `Project Code`, `Role`, `Manager`, `Location`
│   2. Validate the role <interactively depends on 1>
│   3. Construct and print the SQL query using the extracted details <interactively depends on 1>
│   4. Check the execution result and report the result to the user <interactively depends on 1 and 3>
├─► [plan]
│   1. Extract information from the employee summary
│   2. Validate the role
│   3. Construct SQL query using the extracted details
│   4. Report the result to the user
├─► [current_plan_step] 1. Extract information from the employee summary
├─► [stop] InProcess
├──● Thought: I can read and comprehend the employee summary to extract the required information.
│   Result: the `Employee ID` is `E12345`, the `Department` is `Development`, the `Project Code` is `X987`, the `Role` is `Software Engineer`, the `Manager` is `J. Doe`, and the `Location` is `Remote`.
├─► [shared_memory_entry] Add the plan to the shared memory
╰──● sending message to Recepta
╭───< Recepta >
├──● Got it. Now, please go ahead!
╰──● sending message to Planner
╭───< Planner >
├─► [plan_reasoning] I have extracted the information from the employee summary. The next step is to verify if the role is one of the recognized roles.
├─► [init_plan]
│   1. Extract the following information from the employee summary: `Employee ID`, `Department`, `Project Code`, `Role`, `Manager`, `Location`
│   2. Validate the role <interactively depends on 1>
│   3. Construct and print the SQL query using the extracted details <interactively depends on 1>
│   4. Check the execution result and report the result to the user <interactively depends on 1 and 3>
├─► [plan]
│   1. Extract information from the employee summary
│   2. Validate the role
│   3. Construct SQL query using the extracted details
│   4. Report the result to the user
├─► [current_plan_step] 2. Validate the role
├─► [stop] InProcess
├──● Thought: I can verify if the role is one of the recognized roles with my pattern matching capability.
│   Thought: The recognized roles are `Software Engineer, Data Analyst, Project Manager, DevOps Specialist, QA Tester`.
│   Result: The role is `Software Engineer`, which is recognized.
├─► [shared_memory_entry] Add the plan to the shared memory
╰──● sending message to Recepta
╭───< Recepta >
├──● Got it. Now, please go ahead!
╰──● sending message to Planner
╭───< Planner >
├─► [plan_reasoning] I can prepare and print the SQL query using the extracted details to the User.
├─► [init_plan]
│   1. Extract the following information from the employee summary: `Employee ID`, `Department`, `Project Code`, `Role`, `Manager`, `Location`
│   2. Validate the role <interactively depends on 1>
│   3. Construct and print the SQL query using the extracted details <interactively depends on 1>
│   4. Check the execution result and report the result to the user <interactively depends on 1 and 3>
├─► [plan]
│   1. Extract information from the employee summary
│   2. Validate the role
│   3. Construct SQL query using the extracted details
│   4. Report the result to the user
├─► [current_plan_step] 3. Construct and print the SQL query using the extracted details and report the result to the user
├─► [stop] Completed
├──● The role is `Software Engineer`, which is recognized.
│
│   Here is the SQL query using the extracted details:
│   ```sql
│   SELECT EmployeeID, Department, ProjectCode, Role, Manager, Location
│   FROM EmployeeRecords
│   WHERE EmployeeID = 'E12345'
│   AND Department = 'Development'
│   AND ProjectCode = 'X987'
│   AND Role = 'Software Engineer'
│   AND Manager = 'J. Doe'
│   AND Location = 'Remote'
│   ORDER BY EmployeeID;
│   ```
├─► [shared_memory_entry] Add the plan to the shared memory
╰──● sending message to User
 TaskWeaver ▶  The role is `Software Engineer`, which is recognized.

Here is the SQL query using the extracted details:

SELECT EmployeeID, Department, ProjectCode, Role, Manager, Location
FROM EmployeeRecords
WHERE EmployeeID = 'E12345'
AND Department = 'Development'
AND ProjectCode = 'X987'
AND Role = 'Software Engineer'
AND Manager = 'J. Doe'
AND Location = 'Remote'
ORDER BY EmployeeID;

The new log is not only shorter but also more efficient. It is also more reliable according to our observations. Writing code with regex to extract information from text is error-prone.

In summary, we have experimented the idea of adding a special Role called Recepta to support the reasoning process in the Planner. The Recepta is not a real Role in the system, but a design pattern to support the reasoning process in the Planner. This design pattern is simple and effective. It is also flexible and can be extended to support more complex reasoning processes in the Planner. We believe that this design pattern can be useful for building more intelligent agents in the future.

To enable it in TaskWeaver, the user needs to configure the roles in the configure file with the following content:

"session.roles": [
    "planner",
    "code_interpreter",
    "recepta"
]

The recepta role is added to the list of roles.

Static experience selection​

Every role in TaskWeaver can configure its own experience directory, which can be configured by setting the role_name.experience_dir field in the project configuration file. For the Planner and CodeInterpreter roles, you can configure the experience directory by setting the planner.experience_dir and code_generator.experience_dir fields respectively. The default experience directory is experience in the project directory.

By configuring different experience directories for different roles, you can have different experiences for different roles in a static way. Use the Planner role as an example, you can have the following project configuration file to enable the experience selection for the Planner role.

{
  "planner.use_experience": true,
  "planner.experience_dir": "planner_exp_dir"
}

Dynamic experience selection​

In some cases, you may want to select the experience dynamically based on the input task. In TaskWeaver, although we retrieve the experience based on the query content, it is sometimes difficult to obtain the right experience based solely on the similarity between the query content and the experience content.

One real example we encountered is that we need to retrieve the experience based on the task type. We have many task types in the project, let's say task_type_1, task_type_2, and task_type_3. Every task type is about following a set of instructions to complete a task. Although different task types have different instructions, the instructions are similar in structure. For example, they all have similar steps like step_1, step_2, and step_3, only the content of the steps is slightly different for different task types. Actually, even most of the step titles are the same for different task types.

Each task type has its own experience, and we want to select the experience based on the task type. Even though we can mix all the experiences into one experience directory, it is very hard to differentiate the experiences based on the user input or the step content. In this project, the user input is simply a task ID, and we need to first figure out the task type based on the task ID, and then select the experience based on the task type.

To achieve this, we add a layer in the experience selection process. Specifically, we allow having subdirectories in the experience directory. For example, we can have the following experience directory structure:

planner_experience
├── task_type_1
│   ├── exp_1.yaml
│   ├── exp_2.yaml
│   └── ...

When we can identify the task type based on the task ID, we can set the experience subdirectory. This looks straightforward, but how can we set the experience subdirectory in TaskWeaver? As we need to do this in a dynamic way, the only way is to set the experience subdirectory in a role.

TaskWeaver recently introduced the concept of shared memory as discussed in Shared Memory. Shared memory allows a role to share information with other roles, and in this case, we can use shared memory to set the experience subdirectory.

We can add a new role called TaskTypeIdentifier to identify the task type based on the task ID. The key part of the reply function in TaskTypeIdentifier is shown below:

def reply(self, memory: Memory, **kwargs: ...) -> Post:
    # ...
    # get the task type from the last post message
    task_type = get_task_type(last_post.message)
    # create an attachment 
    post_proxy.update_attachment(
        type=AttachmentType.shared_memory_entry,
        message="Add experience sub path",
        extra=SharedMemoryEntry.create(
            type="experience_sub_path",
            scope="conversation", # define the effective scope of the shared memory entry to be the whole conversation
            content="task_type_1",
        ),
    )

    return post_proxy.end()

In a role that needs to set the experience subdirectory, we can get the experience subdirectory from the shared memory.

def reply(
        self,
        memory: Memory,
        post_proxy: Optional[PostEventProxy] = None,
        prompt_log_path: Optional[str] = None,
        **kwargs: ...,
    ) -> Post:
    ...
    rounds = memory.get_role_rounds(
        role=self.alias,
        include_failure_rounds=False,
    )

    # obtain the query from the last round
    query = rounds[-1].post_list[-1].message
    
    # retrieve the experience based on the query
    self.role_load_experience(query=query, memory=memory)
    ...

Conclusion​

In this blog post, we have discussed how to select experiences in TaskWeaver. We have static experience selection by configuring the experience directory for each role. To enable dynamic experience selection, we have introduced the concept of shared memory to set the experience subdirectory.

Motivation​

We've seen many raised issues complaining that it is difficult to run TaskWeaver with locally deployed non-that-large language models (LLMs), such as 7b or 13b. When we examine the issues, we find that the main problem is that the models failed to generate responses following our formatting instructions in the prompt. For instance, we see that the planner's response does not contain a send_to field, which is required to determine the recipient of the message.

In the past, we have tried to address this issue by adding more examples in the prompt, which however did not work well, especially for these relatively small models. Another idea was to ask the model to re-generate the response if it does not follow the format. We include the format error in the prompt to help the model understand the error and correct it. However, this approach also did not work well.

Constrained Generation​

Recently, we discovered a new approach called "Constrained Generation" that can enforce the model to generate responses following the format. Popular frameworks include Outlines, Guidance, lm-format-enforcer, etc. All these frameworks support generating responses following a specific format, e.g., a JSON schema. This makes it possible to control the output format by providing it a schema.

In TaskWeaver, a relatively easy way to integrate this feature is to use a local deployment that supports both constrained generation and OpenAI compatible API, for instance, the vllm. There are other frameworks that support constrained generation, such as llama.cpp. But currently, we found that this feature is still not mature enough, so we start with vllm for experimentation.

To run vllm, you can follow the instructions in the vllm documentation. A simple example is shown below:

python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-8B-Instruct --guided-decoding-backend lm-format-enforcer

where --guided-decoding-backend lm-format-enforcer is used to enable the constrained generation feature and specify the backend. Currently, vllm only supports lm-format-enforcer and outlines.

Here is a sample code to test the vllm server:

from openai import OpenAI

json_schema = {
    "type": "object",
    "properties": {
        "country_name": {
            "type": "string"
        }
    },
    "required": ["country_name"]
}

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Which country is San Francisco in?"}
    ],
    extra_body={
        "guided_json": json_schema,
        "guided_decoding_backend": "lm-format-enforcer"
    }                           
)
print("Completion result:", completion)

If you run the above code, you will get the response following the format specified in the json_schema.

After you have successfully deployed vllm, you can set the following configurations in TaskWeaver:

{
    "llm.model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "llm.api_base": "http://localhost:8000/v1",
    "llm.api_key": "null",
    "llm.api_type": "openai",
    "llm.openai.require_alternative_roles": false,
    "llm.openai.support_system_role": true
}

The llm.openai.require_alternative_roles and llm.openai.support_system_role configurations are discussed in the OpenAI Configuration page. With these configurations, TaskWeaver will send the messages to the vllm server and get the responses.

How we implemented the constrained generation in TaskWeaver​

In order to support the constrained generation in TaskWeaver, we need to provide the schema to the model. Therefore, we made a few changes in the code to support this feature.

First, we add a response_json_schema field to the planner and code interpreter. For planner, you can find it in taskweaver/planner/planner_prompt.py. It looks like this:

response_json_schema: |-
  {
    "type": "object",
    "properties": {
        "response": {
            "type": "object",
            "properties": {
                "init_plan": {
                    "type": "string"
                },
                "plan": {
                    "type": "string"
                },
                "current_plan_step": {
                    "type": "string"
                },
                "send_to": {
                    "type": "string"
                },
                "message": {
                    "type": "string"
                }
            },
            "required": [
                "init_plan",
                "plan",
                "current_plan_step",
                "send_to",
                "message"
            ]
        }
    },
    "required": ["response"]
  }

If you are familiar with the previous output schema, you may notice that we have changed the response field to an object from an array of elements. This is because that it is much easier to express the schema in JSON format if the properties are in an object, not elements in an array.

Correspondingly, we add a response_json_schema field to the code interpreter. You can find it in taskweaver/code_interpreter/code_interpreter/code_generator_prompt.py, which looks like this:

response_json_schema: |-
    {
        "type": "object",
        "properties": {
            "response": {
                "type": "object",
                "properties": {
                    "thought": {
                        "type": "string"
                    },
                    "reply_type": {
                        "type": "string",
                        "enum": ["python", "text"]
                    },
                    "reply_content": {
                        "type": "string"
                    }   
                },
                "required": ["thought", "reply_type", "reply_content"]
            }
        },
        "required": ["response"]
    } 

We also change the response field to an object from an array of elements in the code interpreter. A benefit of this change is that we can now easily restrict the reply_type field to only two values: python and text, which is not possible before.

One consequence of this change is that we need to modify the examples for the code interpreter in order to support the new schema. The old examples contain attachments that have the types of python, text, and sample, which are deprecated. We now need to change them to the new schema. Specifically, we need to change the type field to reply_type and the content field to reply_content. For example, the old example:

- type: python
  content: |-
    file_path = "/abc/def.txt"  

    with open(file_path, "r") as file:  
        file_contents = file.read()  
        print(file_contents)

should be changed to:

- type: reply_type
  content: python # or 'text' if the old type is 'text' or 'sample'
- type: reply_content
  content: |-
    file_path = "/abc/def.txt"  

    with open(file_path, "r") as file:  
        file_contents = file.read()  
        print(file_contents)

There could be multiple thought attachments in the code interpreter examples. But in the new schema, there is only one thought field. So we have added code to do the conversion and no manual work is needed to modify the examples. If you have examples, after these changes, we can now support the constrained generation in TaskWeaver.

Second, we submit the JSON schema to the model when we need to call the endpoint, which you can find in planner.py and code_generator.py, respectively.

Conclusion​

In this blog post, we have introduced a new feature called "Constrained Generation" that can enforce the model to generate responses following the format. We have also shown how to run TaskWeaver with locally deployed non-that-large language models (LLMs) that support constrained generation. We have also explained how we implemented the constrained generation in TaskWeaver. We hope this feature can help you run TaskWeaver with LLMs more easily. If you have any questions or suggestions, please feel free to contact us.

Plugin Basics​

In TaskWeaver, the plugins are the essential components to extend the functionality of the agent. Specifically, a plugin is a piece of code wrapped in a class that can be called as a function by the agent in the generated code snippets. The following is a simple example of a plugin that generates n random numbers:

from taskweaver.plugin import Plugin, register_plugin

@register_plugin
class RandomGenerator(Plugin):
    def __call__(self, n: int):
        import random
        return [random.randint(1, 100) for _ in range(n)]

In this example, the RandomGenerator class inherits the Plugin class and implements the __call__ method, which means it can be called as a function. What would be the function signature of the plugin? It is defined in the associated YAML file. For example, the YAML file for the RandomGenerator plugin is as follows:

name: random_generator
enabled: true
required: true
description: >-
  This plugin generates n random numbers between 1 and 100.
examples: |-
  result = random_generator(n=5)
parameters:
  - name: n
    type: int
    required: true
    description: >-
      The number of random numbers to generate.

returns:
  - name: result
    type: list
    description: >-
      The list of random numbers.

The YAML file specifies the name, description, parameters, and return values of the plugin. When the LLM generates the code snippets, it will use the information in the YAML file to generate the function signature. We did not check the discrepancy between the function signature in the Python implementation and the YAML file. So, it is important to keep them consistent. The examples field is used to provide examples of how to use the plugin for the LLM.

Configurations and States​

Although the plugin is used as a function in the code snippets, it is more than a normal Python function. The plugin can have its own configurations and states. For example, the RandomGenerator plugin can have a configuration to specify the range of the random numbers. The configurations can be set in the YAML file as follows:

# the previous part of the YAML file
configurations:
  - name: range
    type: list
    required: false
    description: >-
      The range of the random numbers.
    default: [1, 100]

We did not show how to use the configurations in the plugin implementation, which could be found in one of our sample plugins, namely sql_pull_data. Supporting configurations in the plugin is a powerful feature to make the plugin more flexible and reusable. For example, we can have multiple YAML files pointing to the same Python implementation but with different configurations. Read this page for more details. When TaskWeaver loads the plugins, it will elaborate the YAML files and create the plugin objects with the configurations. Therefore, two plugins with the same Python implementation but different configurations are actually different objects in memory. That is why different plugins can have different states, and this is especially helpful when the plugin needs to maintain some states across different calls. Consider the example of the sql_pull_data sample plugin, which has the following code snippet:

@register_plugin
class SqlPullData(Plugin):
    db = None

    def __call__(self, query: str):
        ...

        if self.db is None:
            self.db = SQLDatabase.from_uri(self.config.get("sqlite_db_path"))

In the example above, the SqlPullData plugin maintains a database connection across different calls. If we design the plugin to be a stateless normal Python function, we would need to establish a new connection for each call, which is inefficient and not necessary.

The Plugin Lifecycle​

The plugin lifecycle is the process of how the plugin is loaded, initialized, and called by the agent. When TaskWeaver starts, it goes through all the plugin configuration files in the plugins directory and creates the plugin entries in the memory. The Python implementation of the plugin is not loaded at this stage. When the agent generates the code snippets, it will call the plugin by the name specified in the YAML file, and fill in the function signature based on the information in the YAML file.

The plugin is loaded and initialized when the code executor executes the code snippets for the first time in a session. The plugin is initialized with the configurations specified in the YAML file. Although we have the feature to dynamically select the plugins in the LLM, all the plugins are loaded no matter whether they are used in the current conversation round. The only way of controlling the plugin loading is to enable or disable the plugin in the YAML file. In theory, the plugins can be configured separately for different sessions. For example, when a user starts a new session, we can load a different set of plugins based on the user's profile. But this feature is not supported in TaskWeaver yet.

The plugin is called when the agent executes the code snippets. The plugin can maintain states across different calls, which has been discussed in the previous section. As each session is associated with a Jupyter kernel, the plugin objects are created in the kernel memory and can be accessed across different code snippets, from different code cells, in the same session. When the session is closed, the plugin objects are also destroyed with the kernel.

Conclusion​

In this page, we discussed the basics of the plugin in TaskWeaver, including the plugin implementation, the YAML file, the configurations, and the states. We also introduced the plugin lifecycle, which is the process of how the plugin is loaded, initialized, and called by the agent. The plugin is a powerful component in TaskWeaver to extend the functionality of the agent.

It is nontrivial to evaluate the performance of a LLM agent. Existing evaluation methods typically treat the LLM agent as a function that maps input data to output data. If the agent is evaluated against a multi-step task, the evaluation process is then like a chain of calling a stateful function multiple times. To judge the output of the agent, it is typically compared to a ground truth or a reference output. As the output of the agent is in natural language, the evaluation is typically done by matching keywords or phrases in the output to the ground truth.

This evaluation method has its limitations due to its rigid nature. It is sometimes hard to use keywords matching to evaluate the output of the agent, especially when the output is long and complex. For example, if the answer is a date or a number, the evaluation method may not be able to handle the different formats. Moreover, the evaluation method should be able to act more like a human, who can understand the context and the meaning of the output. For example, when different agents are asked to perform the same task, they may behave differently, but still produce correct outputs.

The below example illustrates this point:

Human: What is the weather today?
Agent 1: It is sunny today in New York.

Human: What is the weather today?
Agent 2: Do you want to know the weather in New York today?
Human: Yes.
Agent 2: It is sunny today.

Compared to Agent 1, Agent 2 asks for confirmation before providing the answer, which requires more interaction with the user. However, both agents provide the correct answer to the question. But if the evaluation method takes the agent as a function, it may not be able to handle the different behaviors of the agents and consider Agent 2 as incorrect (as the first response does not match the ground truth, e.g., "sunny").

A new evaluation method​

Therefore, we propose a new evaluation method that treats the agent as a conversational partner as shown in the figure below:

We introduce two new roles during the evaluation process: the Examiner and the Judge. For each test case, the task description is first given to the Examiner. The Examiner then asks questions to the agent and supervises the conversation. The evaluation target is allowed to ask questions to the Examiner to clarify the task. The Examiner can only provide the task description and cannot provide any hints or solutions. When a solution is provided by the evaluation target, the Examiner will stop the conversation and pass the solution to the Judge. The Judge will then evaluate the solution based on the ground truth. Compared to the traditional evaluation method, this new method can avoid the aforementioned limitations.

Let's see an example of how the new evaluation method works. The following YAML file is a task description for the task "Sum of 1 to 50". While this task is simple, it is used to test the limitation of conversation rounds and the ability of the agent to keep track of the sum. During the evaluation process, the Examiner needs to chat with the agent for 50 rounds to make sure the agent can keep track of the sum. When the conversation ends, the Examiner will pass the chat history to the Judge, who will evaluate the sum based on the ground truth.

task_description: |-
  The task has many rounds. The initial total sum is 0. 
  Starting from round 1 to round 50, you should ask the agent to add the current round number to the total sum.
  The agent should keep track of the sum and return the sum after the 50th round.
  Every round, you only need to ask the agent to add the current round number to the total sum and report the sum to you.
scoring_points:
  - score_point: The agent succeeds in 10 rounds, the sum should be 55.
    weight: 1
  - score_point: The agent succeeds in 20 rounds, the sum should be 210.
    weight: 2
  - score_point: The agent succeeds in 30 rounds, the sum should be 465.
    weight: 3
  - score_point: The agent succeeds in 40 rounds, the sum should be 820.
    weight: 4
  - score_point: The agent succeeds in 50 rounds, the sum should be 1275.
    weight: 5

The ground truth is represented by the scoring_points field in the YAML file. Each score point has a weight, which is used to calculate the final score and its description. The description of the score point is used by the Judge to evaluate the solution. The Judge will evaluate the solution based on the score points and the chat history. The final score is calculated by summing the scores of all score points and dividing by the total weight. Therefore, the normalized score is between 0 and 1.

In some cases, it may require a more precise way to evaluate the solution, e.g., with code. This following task description is an example of such a case.

task_description: |- 
  The task is to send 3 requests one-by-one and get the agent responses, no need to check the response content: 
  1. generate 1 random integer number and save it to a file named 'a.txt', just tell me if the task is done
  2. tell me a random joke
  3. save the previously generated random number to a file named 'b.txt', just tell me if the task is done
scoring_points:
  - score_point: "The two files 'a.txt' and 'b.txt' should contain the same number"
    weight: 1
    eval_code: |-
      content_a = open('a.txt', 'r').read().strip()
      content_b = open('b.txt', 'r').read().strip()
      assert content_a == content_b, f"content of a.txt: {content_a}, content of b.txt: {content_b}"

We need to evaluate the solution based on the content of the files 'a.txt' and 'b.txt'. The eval_code field is used to write the evaluation code. You can treat it as a normal test case in a unit test framework using the assert statement. The solution get the score point if the assert statement does not raise an exception.

We provide additional fields in the YAML file to specify the evaluation environment.

version: the version of the evaluation file
config_var: configurations of the agent for this evaluation case
app_dir: the working directory of the agent
dependencies: list of packages required by the agent
data_files: list of files copied to the working directory
max_rounds: the maximum number of rounds for the conversation

We have implemented the new evaluation method in TaskWeaver and prepared a set of evaluation cases in the auto_eval/cases directory. Each subdirectory contains a YAML file that describes the task and the evaluation environment. To run the evaluation, you can find more details in the auto_eval/README.md file.

How to adapt for other agents?​

Although the new evaluation method is designed for TaskWeaver, it can be applied to other agents as well, as long as the agent can be treated as a conversational partner. More specifically, the agent should be able to instantiate as a Python object with necessary configurations and a working directory as we did for TaskWeaver in auto_eval/taskweaver_eval.py:

class TaskWeaverVirtualUser(VirtualUser):
    def __init__(self, task_description: str, app_dir: str, config_var: Optional[dict] = None):
        super().__init__(task_description)

        self.app = TaskWeaverApp(app_dir=app_dir, config=config_var)
        self.session = self.app.get_session()
        self.session_id = self.session.session_id

    def get_reply_from_agent(self, message: str) -> str:
        response_round = self.session.send_message(
            message,
            event_handler=None,
        )
        assert response_round.state != "failed", "Failed to get response from agent."
        return response_round.post_list[-1].message

    def close(self):
        self.app.stop()

To add another agent, you need to implement the VirtualUser class and the get_reply_from_agent, close methods.

Under this design, when the developer needs to extend the agent's capability, they can write a new plugin. A plugin is a piece of code wrapped in a class that can be called as a function by the agent in the generated code snippets. Let's consider an example: the agent is asked to load a CSV file and perform anomaly detection on the data. The workflow of the agent is in the diagram below. It is very natural to represent data to be processed in variables and this task in code snippets.

However, we do find challenges for other tasks that are not naturally represented in code snippets. Let's consider another example: the agent is asked to read a manual and follow the instructions to process the data. We first assume there is a plugin that can read the manual and extract the instructions, called read_manual. The workflow of the agent is in the diagram below. This diagram only shows the first step of the task, which is to read the manual and extract the instructions. Although it does obtain the instructions, and the agent can follow them to complete the task, the behavior of the agent is less natural compared to the previous example.

Why? First, there is no need to generate code to read the manual and extract the instructions. Once the Planner has decided to read the manual, the code to extract the instructions is straightforward. Even though that there might be dynamic parts in the code such as some arguments in the function read_manual, it could be handled by the Planner. Therefore, the Code Generator is not necessary in this case, and the current flow actually incurred unnecessary LLM call overhead to generate the code snippets. Second, it does not make sense to represent the instructions in variables. The instructions are not data to be processed, but a text guide for the agent to follow.

For these reasons, we introduced the concept of roles in TaskWeaver. Roles are actually not new in TaskWeaver as there are already roles like Planner and CodeInterpreter. To add a new role, the developer can follow the documentation here. In general, a role is a class that inherits the Role class and implements the reply method. The reply method is the function that the agent calls to interact with the role, which has the following signature:

def reply(self, memory: Memory, **kwargs) -> Post:
    # implementation

It takes the memory object, which is the memory of the agent, and returns a Post object, which is the response of the role to the Planner. With the memory object, the role can access the history of the conversation and the context of the conversation. You may have noticed that all roles in TaskWeaver can only talk to the Planner, not to each other. If a role needs to talk to another role, it should go through the Planner. This design is to ensure that the Planner can control the conversation and the flow of the conversation. For a task that requires multiple roles to work together, the Planner can orchestrate the roles to work together to complete the task as shown in the diagram below.

The communication between the Planner and the roles is done through the Post object. In other words, they talk to each other by sending messages in natural language. What if a role needs to send some data to another role? If this is the case, we would recommend to implement a new plugin instead of a new role. Otherwise, you may need to store the data in an external storage like a database and let the other role to access it.

In summary, the concept of roles in TaskWeaver is to provide a way to extend the agent's capability by implementing new roles. This is especially useful when the task is not naturally represented in code snippets such as acquire text information from a knowledge base or the internet. Implementing a new role is straightforward by inheriting the Role class and implementing the reply method. All extra roles should be put in the TaskWeaver/taskweaver/ext_role folder, which will be automatically loaded by TaskWeaver. We have provided a few sample roles in the TaskWeaver/taskweaver/ext_role folder, such as the Echo role that echoes the user's message back to the user. More advanced role examples are the Planner and the CodeInterpreter roles, which are the core roles in TaskWeaver.