autogen_ext.agents.video_surfer

class VideoSurfer(name: str, model_client: ChatCompletionClient, *, tools: 列表[BaseTool[BaseModel, BaseModel] | Callable[[...], 任何] | Callable[[...], Awaitable[任何]]] | 无 = None, description: str | 无 = None, system_message: str | 无 = None)

基类：AssistantAgent

VideoSurfer 是一个专门设计的代理，用于回答关于本地视频文件的问题。

安装：

pip install "autogen-ext[video-surfer]"

该代理利用各种工具从视频中提取信息，例如视频长度、特定时间戳的截图以及音频转录。它处理这些元素以提供对用户查询的详细回答。

可用的工具：

• extract_audio()

• get_video_length()

• transcribe_audio_with_timestamps()

• get_screenshot_at()

• save_screenshot()

• transcribe_video_screenshot()

Parameters:

• name (str) – 代理的名称。

• model_client (ChatCompletionClient) – 用于生成响应的模型客户端。

• 工具 (列表[BaseTool[BaseModel, BaseModel] | 可调用[..., Any] | 可调用[..., Awaitable[Any]]] | 无, 可选) – 代理可以使用的工具或函数列表。如果未提供，则默认为动作空间中的所有视频工具。

• description (str, optional) – 代理的简要描述。默认为“一个可以回答关于本地视频问题的代理。”。

• system_message (str | None, optional) – 系统消息指导代理的行为。默认为预定义的消息。

示例用法：

以下示例演示了如何使用模型客户端创建一个视频浏览代理，并生成对名为video.mp4的本地视频的简单查询的响应。

import asyncio
from autogen_agentchat.ui import Console
from autogen_agentchat.conditions import TextMentionTermination
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_ext.agents.video_surfer import VideoSurfer

async def main() -> None:
    """
    Main function to run the video agent.
    """
    # Define an agent
    video_agent = VideoSurfer(
        name="VideoSurfer",
        model_client=OpenAIChatCompletionClient(model="gpt-4o-2024-08-06")
        )

    # Define termination condition
    termination = TextMentionTermination("TERMINATE")

    # Define a team
    agent_team = RoundRobinGroupChat([video_agent], termination_condition=termination)

    # Run the team and stream messages to the console
    stream = agent_team.run_stream(task="How does Adam define complex tasks in video.mp4? What concrete example of complex does his use? Can you save this example to disk as well?")
    await Console(stream)

asyncio.run(main())

以下示例演示了如何使用MagenticOneGroupChat创建和使用VideoSurfer和UserProxyAgent。

import asyncio

from autogen_agentchat.ui import Console
from autogen_agentchat.teams import MagenticOneGroupChat
from autogen_agentchat.agents import UserProxyAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_ext.agents.video_surfer import VideoSurfer

async def main() -> None:
    """
    Main function to run the video agent.
    """

    model_client = OpenAIChatCompletionClient(model="gpt-4o-2024-08-06")

    # Define an agent
    video_agent = VideoSurfer(
        name="VideoSurfer",
        model_client=model_client
        )

    web_surfer_agent = UserProxyAgent(
        name="User"
    )

    # Define a team
    agent_team = MagenticOneGroupChat([web_surfer_agent, video_agent], model_client=model_client,)

    # Run the team and stream messages to the console
    stream = agent_team.run_stream(task="Find a latest video about magentic one on youtube and extract quotes from it that make sense.")
    await Console(stream)

asyncio.run(main())

DEFAULT_DESCRIPTION = 'An agent that can answer questions about a local video.'

DEFAULT_SYSTEM_MESSAGE = '\n    You are a helpful agent that is an expert at answering questions from a video.\n    When asked to answer a question about a video, you should:\n    1. Check if that video is available locally.\n    2. Use the transcription to find which part of the video the question is referring to.\n    3. Optionally use screenshots from those timestamps\n    4. Provide a detailed answer to the question.\n    Reply with TERMINATE when the task has been completed.\n    '

async vs_transribe_video_screenshot(video_path: str, timestamp: float) → str

在特定时间戳转录视频截图。

Parameters:

• video_path (str) – 视频文件的路径。

• timestamp (float) – 截取截图的时间戳。

Returns:

str – 视频截图的转录。