跳至内容

Scrapegraph

Scrapegraph工具规范 #

基类: BaseToolSpec

用于网页抓取操作的scrapegraph工具规范。

Source code in llama-index-integrations/tools/llama-index-tools-scrapegraph/llama_index/tools/scrapegraph/base.py 
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
class ScrapegraphToolSpec(BaseToolSpec):
    """scrapegraph tool specification for web scraping operations."""

    spec_functions = [
        "scrapegraph_smartscraper",
        "scrapegraph_markdownify",
        "scrapegraph_search",
    ]

    def scrapegraph_smartscraper(
        self,
        prompt: str,
        url: str,
        api_key: str,
        schema: Optional[List[BaseModel]] = None,
    ) -> List[Dict]:
        """
        Perform synchronous web scraping using scrapegraph.

        Args:
            prompt (str): User prompt describing the scraping task
            url (str): Target website URL to scrape
            api_key (str): scrapegraph API key
            schema (Optional[List[BaseModel]]): Pydantic models defining the output structure

        Returns:
            List[Dict]: Scraped data matching the provided schema

        """
        client = Client(api_key=api_key)

        # Basic usage
        return client.smartscraper(
            website_url=url, user_prompt=prompt, output_schema=schema
        )

    def scrapegraph_markdownify(self, url: str, api_key: str) -> str:
        """
        Convert webpage content to markdown format using scrapegraph.

        Args:
            url (str): Target website URL to convert
            api_key (str): scrapegraph API key

        Returns:
            str: Markdown representation of the webpage content

        """
        client = Client(api_key=api_key)

        return client.markdownify(website_url=url)

    def scrapegraph_search(self, query: str, api_key: str) -> str:
        """
        Perform a search query using scrapegraph.

        Args:
            query (str): Search query to execute
            api_key (str): scrapegraph API key

        Returns:
            str: Search results from scrapegraph

        """
        client = Client(api_key=api_key)
        return client.search(query=query)

scrapegraph_smartscraper #

scrapegraph_smartscraper(prompt: str, url: str, api_key: str, schema: Optional[List[BaseModel]] = None) -> List[Dict]

使用scrapegraph执行同步网页抓取。

参数:

名称 类型 描述 默认值
prompt str

用户描述网页抓取任务的提示

 required
url str

要抓取的目标网站URL

 required
api_key str

scrapegraph API密钥

 required
schema Optional[List[BaseModel]]

定义输出结构的Pydantic模型

 None

返回：

类型 描述
List[Dict]

List[Dict]: 与提供的模式匹配的抓取数据
Source code in llama-index-integrations/tools/llama-index-tools-scrapegraph/llama_index/tools/scrapegraph/base.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
def scrapegraph_smartscraper(
    self,
    prompt: str,
    url: str,
    api_key: str,
    schema: Optional[List[BaseModel]] = None,
) -> List[Dict]:
    """
    Perform synchronous web scraping using scrapegraph.

    Args:
        prompt (str): User prompt describing the scraping task
        url (str): Target website URL to scrape
        api_key (str): scrapegraph API key
        schema (Optional[List[BaseModel]]): Pydantic models defining the output structure

    Returns:
        List[Dict]: Scraped data matching the provided schema

    """
    client = Client(api_key=api_key)

    # Basic usage
    return client.smartscraper(
        website_url=url, user_prompt=prompt, output_schema=schema
    )

scrapegraph_markdownify #

scrapegraph_markdownify(url: str, api_key: str) -> str

使用scrapegraph将网页内容转换为markdown格式。

参数:

名称 类型 描述 默认值
url str

要转换的目标网站URL

 required
api_key str

scrapegraph API密钥

 required

返回：

名称 类型 描述
str str

网页内容的Markdown表示
Source code in llama-index-integrations/tools/llama-index-tools-scrapegraph/llama_index/tools/scrapegraph/base.py 
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def scrapegraph_markdownify(self, url: str, api_key: str) -> str:
    """
    Convert webpage content to markdown format using scrapegraph.

    Args:
        url (str): Target website URL to convert
        api_key (str): scrapegraph API key

    Returns:
        str: Markdown representation of the webpage content

    """
    client = Client(api_key=api_key)

    return client.markdownify(website_url=url)

scrapegraph_search(query: str, api_key: str) -> str

使用scrapegraph执行搜索查询。

参数:

名称 类型 描述 默认值
query str

要执行的搜索查询

 required
api_key str

scrapegraph API密钥

 required

返回：

名称 类型 描述
str str

来自scrapegraph的搜索结果
Source code in llama-index-integrations/tools/llama-index-tools-scrapegraph/llama_index/tools/scrapegraph/base.py 
63
64
65
66
67
68
69
70
71
72
73
74
75
76
def scrapegraph_search(self, query: str, api_key: str) -> str:
    """
    Perform a search query using scrapegraph.

    Args:
        query (str): Search query to execute
        api_key (str): scrapegraph API key

    Returns:
        str: Search results from scrapegraph

    """
    client = Client(api_key=api_key)
    return client.search(query=query)
优云智算