AI Multi Agents: PyAutogen
Recently I was having a conversation about whats trendy in the Dsc/AI space.
- RAGs
- Function calling
- Structured OutputS,…
- Langfuse and MLFlow…
They are all great. And a must KNOW.
But can we just have LLMs working together?
Actually, this is great tool to have when doing research of some new topic.
🐫 CAMEL: Finding the Scaling Law of Agents. A multi-agent framework. https://www.camel-ai.org
- The Site
- The Agent_Swarm Code at Github
- License: MIT ❤️
Open Source Agents Swarm Projects
LangGraph
Aigent builder framework: alternative to autogen and crewAI.
It is a graph framework and each node is code.
LangGraph | Summarized with Groq 📌
- Thanks to the related repo - https://github.com/whitew1994WW/LangGraphForBeginners
I have replicated the code here for the REact example. But this fwk can do also reflection agent, router…
LangGraph vs LangChain | Summarized with Groq 📌
LangChain is a framework that focuses on chaining together multiple language models to generate more accurate and informative responses. By leveraging the strengths of individual models, LangChain creates a robust system capable of tackling complex linguistic tasks. The video highlights LangChain’s ability to handle long-range dependencies, making it an attractive solution for applications that require detailed explanations or storytelling.
LangGraph, on the other hand, takes a more graph-based approach to language understanding. By representing language as a graph, LangGraph can capture relationships between entities, concepts, and relationships, enabling more nuanced and context-aware interactions. The video showcases LangGraph’s strength in handling ambiguous or unclear input, making it an ideal choice for applications that require high accuracy and precision. Key Differences and Applications
LangChain excels in tasks that require creativity and coherence, such as writing stories or generating dialogues. LangGraph, with its graph-based approach, is better suited for tasks that require precision and accuracy, such as question answering or entity disambiguation.
Takeaways
LangChain and LangGraph are two distinct frameworks with unique strengths and weaknesses.
LangChain is ideal for applications that require creativity, coherence, and long-range dependencies.
LangGraph is better suited for applications that require precision, accuracy, and nuanced language understanding.
Understanding the differences between these frameworks is crucial for harnessing the power of AI in various industries.
The Fabric Project
- The Fabric Code at Github
- License: MIT ❤️
Fabric is an open-source framework for augmenting humans using AI.
A collection of prompts (patterns) to interact effectively with AI:
- Summarize a video
- Create Prompts for Image Generation
- Answer questions about your code…
We can use Fabric together with Ollama and Groq - Both are OpenAI API Compatible.
Fabric with Ollama
Fabric with Groq
Other F/OSS Multi-Agent Projects
The Autogen Project
https://github.com/microsoft/autogen https://microsoft.github.io/autogen/
Camel
https://github.com/camel-ai/camel
🐫 CAMEL: Finding the Scaling Law of Agents. A multi-agent framework. https://www.camel-ai.org
MetaGPT
MetaGPT Project Details:
MetaGPT - A F/OSS Multi Agent Framework that will power your next Projects
CrewAI
CrewAI is a Framework that will make easy for us to get Local AI Agents interacting between them.
Interesting Research Tools 📌
Research Agent
LLM based autonomous agent that conducts local and web research on any topic and generates a comprehensive report with citations - Apache v2
Agents with AutoGen - AG2AI
We have several agent concepts in AG2 to help you build your AI agents.
We introduce the most common ones here:
Conversable Agent: Agents that are able to send messages, receive messages and generate replies using GenAI models, non-GenAI tools, or human inputs.
Human in the loop: Add human input to the conversation
Orchestrating multiple agents: Users can orchestrate multiple agents with built-in conversation patterns such as swarms, group chats, nested chats, sequential chats or customize the orchestration by registering custom reply methods.
Tools: Programs that can be registered, invoked and executed by agents - Agents gain significant utility through tools as they provide access to external data, APIs, and functionality.
Advanced Concepts: AG2 supports more concepts such as structured outputs, rag, code execution, etc.
pip install ag2
#or with the alias!
I got started with PyAutoGen thanks to this .ipynb
:
https://colab.research.google.com/drive/1xee3xdatViM4pespvLVVOrHJ8sB1MgO5?usp=drive_link
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
llm_config = {
"config_list": config_list_from_json(env_or_file="OAI_CONFIG_LIST")
}
assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False})
user_proxy.initiate_chat(assistant, message="Plot a chart of NVDA and TESLA stock price change YTD.")
# This initiates an automated chat between the two agents to solve the task
Advanced agentic design patterns AG2 supports more advanced concepts to help you build your AI agent workflows. You can find more information in the documentation.
Structured Output Ending a conversation Retrieval Augmented Generation (RAG) Code Execution Tools with Secrets
What mor ecan PyAutoGen do? 📌
Yes, PyAutoGen can be used to get information about websites and databases, although the methods and complexity vary. Here’s how:
1. Websites:
- Web Scraping:
- AutoGen agents can use Python libraries like
requests
andBeautiful Soup
(orScrapy
) to scrape information from websites. - Agents can be instructed to:
- Download HTML content.
- Parse the HTML to extract specific data.
- Follow links to navigate through the website.
- Example:
- An agent could be asked to get the current price of a product from an e-commerce website.
- Another agent could be used to summarize the content of a news article.
- AutoGen agents can use Python libraries like
- Web APIs:
- If the website provides a public API, agents can use
requests
to interact with the API and retrieve structured data (JSON, XML). - This is generally more reliable and efficient than web scraping.
- If the website provides a public API, agents can use
- Browser Automation:
- For websites that rely heavily on JavaScript or require user interaction, agents can use libraries like
Selenium
orPlaywright
to automate browser actions. - This allows agents to:
- Load web pages.
- Interact with elements (buttons, forms).
- Extract data from dynamic content.
- For websites that rely heavily on JavaScript or require user interaction, agents can use libraries like
2. Databases:
- SQL Queries:
- AutoGen agents can use Python libraries like
sqlite3
,psycopg2
(for PostgreSQL),mysql-connector-python
(for MySQL), orSQLAlchemy
to connect to databases and execute SQL queries. - Agents can be instructed to:
- Connect to a database.
- Formulate SQL queries based on user requests.
- Retrieve and process data from the database.
- Example:
- An agent could be asked to retrieve a list of customers from a database.
- Another agent could be used to calculate aggregate statistics (e.g., average sales).
- AutoGen agents can use Python libraries like
- Database APIs:
- Some databases provide REST APIs or other interfaces that agents can use to interact with the database.
- This can be useful for cloud-based databases or when direct SQL access is not available.
- Langchain integration:
- AutoGen can work in conjunction with langchain, and therefore use any of the database integrations that langchain provides. This can simplify the database interactions.
Key Considerations:
- Security:
- When interacting with websites or databases, it’s crucial to handle credentials and sensitive data securely.
- Avoid hardcoding API keys or database passwords in your code. Use environment variables or secure configuration files.
- Rate Limiting:
- Websites and APIs often have rate limits to prevent abuse.
- Implement mechanisms to handle rate limits and avoid overloading servers.
- Data Parsing:
- Parsing HTML or database results can be complex, especially when dealing with unstructured or inconsistent data.
- Use robust parsing techniques and error handling.
- Ethical Considerations:
- Respect website terms of service and database usage policies.
- Avoid scraping or accessing data that you are not authorized to access.
AutoGen’s Role:
- AutoGen provides the framework for orchestrating the agents that perform these tasks.
- It enables agents to collaborate, communicate, and coordinate their efforts to achieve complex goals involving websites and databases.
- AutoGen can be used to create agents that can decide which tool, or function to use, in order to get the required information from the website or database.
In summary, AutoGen can be a powerful tool for building applications that retrieve and process information from websites and databases, but it’s important to consider the technical and ethical implications of these activities.
Agents with LangChain
You guessed it.
LangChain can also do Agents tricks.
You might know Langchain because of its useful Chains and its RAGs Features
Agents with Llama-Index
LlamaIndex, as described, is designed to work with various tools, data sources, and workflows.
- https://jalcocert.github.io/JAlcocerT/using-langchain-with-pandas-df/#llamaindex-and-pandas-dataframes
- https://jalcocert.github.io/JAlcocerT/how-to-use-rags-with-python/#llama-index
It helps to augment LLMs with context from different data sources such as APIs, SQL databases, PDFs, etc., without being tied to a specific framework.
The ability to integrate multiple agents, tools, and data sources makes it adaptable to various development environments and workflows.
Context augmentation involves providing an LLM with specific data (private or external) to help solve a problem, enhancing the LLM’s ability to generate relevant answers by accessing relevant data.
Agents are knowledge assistants powered by LLMs that perform tasks using various tools, including Retrieval-Augmented Generation (RAG) as one tool in a larger workflow.
Workflows combine multiple agents, tools, and data connectors to create complex, event-driven processes that can automate tasks, reflecting advanced LLM applications.
Agents with PydanticAI
I was using Pydantic this year.
We can say it is a data validation framework, that now it has an AI version
Agent Framework / shim to use Pydantic with LLMs - MIT Licensed!
What it is Pydantic? Data Validation? 📌
Pydantic is a data validation and settings management library in Python.
It’s widely used for validating data and ensuring that inputs conform to the expected types and formats.
Chat-DEV
- The ChatDev Repository
- License: Apache v2 ❤️
Create Customized Software using Natural Language Idea (through LLM-powered Multi-Agent Collaboration)
Langroid
Example - https://github.com/langroid/langroid/blob/main/examples/basic/chat-search-assistant.py
Harness LLMs with Multi-Agent Programming - MIT LICENSED!
Auto-GPT
AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
CrewAI
You can use Streamlit with CrewAI
Example - https://github.com/tonykipkemboi/trip_planner_agent
CrewAI agents that can plan your vacation.
CrewAI Pandas DF Agent
MetaGPT
MIT | 🌟 The Multi-Agent Framework: First AI Software Company, Towards Natural Language Programming
SuperAGI
<⚡️> SuperAGI - A dev-first open source autonomous AI agent framework. Enabling developers to build, manage & run useful autonomous agents quickly and reliably.
Conclusions
FAQ
Interesting LLMs Architectures
- MoE - Mix of Experts
- MoA - Mix of Agents
- ReACT
More about Solara - ReACT Framework 📌
https://github.com/widgetti/solara
A Pure Python, React-style Framework for Scaling Your Jupyter and Web Apps
Build a Python WebApp in 3 minutes with Solara (Streamlit Alternative)
How to Provide Web Context to the Agents
Via Scrapping
Via Web Search
For example, with duckduckgo:
pip install duckduckgo-search
from duckduckgo_search import ddg_answers
# Perform search query
results = ddg_answers('Python programming')
# Output the search results
print(results)