AI Multi Agents: PyAutogen, LangGraph,...
The Agentic Ecosystem
Lately everyone talks about agents…
And most likely than not, you will have found already these frameworks: LangGraph vs. CrewAI vs. AutoGen vs. OpenAI Agents…
But lets get few concepts in place first.
Prompt chaining, orchestration workers, and agents are distinct but related concepts in the realm of building complex applications with Large Language Models (LLMs).
1. Prompt Chaining:
- Concept: Prompt chaining involves linking multiple LLM prompts sequentially. The output of one prompt serves as the input for the next, allowing you to break down complex tasks into smaller, manageable steps.
- Focus: Primarily on structuring the interaction with one or more LLMs through a predefined sequence of prompts to achieve a specific outcome.
- Intelligence/Autonomy: Limited. The flow of prompts is typically static and determined beforehand. While the content generated by each step can be dynamic, the overall process is pre-scripted.
- Analogy: Think of an assembly line where each station performs a specific task on the incoming piece. The LLM is like the worker at each station, and the prompts are the instructions for each step.
- Use Cases: Generating marketing copy and then translating it, outlining a document and then writing it, performing multi-step reasoning where each step builds upon the previous one.
This leverage the simple Text-to-Text LLM queries, like the ones tested in the MultiChat project
Example with the Database chat with langchain - Where the LLM flow are defined by us, humans and they execute in 1dimension/sequentially
2. Orchestration Worker:
- Concept: An orchestration worker is a component within a broader system that manages and coordinates the execution of various tasks, which can include interacting with LLMs, calling external APIs, retrieving data, or using other tools.
- Focus: Managing the overall workflow and ensuring that the right components are executed in the correct order and with the necessary data. It often acts as a central controller.
- Intelligence/Autonomy: Can vary. The orchestration logic can be rule-based (deterministic) or involve some level of dynamic decision-making based on the state of the workflow or the output of previous steps. It might decide which LLM or tool to invoke next.
- Analogy: Imagine a conductor leading an orchestra. The conductor (orchestration worker) ensures each section (LLMs, tools, etc.) plays its part at the right time to create a harmonious whole.
- Use Cases: Building chatbots that need to access knowledge bases, call APIs, and generate responses; automating complex business processes involving LLM-powered content generation or data analysis.
Langgraph!
3. Agents:
- Concept: Agents are autonomous or semi-autonomous systems powered by LLMs that can perceive their environment, make decisions, and take actions to achieve specific goals. They can dynamically determine the sequence of steps needed to fulfill a task, including which tools to use and when.
- Focus: Intelligent problem-solving and goal-oriented behavior. Agents can reason, plan, and adapt their actions based on the context and the outcomes of their previous steps.
- Intelligence/Autonomy: High. Agents exhibit a significant degree of autonomy in deciding how to approach a problem and what actions to take. They can use tools (including prompt chains or individual LLM calls) in a flexible and dynamic manner.
- Analogy: Think of a skilled project manager who understands the overall objective and can independently decide which resources to use and in what order to complete the project.
- Use Cases: Building sophisticated conversational AI that can handle complex user requests, perform tasks across different domains (e.g., booking flights, setting reminders), and learn from interactions; creating autonomous problem-solving systems.
OpenAI Agents: https://github.com/JAlcocerT/Multi-Agents/tree/main/OpenAI-Agents
- The core idea of agents is to use a language model to choose a sequence of actions to take.
- In chains, a sequence of actions is hardcoded (in code). In agents, a language model is used as a reasoning engine to determine which actions to take and in which order.
Here’s a table summarizing the key differences:
Feature | Prompt Chaining | Orchestration Worker | Agent |
---|---|---|---|
Core Function | Sequential LLM interactions | Workflow management & coordination | Autonomous problem-solving & action |
Focus | Structuring LLM prompts | Controlling the execution flow | Achieving goals through reasoning & tools |
Autonomy | Low (predefined sequence) | Medium (can have dynamic logic) | High (dynamic decision-making) |
Intelligence | Relies on the LLM at each step | Can have some logic for flow control | Exhibits reasoning and planning |
Tool Usage | Implicit within the prompt sequence | Can manage and invoke various tools | Can dynamically choose and use tools |
In essence:
- Prompt chaining is a technique for structuring how you interact with an LLM.
- An orchestration worker is a system component that manages the broader workflow, which might involve prompt chains.
- An agent is a more sophisticated system that uses an LLM as its “brain” to autonomously figure out how to achieve a goal, potentially utilizing orchestration workers and prompt chains as part of its strategy.
You can think of it as a hierarchy of increasing complexity and autonomy.
Agents often leverage orchestration and might employ prompt chaining as one of their tools or strategies.
Recently I was having a conversation about whats trendy in the Dsc/AI space.
Considering that I come from these…
Its time to have a look to these and put some code together into the Multi Agents project:
- RAGs
- Function calling
- Structured OutputS,…
- Langfuse and MLFlow…
- LangFlow and LangGraph (which supports cyclic Graphs)
- MCP - Model Context protocol
They are all great.
And a must KNOW.
But can we just have LLMs working together?
Actually, this is great tool to have when doing research of some new topic.
π« CAMEL: Finding the Scaling Law of Agents. A multi-agent framework. https://www.camel-ai.org
- The Site
- The Agent_Swarm Code at Github
- License: MIT β€οΈ
Open Source Agents Swarm Projects
LangGraph
AIgent builder framework: alternative to autogen and crewAI.
It is a graph framework and each node is code.
LangGraph | Summarized with Groq π
- Thanks to the related repo - https://github.com/whitew1994WW/LangGraphForBeginners
I have replicated the code here for the REact example. But this fwk can do also reflection agent, router…
LangGraph vs LangChain | Summarized with Groq π
LangChain is a framework that focuses on chaining together multiple language models to generate more accurate and informative responses. By leveraging the strengths of individual models, LangChain creates a robust system capable of tackling complex linguistic tasks. The video highlights LangChain’s ability to handle long-range dependencies, making it an attractive solution for applications that require detailed explanations or storytelling.
LangGraph, on the other hand, takes a more graph-based approach to language understanding. By representing language as a graph, LangGraph can capture relationships between entities, concepts, and relationships, enabling more nuanced and context-aware interactions. The video showcases LangGraph’s strength in handling ambiguous or unclear input, making it an ideal choice for applications that require high accuracy and precision. Key Differences and Applications
LangChain excels in tasks that require creativity and coherence, such as writing stories or generating dialogues. LangGraph, with its graph-based approach, is better suited for tasks that require precision and accuracy, such as question answering or entity disambiguation.
Takeaways
LangChain and LangGraph are two distinct frameworks with unique strengths and weaknesses. LangChain is ideal for applications that require creativity, coherence, and long-range dependencies. LangGraph is better suited for applications that require precision, accuracy, and nuanced language understanding. Understanding the differences between these frameworks is crucial for harnessing the power of AI in various industries.
The Fabric Project
- The Fabric Code at Github
- License: MIT β€οΈ
Fabric is an open-source framework for augmenting humans using AI.
A collection of prompts (patterns) to interact effectively with AI:
- Summarize a video
- Create Prompts for Image Generation
- Answer questions about your code…
We can use Fabric together with Ollama and Groq - Both are OpenAI API Compatible.
Fabric with Ollama
Fabric with Groq
OpenAI Agents
These are openai compatible only…but they bring you observability plugged in (no need to use LangFuse for it)
A lightweight, powerful framework for multi-agent workflows
The Agents SDK is designed to be highly flexible, allowing you to model a wide range of LLM workflows including deterministic flows, iterative loops, and more. See examples in examples/agent_patterns.
AI Agents with LangGraph
LangGraph is a very trendy agent orchestration framework:
LangGraph adds:
- Persistence: memory, human in the loop…
- Streaming
A fantastic video which dives into https://www.anthropic.com/engineering/building-effective-agents
- A workflow: a predefine paths for LLMs calls
- Orchestrator-worker: a intermediate approach
- Agents: the LLMs directs its own actions (normally with tools calls) and with the outputs decide what to do next
Other F/OSS Multi-Agent Projects
The Autogen Project
Camel
π« CAMEL: Finding the Scaling Law of Agents. A multi-agent framework. https://www.camel-ai.org
MetaGPT
MetaGPT Project Details:
- The MetaGPT Docs Site
- The MetaGPT Code at Github
- License: MIT β€οΈ
MetaGPT - A F/OSS Multi Agent Framework that will power your next Projects
CrewAI
- The Project on PyPI
- The CrewAI Source Code at Github
- License: MIT β€οΈ
CrewAI is a Framework that will make easy for us to get Local AI Agents interacting between them.
Interesting Research Tools π
Research Agent
LLM based autonomous agent that conducts local and web research on any topic and generates a comprehensive report with citations - Apache v2
Agents with AutoGen - AG2AI
We have several agent concepts in AG2 to help you build your AI agents.
We introduce the most common ones here:
Conversable Agent: Agents that are able to send messages, receive messages and generate replies using GenAI models, non-GenAI tools, or human inputs.
Human in the loop: Add human input to the conversation
Orchestrating multiple agents: Users can orchestrate multiple agents with built-in conversation patterns such as swarms, group chats, nested chats, sequential chats or customize the orchestration by registering custom reply methods.
Tools: Programs that can be registered, invoked and executed by agents - Agents gain significant utility through tools as they provide access to external data, APIs, and functionality.
Advanced Concepts: AG2 supports more concepts such as structured outputs, rag, code execution, etc.
pip install ag2
#or with the alias!
I got started with PyAutoGen thanks to this .ipynb
:
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
llm_config = {
"config_list": config_list_from_json(env_or_file="OAI_CONFIG_LIST")
}
assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False})
user_proxy.initiate_chat(assistant, message="Plot a chart of NVDA and TESLA stock price change YTD.")
# This initiates an automated chat between the two agents to solve the task
Advanced agentic design patterns
AG2 supports more advanced concepts to help you build your AI agent workflows.
- Structured Output
- Ending a conversation
- Retrieval Augmented Generation (RAG)
- Code Execution
- Tools with Secrets
What mor ecan PyAutoGen do? π
Yes, PyAutoGen can be used to get information about websites and databases, although the methods and complexity vary. Here’s how:
1. Websites:
- Web Scraping:
- AutoGen agents can use Python libraries like
requests
andBeautiful Soup
(orScrapy
) to scrape information from websites. - Agents can be instructed to:
- Download HTML content.
- Parse the HTML to extract specific data.
- Follow links to navigate through the website.
- Example:
- An agent could be asked to get the current price of a product from an e-commerce website.
- Another agent could be used to summarize the content of a news article.
- AutoGen agents can use Python libraries like
- Web APIs:
- If the website provides a public API, agents can use
requests
to interact with the API and retrieve structured data (JSON, XML). - This is generally more reliable and efficient than web scraping.
- If the website provides a public API, agents can use
- Browser Automation:
- For websites that rely heavily on JavaScript or require user interaction, agents can use libraries like
Selenium
orPlaywright
to automate browser actions. - This allows agents to:
- Load web pages.
- Interact with elements (buttons, forms).
- Extract data from dynamic content.
- For websites that rely heavily on JavaScript or require user interaction, agents can use libraries like
2. Databases:
- SQL Queries:
- AutoGen agents can use Python libraries like
sqlite3
,psycopg2
(for PostgreSQL),mysql-connector-python
(for MySQL), orSQLAlchemy
to connect to databases and execute SQL queries. - Agents can be instructed to:
- Connect to a database.
- Formulate SQL queries based on user requests.
- Retrieve and process data from the database.
- Example:
- An agent could be asked to retrieve a list of customers from a database.
- Another agent could be used to calculate aggregate statistics (e.g., average sales).
- AutoGen agents can use Python libraries like
- Database APIs:
- Some databases provide REST APIs or other interfaces that agents can use to interact with the database.
- This can be useful for cloud-based databases or when direct SQL access is not available.
- Langchain integration:
- AutoGen can work in conjunction with langchain, and therefore use any of the database integrations that langchain provides. This can simplify the database interactions.
Key Considerations:
- Security:
- When interacting with websites or databases, it’s crucial to handle credentials and sensitive data securely.
- Avoid hardcoding API keys or database passwords in your code. Use environment variables or secure configuration files.
- Rate Limiting:
- Websites and APIs often have rate limits to prevent abuse.
- Implement mechanisms to handle rate limits and avoid overloading servers.
- Data Parsing:
- Parsing HTML or database results can be complex, especially when dealing with unstructured or inconsistent data.
- Use robust parsing techniques and error handling.
- Ethical Considerations:
- Respect website terms of service and database usage policies.
- Avoid scraping or accessing data that you are not authorized to access.
AutoGen’s Role:
- AutoGen provides the framework for orchestrating the agents that perform these tasks.
- It enables agents to collaborate, communicate, and coordinate their efforts to achieve complex goals involving websites and databases.
- AutoGen can be used to create agents that can decide which tool, or function to use, in order to get the required information from the website or database.
In summary, AutoGen can be a powerful tool for building applications that retrieve and process information from websites and databases, but it’s important to consider the technical and ethical implications of these activities.
Agents with LangChain
You guessed it!
LangChain can also do Agents tricks.
You might know Langchain because of its useful Chains and its RAGs Features
Agents with Llama-Index
LlamaIndex, is designed to work with various tools, data sources, and workflows.
- https://jalcocert.github.io/JAlcocerT/using-langchain-with-pandas-df/#llamaindex-and-pandas-dataframes
- https://jalcocert.github.io/JAlcocerT/how-to-use-rags-with-python/#llama-index
It helps to augment LLMs with context from different data sources such as APIs, SQL databases, PDFs, etc., without being tied to a specific framework.
The ability to integrate multiple agents, tools, and data sources makes it adaptable to various development environments and workflows.
Context augmentation involves providing an LLM with specific data (private or external) to help solve a problem, enhancing the LLM’s ability to generate relevant answers by accessing relevant data.
Agents are knowledge assistants powered by LLMs that perform tasks using various tools, including Retrieval-Augmented Generation (RAG) as one tool in a larger workflow.
Workflows combine multiple agents, tools, and data connectors to create complex, event-driven processes that can automate tasks, reflecting advanced LLM applications.
Agents with PydanticAI
I was using Pydantic this year.
We can say it is a data validation framework, that now it has an AI version
Agent Framework / shim to use Pydantic with LLMs - MIT Licensed!
What it is Pydantic? Data Validation? π
Pydantic is a data validation and settings management library in Python.
Itβs widely used for validating data and ensuring that inputs conform to the expected types and formats.
Chat-DEV
- The ChatDev Repository
- License: Apache v2 β€οΈ
Create Customized Software using Natural Language Idea (through LLM-powered Multi-Agent Collaboration)
Langroid
Example - https://github.com/langroid/langroid/blob/main/examples/basic/chat-search-assistant.py
Harness LLMs with Multi-Agent Programming - MIT LICENSED!
Auto-GPT
AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
CrewAI
You can use Streamlit with CrewAI
Example - https://github.com/tonykipkemboi/trip_planner_agent
CrewAI agents that can plan your vacation.
CrewAI Pandas DF Agent
MetaGPT
MIT | π The Multi-Agent Framework: First AI Software Company, Towards Natural Language Programming
SuperAGI
<β‘οΈ> SuperAGI - A dev-first open source autonomous AI agent framework. Enabling developers to build, manage & run useful autonomous agents quickly and reliably.
Conclusions
Isnt it crazy that we already have: https://jobforagent.com/
Ive also started to use: https://windsurf.com/pricing
curl -fsSL "https://windsurf-stable.codeiumdata.com/wVxQEIWkwPUEAGf3/windsurf.gpg" | sudo gpg --dearmor -o /usr/share/keyrings/windsurf-stable-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/windsurf-stable-archive-keyring.gpg arch=amd64] https://windsurf-stable.codeiumdata.com/wVxQEIWkwPUEAGf3/apt stable main" | sudo tee /etc/apt/sources.list.d/windsurf.list > /dev/null
#sudo apt-get update
sudo apt-get upgrade windsurf
Example with DASH
Breakdown of the libraries in your pyproject.toml
that are commonly used for Artificial Intelligence and Machine Learning:
langchain>=0.3.19
: LangChain is a popular framework for building applications powered by large language models (LLMs). It provides tools for prompt management, model interaction, memory, routing, and more.langchain-community>=0.3.18
: This package contains community-contributed integrations and utilities for LangChain, often including connections to various data sources, tools, and other services relevant to AI applications.langchain-openai>=0.3.7
: This is a specific integration package for LangChain that provides seamless access to OpenAI’s models (like GPT-3.5, GPT-4, etc.) and related services.langgraph>=0.2.74
: LangGraph is a library built on top of LangChain that helps you build conversational agents as graphs of steps, making it easier to design more complex and structured AI interactions.openai>=1.64.0
: This is the official Python library for interacting directly with the OpenAI API. It allows you to access their models for tasks like text generation, translation, code generation, and more.openai-agents>=0.0.7
: This library focuses on building AI agents that can use tools and perform actions in the real world, often in conjunction with large language models.
In summary, the following libraries in your list are the primary AI-related ones:
langchain
langchain-community
langchain-openai
langgraph
openai
openai-agents
The other libraries in your dependencies
list serve different purposes:
langfuse
: An observability platform for language model applications, helping with tracking and debugging.
Make and Docker Compose
make --version #4.3
This is a new version to manage dockerfiles:
docker compose version #v2.35.1
FAQ
Interesting LLMs Architectures
- MoE - Mix of Experts
- MoA - Mix of Agents
- ReACT - Reasoning + Acting
- First they think, then they execute an action inside an environment
More about Solara - ReACT Framework π
A Pure Python, React-style Framework for Scaling Your Jupyter and Web Apps
Build a Python WebApp in 3 minutes with Solara (Streamlit Alternative)
How to Provide Web Context to the Agents
Via Scrapping
https://jalcocert.github.io/JAlcocerT/comparing-rag-and-use-cases/#web-scrapping https://github.com/JAlcocerT/Data-Chat/tree/main/LangChain/web
Via Web Search
For example, with duckduckgo:
pip install duckduckgo-search
from duckduckgo_search import ddg_answers
# Perform search query
results = ddg_answers('Python programming')
# Output the search results
print(results)
I improved with this one…
Python deps management - venv uv poetry π
Let’s break down the pros and cons of venv
, uv
, and poetry
for managing Python projects:
venv
(Built-in Virtual Environment)
Pros:
- Built-in: It comes standard with Python 3, so no extra installation is required. This makes it readily available and easy to start with for basic isolation.
- Lightweight and Simple:
venv
focuses solely on creating isolated Python environments. It’s straightforward to use for basic dependency isolation. - Universally Understood: As the standard, it’s the most widely understood and compatible with other Python tools and workflows. Most deployment environments and CI/CD systems have good support for
venv
-based environments. - Good for Simple Projects: For smaller projects with straightforward dependencies,
venv
can be sufficient.
Cons:
- Basic Functionality: It only manages the Python environment itself. It doesn’t handle dependency management, locking, or packaging in a comprehensive way. You typically use
pip
andrequirements.txt
alongside it. - Manual Dependency Management: You need to manually track and update your dependencies in
requirements.txt
. This can be error-prone and doesn’t guarantee reproducible builds across different environments. - No Dependency Resolution:
pip
(when used withvenv
andrequirements.txt
) doesn’t have a built-in dependency resolver that ensures compatible versions of your dependencies are installed. This can lead to dependency conflicts. - Limited Packaging Support:
venv
itself doesn’t provide tools for building and publishing Python packages.
uv
(Fast Package Installer and Resolver)
Pros:
- Extremely Fast:
uv
is written in Rust and is significantly faster thanpip
for installing and resolving dependencies. This can drastically reduce the time spent setting up and managing environments. - PEP 621 Support: It understands and utilizes the standard
pyproject.toml
file for project configuration, making it compatible with other modern Python tooling. - Virtual Environment Management:
uv
can create and manage virtual environments, often placing the.venv
directory directly in the project root by default, which some users prefer. - Faster Dependency Resolution: Its dependency resolver is also much quicker than
pip
’s, leading to faster and more efficient environment setup. - Potentially Replaces
pip
andvenv
:uv
aims to be a faster alternative to bothpip
andvenv
for many common tasks.
Cons:
- Relatively New: While promising,
uv
is a newer tool compared tovenv
andpoetry
. Its ecosystem and integration with all existing tools might still be evolving. - Limited Feature Set Compared to Poetry: While it excels at environment management and package installation, it doesn’t yet have the full suite of features that
poetry
offers for packaging, publishing, and dependency management. For example, it currently doesn’t have its own build backend. - Learning Curve (for some): While designed to be easy to use, switching to a new tool always involves a slight learning curve.
- Not a Complete Replacement for All Workflows Yet: Depending on your specific needs, especially around complex packaging and publishing workflows, you might still need other tools in conjunction with
uv
.
poetry
(Comprehensive Dependency Management and Packaging Tool)
Pros:
- Comprehensive Solution:
poetry
provides a complete solution for dependency management, packaging, and publishing. It manages your virtual environment, dependencies, build process, and package publishing. - Dependency Resolution: It has a sophisticated dependency resolver that automatically finds compatible versions of your dependencies, preventing conflicts and ensuring reproducible builds with the
poetry.lock
file. pyproject.toml
Standard: It uses thepyproject.toml
file (PEP 621) to store project metadata and dependencies, which is becoming the standard in the Python ecosystem.- Reproducible Builds: The
poetry.lock
file ensures that everyone working on the project uses the exact same versions of dependencies. - Packaging and Publishing:
poetry
simplifies the process of building and publishing your Python packages to PyPI. - Dependency Groups: It allows you to define different groups of dependencies (e.g., development dependencies), keeping your main project dependencies clean.
- Script Management: You can define scripts within
pyproject.toml
to automate common development tasks.
Cons:
- Overkill for Simple Projects: For very small and simple projects with few dependencies,
poetry
’s comprehensive nature might feel like overkill. - Can be Slower Than
uv
for Basic Tasks: While its dependency resolver is powerful, the initial environment setup and dependency installation can sometimes be slower thanuv
. - Abstraction Layer: Some users prefer the more direct control offered by
venv
andpip
.poetry
introduces its own layer of abstraction, which might feel less transparent to some. - Lock File Conflicts: While the lock file ensures reproducibility, merge conflicts in the
poetry.lock
file can sometimes be challenging to resolve.
Here’s a table summarizing the key differences:
Feature | venv | uv | poetry |
---|---|---|---|
Environment Mgmt. | Basic creation/activation | Fast creation/activation | Integrated |
Package Mgmt. | Relies on pip | Fast installation/resolution | Integrated, with dependency resolution |
Dependency Resolution | None (via standard pip ) | Fast and efficient | Robust and automatic |
Locking | Requires manual pip freeze > requirements.txt | Uses pyproject.toml for constraints | poetry.lock for reproducible builds |
Packaging | Requires separate tools | Limited | Built-in |
Speed | Standard pip speed | Very fast | Can be slower for initial setup |
Complexity | Simplest | Moderate | More comprehensive |
Standard Compliance | Basic | Good (pyproject.toml ) | Excellent (pyproject.toml , PEP 517) |
Maturity | Long-established standard | Relatively new | Mature and widely adopted |
pyproject.toml
to define your project’s core metadata (name, version, authors) and your direct dependencies.They both understand the [project] and [project.dependencies] sections to know which top-level packages your project requires.
Which one should you choose?
venv
: Best for very simple projects or when you need maximum flexibility and direct control and are comfortable managing dependencies manually withpip
andrequirements.txt
.uv
: A great option if speed is a primary concern and you want a faster alternative topip
andvenv
for environment management and dependency installation. It’s particularly appealing for its performance and adherence to modern standards.poetry
: Ideal for larger projects, libraries, and applications where reproducible builds, robust dependency management, and streamlined packaging are crucial. It offers a more opinionated but comprehensive workflow.
Ultimately, the best choice depends on the specific needs and complexity of your Python projects and your personal preferences.
You might even find yourself using a combination of these tools in different situations. For instance, you might use uv
for its speed in setting up environments for projects managed by poetry
.