AI Multi Agents: PyAutogen, LangGraph,...

AI Multi Agents: PyAutogen, LangGraph,...

April 10, 2025

The Agentic Ecosystem

Lately everyone talks about agents

And most likely than not, you will have found already these frameworks: LangGraph vs. CrewAI vs. AutoGen vs. OpenAI Agents…

But lets get few concepts in place first.

Prompt chaining, orchestration workers, and agents are distinct but related concepts in the realm of building complex applications with Large Language Models (LLMs).

1. Prompt Chaining:

  • Concept: Prompt chaining involves linking multiple LLM prompts sequentially. The output of one prompt serves as the input for the next, allowing you to break down complex tasks into smaller, manageable steps.
  • Focus: Primarily on structuring the interaction with one or more LLMs through a predefined sequence of prompts to achieve a specific outcome.
  • Intelligence/Autonomy: Limited. The flow of prompts is typically static and determined beforehand. While the content generated by each step can be dynamic, the overall process is pre-scripted.
  • Analogy: Think of an assembly line where each station performs a specific task on the incoming piece. The LLM is like the worker at each station, and the prompts are the instructions for each step.
  • Use Cases: Generating marketing copy and then translating it, outlining a document and then writing it, performing multi-step reasoning where each step builds upon the previous one.

This leverage the simple Text-to-Text LLM queries, like the ones tested in the MultiChat project

Example with the Database chat with langchain - Where the LLM flow are defined by us, humans and they execute in 1dimension/sequentially

Open in Google Colab

ℹ️
With LangChain it was also possible to do agent stuff as per agent_executor. But its going to be depricated in favour of langgraph

2. Orchestration Worker:

  • Concept: An orchestration worker is a component within a broader system that manages and coordinates the execution of various tasks, which can include interacting with LLMs, calling external APIs, retrieving data, or using other tools.
  • Focus: Managing the overall workflow and ensuring that the right components are executed in the correct order and with the necessary data. It often acts as a central controller.
  • Intelligence/Autonomy: Can vary. The orchestration logic can be rule-based (deterministic) or involve some level of dynamic decision-making based on the state of the workflow or the output of previous steps. It might decide which LLM or tool to invoke next.
  • Analogy: Imagine a conductor leading an orchestra. The conductor (orchestration worker) ensures each section (LLMs, tools, etc.) plays its part at the right time to create a harmonious whole.
  • Use Cases: Building chatbots that need to access knowledge bases, call APIs, and generate responses; automating complex business processes involving LLM-powered content generation or data analysis.

Langgraph!

ℹ️
With langgraph we can define this kind of stategraphs, as seen on the local deep researcher

3. Agents:

  • Concept: Agents are autonomous or semi-autonomous systems powered by LLMs that can perceive their environment, make decisions, and take actions to achieve specific goals. They can dynamically determine the sequence of steps needed to fulfill a task, including which tools to use and when.
  • Focus: Intelligent problem-solving and goal-oriented behavior. Agents can reason, plan, and adapt their actions based on the context and the outcomes of their previous steps.
  • Intelligence/Autonomy: High. Agents exhibit a significant degree of autonomy in deciding how to approach a problem and what actions to take. They can use tools (including prompt chains or individual LLM calls) in a flexible and dynamic manner.
  • Analogy: Think of a skilled project manager who understands the overall objective and can independently decide which resources to use and in what order to complete the project.
  • Use Cases: Building sophisticated conversational AI that can handle complex user requests, perform tasks across different domains (e.g., booking flights, setting reminders), and learn from interactions; creating autonomous problem-solving systems.

OpenAI Agents: https://github.com/JAlcocerT/Multi-Agents/tree/main/OpenAI-Agents

ℹ️
  • The core idea of agents is to use a language model to choose a sequence of actions to take.
  • In chains, a sequence of actions is hardcoded (in code). In agents, a language model is used as a reasoning engine to determine which actions to take and in which order.

Here’s a table summarizing the key differences:

FeaturePrompt ChainingOrchestration WorkerAgent
Core FunctionSequential LLM interactionsWorkflow management & coordinationAutonomous problem-solving & action
FocusStructuring LLM promptsControlling the execution flowAchieving goals through reasoning & tools
AutonomyLow (predefined sequence)Medium (can have dynamic logic)High (dynamic decision-making)
IntelligenceRelies on the LLM at each stepCan have some logic for flow controlExhibits reasoning and planning
Tool UsageImplicit within the prompt sequenceCan manage and invoke various toolsCan dynamically choose and use tools

In essence:

  • Prompt chaining is a technique for structuring how you interact with an LLM.
  • An orchestration worker is a system component that manages the broader workflow, which might involve prompt chains.
  • An agent is a more sophisticated system that uses an LLM as its “brain” to autonomously figure out how to achieve a goal, potentially utilizing orchestration workers and prompt chains as part of its strategy.

You can think of it as a hierarchy of increasing complexity and autonomy.

Agents often leverage orchestration and might employ prompt chaining as one of their tools or strategies.

Recently I was having a conversation about whats trendy in the Dsc/AI space.

Considering that I come from these…

Its time to have a look to these and put some code together into the Multi Agents project:

  1. RAGs
  2. Function calling
  3. Structured OutputS,…
  4. Langfuse and MLFlow
  5. LangFlow and LangGraph (which supports cyclic Graphs)
  6. MCP - Model Context protocol

They are all great.

And a must KNOW.

But can we just have LLMs working together?

Actually, this is great tool to have when doing research of some new topic.

  1. Fabric
  2. Autogen
  3. CrewAI
  4. MetaGPT
  5. Camel
  6. LangGraph
  7. OpenAI Agents

🐫 CAMEL: Finding the Scaling Law of Agents. A multi-agent framework. https://www.camel-ai.org

Open Source Agents Swarm Projects

LangGraph

AIgent builder framework: alternative to autogen and crewAI.

It is a graph framework and each node is code.

LangGraph | Summarized with Groq πŸ“Œ

I have replicated the code here for the REact example. But this fwk can do also reflection agent, router…

LangGraph vs LangChain | Summarized with Groq πŸ“Œ

LangChain is a framework that focuses on chaining together multiple language models to generate more accurate and informative responses. By leveraging the strengths of individual models, LangChain creates a robust system capable of tackling complex linguistic tasks. The video highlights LangChain’s ability to handle long-range dependencies, making it an attractive solution for applications that require detailed explanations or storytelling.

LangGraph, on the other hand, takes a more graph-based approach to language understanding. By representing language as a graph, LangGraph can capture relationships between entities, concepts, and relationships, enabling more nuanced and context-aware interactions. The video showcases LangGraph’s strength in handling ambiguous or unclear input, making it an ideal choice for applications that require high accuracy and precision. Key Differences and Applications

LangChain excels in tasks that require creativity and coherence, such as writing stories or generating dialogues. LangGraph, with its graph-based approach, is better suited for tasks that require precision and accuracy, such as question answering or entity disambiguation.

Takeaways

LangChain and LangGraph are two distinct frameworks with unique strengths and weaknesses. LangChain is ideal for applications that require creativity, coherence, and long-range dependencies. LangGraph is better suited for applications that require precision, accuracy, and nuanced language understanding. Understanding the differences between these frameworks is crucial for harnessing the power of AI in various industries.

The Fabric Project

Fabric is an open-source framework for augmenting humans using AI.

A collection of prompts (patterns) to interact effectively with AI:

  • Summarize a video
  • Create Prompts for Image Generation
  • Answer questions about your code…

We can use Fabric together with Ollama and Groq - Both are OpenAI API Compatible.

Fabric with Ollama

Fabric with Groq

OpenAI Agents

These are openai compatible only…but they bring you observability plugged in (no need to use LangFuse for it)

A lightweight, powerful framework for multi-agent workflows

The Agents SDK is designed to be highly flexible, allowing you to model a wide range of LLM workflows including deterministic flows, iterative loops, and more. See examples in examples/agent_patterns.

AI Agents with LangGraph

LangGraph is a very trendy agent orchestration framework:

LangGraph adds:

  1. Persistence: memory, human in the loop…
  2. Streaming

A fantastic video which dives into https://www.anthropic.com/engineering/building-effective-agents

  • A workflow: a predefine paths for LLMs calls
  • Orchestrator-worker: a intermediate approach
  • Agents: the LLMs directs its own actions (normally with tools calls) and with the outputs decide what to do next

Other F/OSS Multi-Agent Projects

Star History Chart

The Autogen Project

Camel

🐫 CAMEL: Finding the Scaling Law of Agents. A multi-agent framework. https://www.camel-ai.org

MetaGPT

MetaGPT Project Details:

MetaGPT - A F/OSS Multi Agent Framework that will power your next Projects

CrewAI

CrewAI Project

CrewAI is a Framework that will make easy for us to get Local AI Agents interacting between them.

Interesting Research Tools πŸ“Œ

Research Agent

LLM based autonomous agent that conducts local and web research on any topic and generates a comprehensive report with citations - Apache v2

Star History Chart

ℹ️
Im testing these tools at the MultiAgents Repo πŸ’»

Agents with AutoGen - AG2AI

We have several agent concepts in AG2 to help you build your AI agents.

We introduce the most common ones here:

  • Conversable Agent: Agents that are able to send messages, receive messages and generate replies using GenAI models, non-GenAI tools, or human inputs.

  • Human in the loop: Add human input to the conversation

  • Orchestrating multiple agents: Users can orchestrate multiple agents with built-in conversation patterns such as swarms, group chats, nested chats, sequential chats or customize the orchestration by registering custom reply methods.

  • Tools: Programs that can be registered, invoked and executed by agents - Agents gain significant utility through tools as they provide access to external data, APIs, and functionality.

Advanced Concepts: AG2 supports more concepts such as structured outputs, rag, code execution, etc.

pip install ag2
#or with the alias!

I got started with PyAutoGen thanks to this .ipynb:

from autogen import AssistantAgent, UserProxyAgent, config_list_from_json

llm_config = {
    "config_list": config_list_from_json(env_or_file="OAI_CONFIG_LIST")
}

assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False})
user_proxy.initiate_chat(assistant, message="Plot a chart of NVDA and TESLA stock price change YTD.")
# This initiates an automated chat between the two agents to solve the task

Advanced agentic design patterns

AG2 supports more advanced concepts to help you build your AI agent workflows.

  • Structured Output
  • Ending a conversation
  • Retrieval Augmented Generation (RAG)
  • Code Execution
  • Tools with Secrets
What mor ecan PyAutoGen do? πŸ“Œ

Yes, PyAutoGen can be used to get information about websites and databases, although the methods and complexity vary. Here’s how:

1. Websites:

  • Web Scraping:
    • AutoGen agents can use Python libraries like requests and Beautiful Soup (or Scrapy) to scrape information from websites.
    • Agents can be instructed to:
      • Download HTML content.
      • Parse the HTML to extract specific data.
      • Follow links to navigate through the website.
    • Example:
      • An agent could be asked to get the current price of a product from an e-commerce website.
      • Another agent could be used to summarize the content of a news article.
  • Web APIs:
    • If the website provides a public API, agents can use requests to interact with the API and retrieve structured data (JSON, XML).
    • This is generally more reliable and efficient than web scraping.
  • Browser Automation:
    • For websites that rely heavily on JavaScript or require user interaction, agents can use libraries like Selenium or Playwright to automate browser actions.
    • This allows agents to:
      • Load web pages.
      • Interact with elements (buttons, forms).
      • Extract data from dynamic content.

2. Databases:

  • SQL Queries:
    • AutoGen agents can use Python libraries like sqlite3, psycopg2 (for PostgreSQL), mysql-connector-python (for MySQL), or SQLAlchemy to connect to databases and execute SQL queries.
    • Agents can be instructed to:
      • Connect to a database.
      • Formulate SQL queries based on user requests.
      • Retrieve and process data from the database.
    • Example:
      • An agent could be asked to retrieve a list of customers from a database.
      • Another agent could be used to calculate aggregate statistics (e.g., average sales).
  • Database APIs:
    • Some databases provide REST APIs or other interfaces that agents can use to interact with the database.
    • This can be useful for cloud-based databases or when direct SQL access is not available.
  • Langchain integration:
    • AutoGen can work in conjunction with langchain, and therefore use any of the database integrations that langchain provides. This can simplify the database interactions.

Key Considerations:

  • Security:
    • When interacting with websites or databases, it’s crucial to handle credentials and sensitive data securely.
    • Avoid hardcoding API keys or database passwords in your code. Use environment variables or secure configuration files.
  • Rate Limiting:
    • Websites and APIs often have rate limits to prevent abuse.
    • Implement mechanisms to handle rate limits and avoid overloading servers.
  • Data Parsing:
    • Parsing HTML or database results can be complex, especially when dealing with unstructured or inconsistent data.
    • Use robust parsing techniques and error handling.
  • Ethical Considerations:
    • Respect website terms of service and database usage policies.
    • Avoid scraping or accessing data that you are not authorized to access.

AutoGen’s Role:

  • AutoGen provides the framework for orchestrating the agents that perform these tasks.
  • It enables agents to collaborate, communicate, and coordinate their efforts to achieve complex goals involving websites and databases.
  • AutoGen can be used to create agents that can decide which tool, or function to use, in order to get the required information from the website or database.

In summary, AutoGen can be a powerful tool for building applications that retrieve and process information from websites and databases, but it’s important to consider the technical and ethical implications of these activities.

Agents with LangChain

You guessed it!

LangChain can also do Agents tricks.

You might know Langchain because of its useful Chains and its RAGs Features

ℹ️
LangChain as RAG can chat with CSV & DBs. Also with PDFs

Agents with Llama-Index

LlamaIndex, is designed to work with various tools, data sources, and workflows.

It helps to augment LLMs with context from different data sources such as APIs, SQL databases, PDFs, etc., without being tied to a specific framework.

The ability to integrate multiple agents, tools, and data sources makes it adaptable to various development environments and workflows.

Context augmentation involves providing an LLM with specific data (private or external) to help solve a problem, enhancing the LLM’s ability to generate relevant answers by accessing relevant data.

Agents are knowledge assistants powered by LLMs that perform tasks using various tools, including Retrieval-Augmented Generation (RAG) as one tool in a larger workflow.

Workflows combine multiple agents, tools, and data connectors to create complex, event-driven processes that can automate tasks, reflecting advanced LLM applications.

Agents with PydanticAI

I was using Pydantic this year.

We can say it is a data validation framework, that now it has an AI version

Agent Framework / shim to use Pydantic with LLMs - MIT Licensed!

What it is Pydantic? Data Validation? πŸ“Œ

Pydantic is a data validation and settings management library in Python.

It’s widely used for validating data and ensuring that inputs conform to the expected types and formats.

Chat-DEV

Create Customized Software using Natural Language Idea (through LLM-powered Multi-Agent Collaboration)

Langroid

Example - https://github.com/langroid/langroid/blob/main/examples/basic/chat-search-assistant.py

Harness LLMs with Multi-Agent Programming - MIT LICENSED!

Auto-GPT

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

CrewAI

You can use Streamlit with CrewAI

Example - https://github.com/tonykipkemboi/trip_planner_agent

CrewAI agents that can plan your vacation.

CrewAI Pandas DF Agent

MetaGPT

MIT | 🌟 The Multi-Agent Framework: First AI Software Company, Towards Natural Language Programming

MetaGPT

SuperAGI

<⚑️> SuperAGI - A dev-first open source autonomous AI agent framework. Enabling developers to build, manage & run useful autonomous agents quickly and reliably.


Conclusions

Isnt it crazy that we already have: https://jobforagent.com/

curl -fsSL "https://windsurf-stable.codeiumdata.com/wVxQEIWkwPUEAGf3/windsurf.gpg" | sudo gpg --dearmor -o /usr/share/keyrings/windsurf-stable-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/windsurf-stable-archive-keyring.gpg arch=amd64] https://windsurf-stable.codeiumdata.com/wVxQEIWkwPUEAGf3/apt stable main" | sudo tee /etc/apt/sources.list.d/windsurf.list > /dev/null

#sudo apt-get update
sudo apt-get upgrade windsurf

Open in Google Colab

Star History Chart

Example with DASH

Breakdown of the libraries in your pyproject.toml that are commonly used for Artificial Intelligence and Machine Learning:

  • langchain>=0.3.19: LangChain is a popular framework for building applications powered by large language models (LLMs). It provides tools for prompt management, model interaction, memory, routing, and more.
  • langchain-community>=0.3.18: This package contains community-contributed integrations and utilities for LangChain, often including connections to various data sources, tools, and other services relevant to AI applications.
  • langchain-openai>=0.3.7: This is a specific integration package for LangChain that provides seamless access to OpenAI’s models (like GPT-3.5, GPT-4, etc.) and related services.
  • langgraph>=0.2.74: LangGraph is a library built on top of LangChain that helps you build conversational agents as graphs of steps, making it easier to design more complex and structured AI interactions.
  • openai>=1.64.0: This is the official Python library for interacting directly with the OpenAI API. It allows you to access their models for tasks like text generation, translation, code generation, and more.
  • openai-agents>=0.0.7: This library focuses on building AI agents that can use tools and perform actions in the real world, often in conjunction with large language models.

In summary, the following libraries in your list are the primary AI-related ones:

  • langchain
  • langchain-community
  • langchain-openai
  • langgraph
  • openai
  • openai-agents

The other libraries in your dependencies list serve different purposes:

  • langfuse: An observability platform for language model applications, helping with tracking and debugging.

Make and Docker Compose

make --version #4.3

This is a new version to manage dockerfiles:

docker compose version #v2.35.1

FAQ

Interesting LLMs Architectures

  • MoE - Mix of Experts
  • MoA - Mix of Agents
  • ReACT - Reasoning + Acting
    • First they think, then they execute an action inside an environment
More about Solara - ReACT Framework πŸ“Œ

A Pure Python, React-style Framework for Scaling Your Jupyter and Web Apps

Build a Python WebApp in 3 minutes with Solara (Streamlit Alternative)

How to Provide Web Context to the Agents

Via Scrapping

https://jalcocert.github.io/JAlcocerT/comparing-rag-and-use-cases/#web-scrapping https://github.com/JAlcocerT/Data-Chat/tree/main/LangChain/web

Via Web Search

For example, with duckduckgo:

pip install duckduckgo-search
from duckduckgo_search import ddg_answers

# Perform search query
results = ddg_answers('Python programming')

# Output the search results
print(results)

I improved with this one…

Python deps management - venv uv poetry πŸ“Œ

Let’s break down the pros and cons of venv, uv, and poetry for managing Python projects:

venv (Built-in Virtual Environment)

Pros:

  • Built-in: It comes standard with Python 3, so no extra installation is required. This makes it readily available and easy to start with for basic isolation.
  • Lightweight and Simple: venv focuses solely on creating isolated Python environments. It’s straightforward to use for basic dependency isolation.
  • Universally Understood: As the standard, it’s the most widely understood and compatible with other Python tools and workflows. Most deployment environments and CI/CD systems have good support for venv-based environments.
  • Good for Simple Projects: For smaller projects with straightforward dependencies, venv can be sufficient.

Cons:

  • Basic Functionality: It only manages the Python environment itself. It doesn’t handle dependency management, locking, or packaging in a comprehensive way. You typically use pip and requirements.txt alongside it.
  • Manual Dependency Management: You need to manually track and update your dependencies in requirements.txt. This can be error-prone and doesn’t guarantee reproducible builds across different environments.
  • No Dependency Resolution: pip (when used with venv and requirements.txt) doesn’t have a built-in dependency resolver that ensures compatible versions of your dependencies are installed. This can lead to dependency conflicts.
  • Limited Packaging Support: venv itself doesn’t provide tools for building and publishing Python packages.

uv (Fast Package Installer and Resolver)

Pros:

  • Extremely Fast: uv is written in Rust and is significantly faster than pip for installing and resolving dependencies. This can drastically reduce the time spent setting up and managing environments.
  • PEP 621 Support: It understands and utilizes the standard pyproject.toml file for project configuration, making it compatible with other modern Python tooling.
  • Virtual Environment Management: uv can create and manage virtual environments, often placing the .venv directory directly in the project root by default, which some users prefer.
  • Faster Dependency Resolution: Its dependency resolver is also much quicker than pip’s, leading to faster and more efficient environment setup.
  • Potentially Replaces pip and venv: uv aims to be a faster alternative to both pip and venv for many common tasks.

Cons:

  • Relatively New: While promising, uv is a newer tool compared to venv and poetry. Its ecosystem and integration with all existing tools might still be evolving.
  • Limited Feature Set Compared to Poetry: While it excels at environment management and package installation, it doesn’t yet have the full suite of features that poetry offers for packaging, publishing, and dependency management. For example, it currently doesn’t have its own build backend.
  • Learning Curve (for some): While designed to be easy to use, switching to a new tool always involves a slight learning curve.
  • Not a Complete Replacement for All Workflows Yet: Depending on your specific needs, especially around complex packaging and publishing workflows, you might still need other tools in conjunction with uv.

poetry (Comprehensive Dependency Management and Packaging Tool)

Pros:

  • Comprehensive Solution: poetry provides a complete solution for dependency management, packaging, and publishing. It manages your virtual environment, dependencies, build process, and package publishing.
  • Dependency Resolution: It has a sophisticated dependency resolver that automatically finds compatible versions of your dependencies, preventing conflicts and ensuring reproducible builds with the poetry.lock file.
  • pyproject.toml Standard: It uses the pyproject.toml file (PEP 621) to store project metadata and dependencies, which is becoming the standard in the Python ecosystem.
  • Reproducible Builds: The poetry.lock file ensures that everyone working on the project uses the exact same versions of dependencies.
  • Packaging and Publishing: poetry simplifies the process of building and publishing your Python packages to PyPI.
  • Dependency Groups: It allows you to define different groups of dependencies (e.g., development dependencies), keeping your main project dependencies clean.
  • Script Management: You can define scripts within pyproject.toml to automate common development tasks.

Cons:

  • Overkill for Simple Projects: For very small and simple projects with few dependencies, poetry’s comprehensive nature might feel like overkill.
  • Can be Slower Than uv for Basic Tasks: While its dependency resolver is powerful, the initial environment setup and dependency installation can sometimes be slower than uv.
  • Abstraction Layer: Some users prefer the more direct control offered by venv and pip. poetry introduces its own layer of abstraction, which might feel less transparent to some.
  • Lock File Conflicts: While the lock file ensures reproducibility, merge conflicts in the poetry.lock file can sometimes be challenging to resolve.

Here’s a table summarizing the key differences:

Featurevenvuvpoetry
Environment Mgmt.Basic creation/activationFast creation/activationIntegrated
Package Mgmt.Relies on pipFast installation/resolutionIntegrated, with dependency resolution
Dependency ResolutionNone (via standard pip)Fast and efficientRobust and automatic
LockingRequires manual pip freeze > requirements.txtUses pyproject.toml for constraintspoetry.lock for reproducible builds
PackagingRequires separate toolsLimitedBuilt-in
SpeedStandard pip speedVery fastCan be slower for initial setup
ComplexitySimplestModerateMore comprehensive
Standard ComplianceBasicGood (pyproject.toml)Excellent (pyproject.toml, PEP 517)
MaturityLong-established standardRelatively newMature and widely adopted
ℹ️
Both uv and poetry use pyproject.toml to define your project’s core metadata (name, version, authors) and your direct dependencies.

They both understand the [project] and [project.dependencies] sections to know which top-level packages your project requires.

Which one should you choose?

  • venv: Best for very simple projects or when you need maximum flexibility and direct control and are comfortable managing dependencies manually with pip and requirements.txt.
  • uv: A great option if speed is a primary concern and you want a faster alternative to pip and venv for environment management and dependency installation. It’s particularly appealing for its performance and adherence to modern standards.
  • poetry: Ideal for larger projects, libraries, and applications where reproducible builds, robust dependency management, and streamlined packaging are crucial. It offers a more opinionated but comprehensive workflow.

Ultimately, the best choice depends on the specific needs and complexity of your Python projects and your personal preferences.

You might even find yourself using a combination of these tools in different situations. For instance, you might use uv for its speed in setting up environments for projects managed by poetry.