AI Multi Agents: PyAutogen, LangGraph,...

Couple of years back there where prjects like https://github.com/unconv/gpt-autopilot

MIT | A GPT-4 powered AI agent that can create full projects with iterative prompting

A ChatGPT API powered Python script that can build almost anything with the power of the Function Calling. Just tell it what you want to build, and it will build it and ask you clarifying questions along the way.

And look at this now…

The Agentic Ecosystem

Lately everyone talks about agents…

And most likely than not, you will have found already these frameworks: LangGraph vs. CrewAI vs. AutoGen vs. OpenAI Agents…

But lets get few concepts in place first.

Prompt chaining, orchestration workers, and agents are distinct but related concepts in the realm of building complex applications with Large Language Models (LLMs).

1. Prompt Chaining:

• Concept: Prompt chaining involves linking multiple LLM prompts sequentially. The output of one prompt serves as the input for the next, allowing you to break down complex tasks into smaller, manageable steps.
• Focus: Primarily on structuring the interaction with one or more LLMs through a predefined sequence of prompts to achieve a specific outcome.
• Intelligence/Autonomy: Limited. The flow of prompts is typically static and determined beforehand. While the content generated by each step can be dynamic, the overall process is pre-scripted.
• Analogy: Think of an assembly line where each station performs a specific task on the incoming piece. The LLM is like the worker at each station, and the prompts are the instructions for each step.
• Use Cases: Generating marketing copy and then translating it, outlining a document and then writing it, performing multi-step reasoning where each step builds upon the previous one.

This leverage the simple Text-to-Text LLM queries, like the ones tested in the MultiChat project

Example with the Database chat with langchain - Where the LLM flow are defined by us, humans and they execute in 1dimension/sequentially

2. Orchestration Worker:

• Concept: An orchestration worker is a component within a broader system that manages and coordinates the execution of various tasks, which can include interacting with LLMs, calling external APIs, retrieving data, or using other tools.
• Focus: Managing the overall workflow and ensuring that the right components are executed in the correct order and with the necessary data. It often acts as a central controller.
• Intelligence/Autonomy: Can vary. The orchestration logic can be rule-based (deterministic) or involve some level of dynamic decision-making based on the state of the workflow or the output of previous steps. It might decide which LLM or tool to invoke next.
• Analogy: Imagine a conductor leading an orchestra. The conductor (orchestration worker) ensures each section (LLMs, tools, etc.) plays its part at the right time to create a harmonious whole.
• Use Cases: Building chatbots that need to access knowledge bases, call APIs, and generate responses; automating complex business processes involving LLM-powered content generation or data analysis.

Langgraph!

3. Agents:

• Concept: Agents are autonomous or semi-autonomous systems powered by LLMs that can perceive their environment, make decisions, and take actions to achieve specific goals. They can dynamically determine the sequence of steps needed to fulfill a task, including which tools to use and when.
• Focus: Intelligent problem-solving and goal-oriented behavior. Agents can reason, plan, and adapt their actions based on the context and the outcomes of their previous steps.
• Intelligence/Autonomy: High. Agents exhibit a significant degree of autonomy in deciding how to approach a problem and what actions to take. They can use tools (including prompt chains or individual LLM calls) in a flexible and dynamic manner.
• Analogy: Think of a skilled project manager who understands the overall objective and can independently decide which resources to use and in what order to complete the project.
• Use Cases: Building sophisticated conversational AI that can handle complex user requests, perform tasks across different domains (e.g., booking flights, setting reminders), and learn from interactions; creating autonomous problem-solving systems.

OpenAI Agents: https://github.com/JAlcocerT/Multi-Agents/tree/main/OpenAI-Agents

Here’s a table summarizing the key differences:

In essence:

• Prompt chaining is a technique for structuring how you interact with an LLM.
• An orchestration worker is a system component that manages the broader workflow, which might involve prompt chains.
• An agent is a more sophisticated system that uses an LLM as its “brain” to autonomously figure out how to achieve a goal, potentially utilizing orchestration workers and prompt chains as part of its strategy.

You can think of it as a hierarchy of increasing complexity and autonomy.

Agents often leverage orchestration and might employ prompt chaining as one of their tools or strategies.

RAG 101

My Fav RAGs Post

Multi-Agents

Tinkering with Python and MultiAgents

Recently I was having a conversation about whats trendy in the Dsc/AI space.

Considering that I come from these…

Its time to have a look to these and put some code together into the Multi Agents project:

1. RAGs
2. Function calling
3. Structured OutputS,…
4. Langfuse and MLFlow…
5. LangFlow and LangGraph (which supports cyclic Graphs)
6. MCP - Model Context protocol

They are all great.

And a must KNOW.

But can we just have LLMs working together?

Actually, this is great tool to have when doing research of some new topic.

1. Fabric
2. Autogen
3. CrewAI
4. MetaGPT
5. Camel
6. LangGraph
7. OpenAI Agents

• https://github.com/camel-ai/camel

🐫 CAMEL: Finding the Scaling Law of Agents. A multi-agent framework. https://www.camel-ai.org

• The Site
• The Agent_Swarm Code at Github
  • License: MIT ❤️

Open Source Agents Swarm Projects

LangGraph

AIgent builder framework: alternative to autogen and crewAI.

It is a graph framework and each node is code.

• https://www.langchain.com/langgraph

  • https://pypi.org/project/langgraph/

• Thanks to the related repo - https://github.com/whitew1994WW/LangGraphForBeginners

I have replicated the code here for the REact example. But this fwk can do also reflection agent, router…

The Fabric Project

• The Fabric Code at Github
  • License: MIT ❤️

Fabric is an open-source framework for augmenting humans using AI.

A collection of prompts (patterns) to interact effectively with AI:

• Summarize a video
• Create Prompts for Image Generation
• Answer questions about your code…

We can use Fabric together with Ollama and Groq - Both are OpenAI API Compatible.

Fabric with Ollama

Fabric with Groq

OpenAI Agents

These are openai compatible only…but they bring you observability plugged in (no need to use LangFuse for it)

• https://github.com/openai/openai-agents-python
  • https://pypi.org/project/openai-agents/

A lightweight, powerful framework for multi-agent workflows

The Agents SDK is designed to be highly flexible, allowing you to model a wide range of LLM workflows including deterministic flows, iterative loops, and more. See examples in examples/agent_patterns.

AI Agents with LangGraph

LangGraph is a very trendy agent orchestration framework:

• https://www.deeplearning.ai/short-courses/ai-agents-in-langgraph/

LangGraph adds:

1. Persistence: memory, human in the loop…
2. Streaming

A fantastic video which dives into https://www.anthropic.com/engineering/building-effective-agents

• A workflow: a predefine paths for LLMs calls
• Orchestrator-worker: a intermediate approach
• Agents: the LLMs directs its own actions (normally with tools calls) and with the outputs decide what to do next

Other F/OSS Multi-Agent Projects

The Autogen Project

• https://github.com/microsoft/autogen
  • https://microsoft.github.io/autogen/

Camel

• https://github.com/camel-ai/camel

🐫 CAMEL: Finding the Scaling Law of Agents. A multi-agent framework. https://www.camel-ai.org

MetaGPT

MetaGPT Project Details:

• The MetaGPT Docs Site
• The MetaGPT Code at Github
  • License: MIT ❤️

MetaGPT - A F/OSS Multi Agent Framework that will power your next Projects

CrewAI

CrewAI Project

• The Project on PyPI
• The CrewAI Source Code at Github
  • License: MIT ❤️

CrewAI is a Framework that will make easy for us to get Local AI Agents interacting between them.

Agents with AutoGen - AG2AI

We have several agent concepts in AG2 to help you build your AI agents.

We introduce the most common ones here:

• Conversable Agent: Agents that are able to send messages, receive messages and generate replies using GenAI models, non-GenAI tools, or human inputs.

• Human in the loop: Add human input to the conversation

• Orchestrating multiple agents: Users can orchestrate multiple agents with built-in conversation patterns such as swarms, group chats, nested chats, sequential chats or customize the orchestration by registering custom reply methods.

• Tools: Programs that can be registered, invoked and executed by agents - Agents gain significant utility through tools as they provide access to external data, APIs, and functionality.

Advanced Concepts: AG2 supports more concepts such as structured outputs, rag, code execution, etc.

• https://pypi.org/project/pyautogen/
• https://github.com/ag2ai/ag2

pip install ag2
#or with the alias!

I got started with PyAutoGen thanks to this .ipynb:

• https://colab.research.google.com/drive/1xee3xdatViM4pespvLVVOrHJ8sB1MgO5?usp=drive_link

from autogen import AssistantAgent, UserProxyAgent, config_list_from_json

llm_config = {
    "config_list": config_list_from_json(env_or_file="OAI_CONFIG_LIST")
}

assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False})
user_proxy.initiate_chat(assistant, message="Plot a chart of NVDA and TESLA stock price change YTD.")
# This initiates an automated chat between the two agents to solve the task

Advanced agentic design patterns

AG2 supports more advanced concepts to help you build your AI agent workflows.

• Structured Output
• Ending a conversation
• Retrieval Augmented Generation (RAG)
• Code Execution
• Tools with Secrets

Agents with LangChain

You guessed it!

LangChain can also do Agents tricks.

• https://www.langchain.com/stateofaiagents

You might know Langchain because of its useful Chains and its RAGs Features

Agents with Llama-Index

LlamaIndex, is designed to work with various tools, data sources, and workflows.

• https://jalcocert.github.io/JAlcocerT/using-langchain-with-pandas-df/#llamaindex-and-pandas-dataframes
• https://jalcocert.github.io/JAlcocerT/how-to-use-rags-with-python/#llama-index

It helps to augment LLMs with context from different data sources such as APIs, SQL databases, PDFs, etc., without being tied to a specific framework.

• https://docs.llamaindex.ai/en/stable/

The ability to integrate multiple agents, tools, and data sources makes it adaptable to various development environments and workflows.

Context augmentation involves providing an LLM with specific data (private or external) to help solve a problem, enhancing the LLM’s ability to generate relevant answers by accessing relevant data.

Agents are knowledge assistants powered by LLMs that perform tasks using various tools, including Retrieval-Augmented Generation (RAG) as one tool in a larger workflow.

Workflows combine multiple agents, tools, and data connectors to create complex, event-driven processes that can automate tasks, reflecting advanced LLM applications.

Agents with PydanticAI

I was using Pydantic this year.

We can say it is a data validation framework, that now it has an AI version

• https://github.com/pydantic/pydantic-ai
  • https://ai.pydantic.dev/
  • https://pypi.org/project/pydantic-ai/

Agent Framework / shim to use Pydantic with LLMs - MIT Licensed!

Chat-DEV

• The ChatDev Repository
  • License: Apache v2 ❤️

Create Customized Software using Natural Language Idea (through LLM-powered Multi-Agent Collaboration)

Langroid

• https://github.com/langroid/langroid
  • https://langroid.github.io/langroid/

Example - https://github.com/langroid/langroid/blob/main/examples/basic/chat-search-assistant.py

Harness LLMs with Multi-Agent Programming - MIT LICENSED!

Auto-GPT

• https://github.com/Significant-Gravitas/AutoGPT
  • https://agpt.co/

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

CrewAI

You can use Streamlit with CrewAI

Example - https://github.com/tonykipkemboi/trip_planner_agent

CrewAI agents that can plan your vacation.

CrewAI Pandas DF Agent

• https://medium.com/pythoneers/building-a-multi-agent-system-using-crewai-a7305450253e

MetaGPT

• https://github.com/geekan/MetaGPT

MIT | 🌟 The Multi-Agent Framework: First AI Software Company, Towards Natural Language Programming

MetaGPT

SuperAGI

• https://github.com/TransformerOptimus/SuperAGI
  • https://github.com/TransformerOptimus/SuperAGI?tab=MIT-1-ov-file#readme

<⚡️> SuperAGI - A dev-first open source autonomous AI agent framework. Enabling developers to build, manage & run useful autonomous agents quickly and reliably.

Conclusions

Isnt it crazy that we already have: https://jobforagent.com/

• https://platform.openai.com/logs

  • https://platform.openai.com/playground/prompts?models=gpt-4.1

• Ive also started to use: https://windsurf.com/pricing

  • https://windsurf.com/editor/download-linux

curl -fsSL "https://windsurf-stable.codeiumdata.com/wVxQEIWkwPUEAGf3/windsurf.gpg" | sudo gpg --dearmor -o /usr/share/keyrings/windsurf-stable-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/windsurf-stable-archive-keyring.gpg arch=amd64] https://windsurf-stable.codeiumdata.com/wVxQEIWkwPUEAGf3/apt stable main" | sudo tee /etc/apt/sources.list.d/windsurf.list > /dev/null

#sudo apt-get update
sudo apt-get upgrade windsurf

Example with DASH

Breakdown of the libraries in your pyproject.toml that are commonly used for Artificial Intelligence and Machine Learning:

• langchain>=0.3.19: LangChain is a popular framework for building applications powered by large language models (LLMs). It provides tools for prompt management, model interaction, memory, routing, and more.
• langchain-community>=0.3.18: This package contains community-contributed integrations and utilities for LangChain, often including connections to various data sources, tools, and other services relevant to AI applications.
• langchain-openai>=0.3.7: This is a specific integration package for LangChain that provides seamless access to OpenAI’s models (like GPT-3.5, GPT-4, etc.) and related services.
• langgraph>=0.2.74: LangGraph is a library built on top of LangChain that helps you build conversational agents as graphs of steps, making it easier to design more complex and structured AI interactions.
• openai>=1.64.0: This is the official Python library for interacting directly with the OpenAI API. It allows you to access their models for tasks like text generation, translation, code generation, and more.
• openai-agents>=0.0.7: This library focuses on building AI agents that can use tools and perform actions in the real world, often in conjunction with large language models.

In summary, the following libraries in your list are the primary AI-related ones:

• langchain
• langchain-community
• langchain-openai
• langgraph
• openai
• openai-agents

The other libraries in your dependencies list serve different purposes:

• langfuse: An observability platform for language model applications, helping with tracking and debugging.

Make and Docker Compose

make --version #4.3

This is a new version to manage dockerfiles:

docker compose version #v2.35.1

FAQ

• https://e2b.dev/ai-agents/open-source

Interesting LLMs Architectures

• MoE - Mix of Experts
• MoA - Mix of Agents
• ReACT - Reasoning + Acting
  • First they think, then they execute an action inside an environment

How to Provide Web Context to the Agents

Via Scrapping

https://jalcocert.github.io/JAlcocerT/comparing-rag-and-use-cases/#web-scrapping https://github.com/JAlcocerT/Data-Chat/tree/main/LangChain/web

Via Web Search

For example, with duckduckgo:

pip install duckduckgo-search

from duckduckgo_search import ddg_answers

# Perform search query
results = ddg_answers('Python programming')

# Output the search results
print(results)

I improved with this one…

They both understand the [project] and [project.dependencies] sections to know which top-level packages your project requires.

Which one should you choose?

• venv: Best for very simple projects or when you need maximum flexibility and direct control and are comfortable managing dependencies manually with pip and requirements.txt.
• uv: A great option if speed is a primary concern and you want a faster alternative to pip and venv for environment management and dependency installation. It’s particularly appealing for its performance and adherence to modern standards.
• poetry: Ideal for larger projects, libraries, and applications where reproducible builds, robust dependency management, and streamlined packaging are crucial. It offers a more opinionated but comprehensive workflow.

Ultimately, the best choice depends on the specific needs and complexity of your Python projects and your personal preferences.

You might even find yourself using a combination of these tools in different situations. For instance, you might use uv for its speed in setting up environments for projects managed by poetry.