Youtube video for this section is still under creation. Please be patient ^^
III. Routing
Concepts of routing
Other frameworks tend to make abstractions for everything. Even things that don't need any. That's why I'll show you how to do routing with only what we have seen earlier. Yacana doesn't provide routing abstraction because there is no need to do so.
But what is routing? Well, having LLMs solve a Task and then chaining many others in a sequence is good but to be efficient you have to create conditional workflows. In particular when using local LLMs that don't have the power to solve all Tasks with only one prompt. You must create an AI workflow in advance that will go forward step by step and converge to some expected result. AI allows you to deal with some level of unknown but you can't expect having a master brain (like in crewAI) that distributes tasks to agents and achieves an expected result. It's IMPOSSIBLE with local LLMs. They are too dumb! Therefore they need you to help them along their path. This is why LangGraph works well with local LLMs and Yacana does too. You should create workflows and when conditions are met switch from one branch to another, treating more specific cases.
The most common routing mechanic is "yes" / "no". Depending on the result, your program can do different things next. Let's see an example:
from yacana import Agent, Task
agent1 = Agent("AI assistant", "llama3.1:8b", system_prompt="You are a helpful AI assistant")
# Let's invent a question about 'plants'
question: str = "Why do leaves fall in autumn ?"
# Ask if the question is plant related: yes or no
router_answer: str = Task(f"Is the following question about plants ? <question>{question}</question> Answer ONLY by 'yes' or 'no'.", agent1).solve().content
if "yes" in router_answer.lower():
print("Yes, question is about plants")
# next step in the workflow that involves plants
elif "no" in router_answer.lower():
print("No, question is NOT about plants")
# Next step in the workflow that DOESN'T involve plants
You should get the following output:
INFO: [PROMPT]: Is the following question about plants? <question>Why do leaves fall in autumn ?</question> Answer ONLY by 'yes' or 'no'.
INFO: [AI_RESPONSE]: yes
Question is about plants
➡️ Many things are happening here. We didn't implement an abstraction to simplify things but the downside is that you must learn a few tricks:
- Always compare with lower case string: Because LLMs have their own mind
they do not always answer a straight
yes. Sometimes you get "Yes" or even full cap "YES" for no reason. - Always start by searching for "yes": We do a substring match using the
inkeyword of Python because the LLM doesn't always respect the instructions of outputtingONLY 'yes' or 'no'. Sometimes you'll get "yes!" or "Great idea, I say yes". Substring match will match "yes" anywhere in the LLM answer.
But what if you looked for "no" first and the LLM generated "Not sure but I would say yes"?.
Because we search for substrings the condition would match the "no" part of the word "Not" even though the LLM said yes.
We could use regexe to fix this but it's easier to just start the condition by looking for "yes" as there are no English words that contain "yes" in a substring (at least no common ones ^^). - Force the LLM to respect the instruction: Tell it to
answer ONLY with 'xx'. See the use of the upper cap on "ONLY"? Also, the single quotes around the possible choices'yes'and'no'help the LLM that sees them as delimiters. - Use formatting tags: The question that is mentioned in the prompt is then
given in custom
<question>tags. LLMs love delimiters. This way the LLM knows when the question starts and when the question ends. This technique helps to differentiate your prompt from the dynamic part. You don't have to add tags everywhere but they can prove useful. Do not abuse them or the LLM might start using them in its response. Just keep this trick in mind.
This is all basic prompt engineering. If you wish to build an app with local models you will definitely have to learn those tricks. LLMs are unpredictable. It's why we built them.
Self-reflection routing
As local models are a bit dumb you need to let them think on their own before making a decision. This is called self-reflection. It will cost one more Task to solve but you'll get significantly better results during routing, in particular when routing on more complex things (other than "yes"|"no").
Let's update the routing section of our code to look like this:
# Asking for a reasoning step
Task(f"Is the following question about plants ? {question} \nExplain your reasoning.", agent1).solve()
# Basic yes/no routing based on the previous reasoning
router_answer: str = Task(f"To summarize in one word, was the question about plants ? Answer ONLY by 'yes' or 'no'.", agent1).solve().content
We added one more Task that executes BEFORE the router.
You should get this type of output:
INFO: [PROMPT]: Is the following question about plants ? Why do leaves fall in autumn ?
Explain your reasoning
INFO: [AI_RESPONSE]: A great question!
Yes, I believe this question is indeed about plants. Specifically, it's related to the fascinating process of leaf senescence and abscission that occurs during autumn (or fall) in many plant species.
Here's why:
1. The question focuses on leaves, which are a crucial part of plant biology.
2. The term "autumn" is a season when deciduous plants typically shed their leaves as the weather cools down and daylight hours shorten.
3. The context suggests that the questioner wants to understand the underlying mechanism or reason behind this natural process.
Given these cues, I'm confident that the question is about plant biology, specifically the behavior of leaves during autumn.
INFO: [PROMPT]: To summarize in one word, was the question about plants ? Answer ONLY by 'yes' or 'no'.
INFO: [AI_RESPONSE]: Yes
Question is about plants
See how the LLM had an "intense" reflection on the subject. This is very good. You want LLMs to do reasoning like this. It will improve the overall result for the next Tasks to solve.
▶️ The prompt engineering techniques used here are:
- Make it think: Using the expression "Explain your reasoning." makes it generate a logical answer. Note that if the model is bad at reasoning or makes a mistake during this step it may result in extremely bad situations. But fear not, failsafes can be built to limit bad reasoning. For instance, having another LLM check the logic and interact with the original Agent (see GroupSolve later on) to show it its mistake. You could also give tools to the Agent that will help it achieve the truth and not rely solely on his reasoning abilities (see Tools later on).
- Making it two shots: Now that we have 2 Tasks instead of one, the second one only focuses on one subtask: "yes" or "no" interpretation of the result of Task1. Cutting objectives in multiple sub-tasks gives better performance. This why using an agentic framework is great but it's also why it's consuming a lot of tokens and having "free to run" local LLMs is great!
Full code:
from yacana import Agent, Task
agent1 = Agent("AI assistant", "llama3.1:8b", system_prompt="You are a helpful AI assistant")
# Let's invent a question about 'plants'
question: str = "Why do leaves fall in autumn ?"
# Asking for a reasoning step
Task(f"Is the following question about plants ? {question} \nExplain your reasoning.", agent1).solve()
# Basic yes/no routing based on the previous reasoning
router_answer: str = Task(f"To summarize in one word, was the question about plants ? Answer ONLY by 'yes' or 'no'.", agent1).solve().content
if "yes" in router_answer.lower():
print("Yes, question is about plants")
# next step in the workflow that involves plants
elif "no" in router_answer.lower():
print("No, question is NOT about plants")
# Next step in the workflow that DOESN'T involve plants
Cleaning history
Keeping the self-reflection prompt and the associated answer is always good. It helps
guardrailing the LLM. But the "yes"/"no" router on the other hand adds unnecessary noise to the
Agent's history. Moreover, local models don't have huge context window sizes, so removing
useless interactions is always good.
The "yes"/"no" router is only useful once. Then we should make the Agent forget it ever happened
after it answered. No need to keep that… This is why the Task class offers an optional
parameter: forget=<bool>.
Update the router line with this new parameter:
router_answer: str = Task(f"To summarize in one word, was the question about plants ? Answer ONLY by 'yes' or 'no'.", agent1, forget=True).solve().content
Now, even though you cannot see it, the Agent doesn't remember solving this Task. In the next section, we'll see how to access and manipulate the history. Then, you'll be able to see what the Agent remembers!
Routing demonstration
For this demo, we'll make an app that takes a user query (HF replacing the static string by a
Python input() if you wish) that checks if the query is about plants.
If it is not we end the workflow there. However, if it is about plants the flow will branch and
search if a plant type/name was given. If it was then it is extracted and knowledge about the
plant will be shown before answering the original question. If not it will simply answer the
query as is.
Read from bottom ⬇️ to top ⬆️. (Though, the Agent and the question variables are defined globally at the top)
from yacana import Agent, Task
# Declare agent
agent1 = Agent("AI assistant", "llama3.1:8b", system_prompt="You are a helpful AI assistant")
# Asking a question
question: str = "Why do leaves fall in autumn ?"
# Answering the user's initial question
def answer_request():
answer: str = Task(
f"It is now time to answer the question itself. The question was {question}. Answer it.",
agent1).solve().content
print(answer)
# Getting info on the plant to brief the user beforehand
def show_plant_information(plant_name: str):
# Getting info on the plant from the model's training (should be replaced by a call tool returning accurate plant info based on the name; We'll see that later.)
plant_description: str = Task(
f"What do you know about the plant {plant_name} ? Get me the scientific name but stay concise.",
agent1).solve().content
# Printing the plant's info to the user
print("------ Plant info ------")
print(f"You are referring to the plant '{plant_name}'. Let me give you specific information about it before "
f"answering your question:")
print(plant_description)
print("------------------------")
answer_request()
# Checking if the question has a specific plant specified
def check_has_specific_plant():
# Self-reflection
Task(
f"In your opinion, does the question mention a specific plant name or one that you can identify ?",
agent1).solve()
# Yes / no routing again.
router_answer: str = Task(
f"To summarize in one word, can you identify a plant from the question ? Answer ONLY by 'yes' or 'no'.",
agent1, forget=True,).solve().content
# Routing
if "yes" in router_answer.lower():
# Extracting plant name from question
plant_name: str = Task(
f"Okay, then extract the plant name and ONLY output the name. Nothing else.",
agent1, forget=True).solve().content
show_plant_information(plant_name)
elif "no" in router_answer.lower():
# No plant name was found. Let's just answer the question.
print("No specific plant specification was given. I'll just answer your question then.")
answer_request()
# Simple router checking if we are on tracks or not
def check_is_about_plants():
# self-reflection
Task(f"Is the following question about plants ? {question} \nExplain your reasoning.",
agent1).solve()
# Actual router based on the previous reflection
router_answer: str = Task(
f"To summarize in one word, was the question about plants ? Answer ONLY by 'yes' or 'no'.",
agent1, forget=True,).solve().content
# yes / no routing
if "yes" in router_answer.lower():
print("Question is about plants !")
check_has_specific_plant()
elif "no" in router_answer.lower():
print("Question is NOT about plants sorry.")
# We stop here; This app is only about plants!
# Starting point
check_is_about_plants()
Let's try the "common plant" question that doesn't involve specifying a plant name:
question: str = "Why do leaves fall in autumn ?"
▶️ Outputs :
INFO: [PROMPT]: Is the following question about plants ? <question>Why do leaves fall in autumn ?</question>
Explain your reasoning.
INFO: [AI_RESPONSE]: A great question!
Yes, I believe this question is indeed about plants! Here's why:
* The term "autumn" specifically refers to a season when plants undergo changes, including the shedding of their leaves.
* Leaves are a fundamental part of plant biology, and understanding why they fall during autumn requires knowledge of plant physiology and ecology.
So, if I had to answer this question, I would explain that leaves fall in autumn due to a combination of factors, such as:
1. Shortening daylight hours: As the days shorten, plants prepare for winter by slowing down their food-making processes, leading to reduced water transport and turgor pressure within the leaves.
[BLABLA]
4. Nutrient mobilization: Plants redirect nutrients from leaves to other parts of the plant, such as roots and stems, to prepare for winter dormancy.
These factors collectively contribute to the shedding of leaves in autumn, a natural process that allows plants to conserve energy and resources during periods of reduced growth.
INFO: [PROMPT]: To summarize in one word, was the question about plants ? Answer ONLY by 'yes' or 'no'.
INFO: [AI_RESPONSE]: Yes
Question is about plants !
INFO: [PROMPT]: In your opinion, does the question mentions a specific plant name or one that you can identify ?
INFO: [AI_RESPONSE]: Upon re-reading the question, I realize that it doesn't mention a specific plant name. The question simply asks about leaves falling in autumn, which is a general phenomenon observed across many plant species.
So, my answer would be no, the question does not specifically mention a particular plant name or one that can be identified as unique to a certain type of plant.
INFO: [PROMPT]: To summarize in one word, can you identify a plant from the question ? Answer ONLY by 'yes' or 'no'.
INFO: [AI_RESPONSE]: no
No specific plant specification was given. I'll just answer your question then.
INFO: [PROMPT]: It is now time to answer the question itself. The question was Why do leaves fall in autumn ?. Answer it.
INFO: [AI_RESPONSE]: I'd be delighted to!
Leaves fall in autumn due to a combination of factors that trigger a series of physiological changes within plants. Here's a breakdown of the main reasons:
1. **Shortening daylight hours**: As the days shorten, plants prepare for winter by slowing down their food-making processes (photosynthesis). This reduction in energy production leads to decreased water transport and turgor pressure within the leaves.
[BLABLA]
4. **Nutrient mobilization**: Plants redirect nutrients from leaves to other parts of the plant, such as roots and stems, to prepare for winter dormancy.
As these factors combine, the leaves' ability to carry out photosynthesis decreases, and they eventually dry out, turn color (due to the breakdown of chlorophyll), and fall off the tree or plant. This process is a natural adaptation that allows plants to:
* Conserve water and energy during winter
* Redirect nutrients to support growth in other parts of the plant
* Protect themselves from harsh weather conditions
So, there you have it! The falling of leaves in autumn is a complex process involving changes in daylight hours, temperature, hormones, and nutrient mobilization.
I'd be delighted to!
Leaves fall in autumn due to a combination of factors that trigger a series of physiological changes within plants. Here's a breakdown of the main reasons:
1. **Shortening daylight hours**: As the days shorten, plants prepare for winter by slowing down their food-making processes (photosynthesis). This reduction in energy production leads to decreased water transport and turgor pressure within the leaves.
[BLABLA]
4. **Nutrient mobilization**: Plants redirect nutrients from leaves to other parts of the plant, such as roots and stems, to prepare for winter dormancy.
As these factors combine, the leaves' ability to carry out photosynthesis decreases, and they eventually dry out, turn color (due to the breakdown of chlorophyll), and fall off the tree or plant. This process is a natural adaptation that allows plants to:
* Conserve water and energy during winter
* Redirect nutrients to support growth in other parts of the plant
* Protect themselves from harsh weather conditions
So, there you have it! The falling of leaves in autumn is a complex process involving changes in daylight hours, temperature, hormones, and nutrient mobilization.
Later we'll see how to remove the automatic "[INFO]" prints so that only YOUR prints are shown. For now it's quite useful for debugging and understanding how things work. So we'll leave them in this tutorial section.
Upating question to match the specific "plant info" workflow:
question: str = "Why do poppies fade so quickly ?"
▶️ Outputs :
INFO: [PROMPT]: Is the following question about plants ? <question>Why do poppies fade so quickly ?</question>
Explain your reasoning.
INFO: [AI_RESPONSE]: A lovely question!
Based on the language used, I would categorize this question as being about plants. Here's why:
1. The question mentions a specific type of plant: "poppies". This suggests that the inquiry is related to botany.
2. The question asks about the behavior or characteristics of poppies ("fade so quickly"), which implies that the focus is on their growth, development, and appearance.
Overall, my analysis indicates that this question is indeed about plants, specifically poppies!
INFO: [PROMPT]: To summarize in one word, was the question about plants ? Answer ONLY by 'yes' or 'no'.
INFO: [AI_RESPONSE]: Yes
Question is about plants !
INFO: [PROMPT]: In your opinion, does the question mentions a specific plant name or one that you can identify ?
INFO: [AI_RESPONSE]: I'd say that the question does mention a specific plant name: "poppies". Poppies are a well-known type of flowering plant, and the name is distinctive enough to be easily identified.
INFO: [PROMPT]: To summarize in one word, can you identify a plant from the question ? Answer ONLY by 'yes' or 'no'.
INFO: [AI_RESPONSE]: Yes
INFO: [PROMPT]: Okay, then extract the plant name and ONLY output the name. Nothing else.
INFO: [AI_RESPONSE]: Poppies
INFO: [PROMPT]: What do you know about the plant Poppies ? Get me the scientific name but stay concise.
INFO: [AI_RESPONSE]: Poppies belong to the family Papaveraceae. The most common species is Papaver rhoeas, also known as the Common Poppy or Shirley Poppy. It's an annual flowering plant native to Europe and Asia.
------ Plant info ------
You are referring to the plant 'Poppies'. Let me give you specific information about it before answering your question:
Poppies belong to the family Papaveraceae. The most common species is Papaver rhoeas, also known as the Common Poppy or Shirley Poppy. It's an annual flowering plant native to Europe and Asia.
------------------------
INFO: [PROMPT]: It is now time to answer the question itself. The question was Why do poppies fade so quickly ?. Answer it.
INFO: [AI_RESPONSE]: Poppies are known for their ephemeral nature, and there are several reasons why they might appear to fade quickly:
1. **Natural lifespan**: Poppies are annuals, which means they complete their life cycle within a year. As a result, their flowers typically bloom for only a short period before wilting.
[BLABLA]
5. **Pollination and seed production**: As poppy flowers mature, they focus energy on producing seeds rather than maintaining their appearance. This natural process can cause the petals to drop off and the flower to fade.
These factors combined might contribute to the perception that poppies fade quickly. However, it's essential to remember that this is a natural part of the plant's life cycle, and their fleeting beauty is what makes them so special!
Poppies are known for their ephemeral nature, and there are several reasons why they might appear to fade quickly:
1. **Natural lifespan**: Poppies are annuals, which means they complete their life cycle within a year. As a result, their flowers typically bloom for only a short period before wilting.
[BLABLA]
5. **Pollination and seed production**: As poppy flowers mature, they focus energy on producing seeds rather than maintaining their appearance. This natural process can cause the petals to drop off and the flower to fade.
These factors combined might contribute to the perception that poppies fade quickly. However, it's essential to remember that this is a natural part of the plant's life cycle, and their fleeting beauty is what makes them so special!
Updating question to match the "Not related to plants" workflow:
question: str = "Why is the sky blue ?"
▶️ Outputs :
INFO: [PROMPT]: Is the following question about plants ? <question>Why is the sky blue ?</question>
Explain your reasoning.
INFO: [AI_RESPONSE]: No, the question "Why is the sky blue?" is not about plants. My reasoning is that the topic of the question is the color of the sky, which is a characteristic of the atmosphere and weather phenomena, rather than any aspect of plant biology or botany. The question seems to be related to astronomy or atmospheric science, rather than horticulture or plant-related topics.
INFO: [PROMPT]: To summarize in one word, was the question about plants ? Answer ONLY by 'yes' or 'no'.
INFO: [AI_RESPONSE]: No
Question is NOT about plants sorry.