LLM Router

Intelligently route user queries to the most relevant content, action, or response using AI-powered decision making.

What is LLM Router?

LLM Router uses AI to intelligently match user queries to the most relevant option from a set of choices. Unlike traditional keyword matching or rules-based routing, it understands semantic meaning and context. Perfect for:

Intent Classification: Route users to the right service or department
Smart Q&A: Match questions to the most relevant FAQ or knowledge base article
Content Recommendation: Find the best matching content for user queries
Workflow Automation: Trigger appropriate actions based on user input
Chatbot Decision Making: Select the right response or tool to use

How It Works

The LLM Router uses an LLM to analyze the query and all available choices, then selects the best match with a confidence score and reasoning:

1. Add Choices: Provide a list of possible options with optional metadata
2. Set Confidence Threshold: Define minimum confidence level (0.0-1.0)
3. Route Query: Submit user query for intelligent matching
4. Get Result: Receive best match with confidence score and reasoning

Basic Usage

from SimplerLLM.language.llm import LLM, LLMProvider
from SimplerLLM.language.llm_router import LLMRouter

# Create LLM instance
llm = LLM.create(
    provider=LLMProvider.OPENAI,
    model_name="gpt-4o-mini"  # Cost-efficient model for routing
)

# Create router with confidence threshold
router = LLMRouter(
    llm_instance=llm,
    confidence_threshold=0.7  # Only accept matches with 70%+ confidence
)

# Add choices
choices = [
    ("I need help with billing", {"category": "support", "department": "billing"}),
    ("How do I reset my password?", {"category": "support", "department": "technical"}),
    ("I want to upgrade my plan", {"category": "sales", "department": "sales"}),
    ("Report a bug in the application", {"category": "support", "department": "technical"})
]

router.add_choices(choices)

# Route a user query
user_query = "My credit card was charged twice"
result = router.route(user_query)

if result:
    choice_content, choice_metadata = router.get_choice(result.selected_index)
    print(f"Matched: {choice_content}")
    print(f"Department: {choice_metadata['department']}")
    print(f"Confidence: {result.confidence_score:.2%}")
    print(f"Reasoning: {result.reasoning}")
else:
    print("No confident match found")

Advanced Features

Top-K Routing

Get multiple best matches instead of just one:

from SimplerLLM.language.llm_router import LLMRouter

router = LLMRouter(llm_instance=llm, confidence_threshold=0.5)

# Add FAQ choices
faqs = [
    ("How to install SimplerLLM?", {"type": "installation"}),
    ("Supported LLM providers list", {"type": "features"}),
    ("Pricing and API costs", {"type": "billing"}),
    ("How to use async features?", {"type": "advanced"}),
    ("Getting started tutorial", {"type": "tutorial"})
]
router.add_choices(faqs)

# Get top 3 matches
query = "How do I get started with this library?"
results = router.route_top_k(query, k=3)

for i, result in enumerate(results, 1):
    choice_content, metadata = router.get_choice(result.selected_index)
    print(f"{i}. {choice_content}")
    print(f"   Confidence: {result.confidence_score:.2%}")
    print(f"   Type: {metadata['type']}\n")

Dynamic Confidence Threshold

Adjust confidence requirements based on context:

# Strict matching for critical operations
strict_router = LLMRouter(llm_instance=llm, confidence_threshold=0.9)

# Lenient matching for general queries
lenient_router = LLMRouter(llm_instance=llm, confidence_threshold=0.5)

# Or adjust dynamically
def route_with_fallback(query, choices):
    router = LLMRouter(llm_instance=llm, confidence_threshold=0.8)
    router.add_choices(choices)

    result = router.route(query)

    if not result:
        # Lower threshold and try again
        router.confidence_threshold = 0.5
        result = router.route(query)

    return result

Batch Processing

Route multiple queries efficiently:

from SimplerLLM.language.llm_router import LLMRouter

router = LLMRouter(llm_instance=llm, confidence_threshold=0.7)

# Add choices once
choices = [
    ("Product returns policy", {"category": "policy"}),
    ("Shipping information", {"category": "logistics"}),
    ("Payment methods accepted", {"category": "payment"})
]
router.add_choices(choices)

# Route multiple queries
queries = [
    "How long does delivery take?",
    "Can I return my order?",
    "Do you accept PayPal?"
]

results = []
for query in queries:
    result = router.route(query)
    if result:
        choice, metadata = router.get_choice(result.selected_index)
        results.append({
            'query': query,
            'match': choice,
            'confidence': result.confidence_score,
            'category': metadata['category']
        })

for r in results:
    print(f"Q: {r['query']}")
    print(f"A: {r['match']} ({r['confidence']:.0%} confidence)\n")

Real-World Examples

Customer Support Routing

from SimplerLLM.language.llm import LLM, LLMProvider
from SimplerLLM.language.llm_router import LLMRouter

class SupportRouter:
    def __init__(self):
        llm = LLM.create(provider=LLMProvider.OPENAI, model_name="gpt-4o-mini")
        self.router = LLMRouter(llm_instance=llm, confidence_threshold=0.6)

        # Define support categories
        self.departments = [
            ("Billing issues, payments, invoices, refunds",
             {"dept": "billing", "priority": "high"}),
            ("Technical problems, bugs, errors, not working",
             {"dept": "technical", "priority": "high"}),
            ("Account settings, password, login issues",
             {"dept": "account", "priority": "medium"}),
            ("General questions, how to use features",
             {"dept": "general", "priority": "low"}),
            ("Sales inquiries, upgrades, new purchases",
             {"dept": "sales", "priority": "medium"})
        ]
        self.router.add_choices(self.departments)

    def route_ticket(self, customer_message):
        result = self.router.route(customer_message)

        if result:
            _, metadata = self.router.get_choice(result.selected_index)
            return {
                'department': metadata['dept'],
                'priority': metadata['priority'],
                'confidence': result.confidence_score,
                'reasoning': result.reasoning
            }
        else:
            # Fallback to general support
            return {
                'department': 'general',
                'priority': 'low',
                'confidence': 0.0,
                'reasoning': 'No confident match - routing to general support'
            }

# Usage
support = SupportRouter()

tickets = [
    "I was charged twice for the same transaction",
    "The app keeps crashing when I try to export data",
    "How do I change my email address?"
]

for ticket in tickets:
    routing = support.route_ticket(ticket)
    print(f"Ticket: {ticket}")
    print(f"Route to: {routing['department']} (Priority: {routing['priority']})")
    print(f"Confidence: {routing['confidence']:.0%}\n")

FAQ Matching System

class FAQMatcher:
    def __init__(self, faq_data):
        llm = LLM.create(provider=LLMProvider.OPENAI, model_name="gpt-4o-mini")
        self.router = LLMRouter(llm_instance=llm, confidence_threshold=0.6)

        # Add FAQ questions with answers as metadata
        choices = [
            (faq['question'], {'answer': faq['answer'], 'id': faq['id']})
            for faq in faq_data
        ]
        self.router.add_choices(choices)

    def find_answer(self, user_question, top_k=1):
        if top_k > 1:
            results = self.router.route_top_k(user_question, k=top_k)
        else:
            single_result = self.router.route(user_question)
            results = [single_result] if single_result else []

        answers = []
        for result in results:
            _, metadata = self.router.get_choice(result.selected_index)
            answers.append({
                'answer': metadata['answer'],
                'confidence': result.confidence_score,
                'faq_id': metadata['id']
            })

        return answers

# Usage
faq_data = [
    {
        'id': 1,
        'question': 'How do I install SimplerLLM?',
        'answer': 'Run: pip install simplerllm'
    },
    {
        'id': 2,
        'question': 'Which LLM providers are supported?',
        'answer': 'OpenAI, Anthropic, Google Gemini, Cohere, DeepSeek, OpenRouter, Ollama, and XAI'
    },
    {
        'id': 3,
        'question': 'How do I set up API keys?',
        'answer': 'Add them to a .env file or pass directly when creating LLM instances'
    }
]

faq = FAQMatcher(faq_data)

# Find best matching FAQ
user_q = "What's the command to get this library?"
answers = faq.find_answer(user_q, top_k=2)

for i, ans in enumerate(answers, 1):
    print(f"{i}. {ans['answer']}")
    print(f"   Confidence: {ans['confidence']:.0%}\n")

Content Recommendation

class ContentRecommender:
    def __init__(self, content_library):
        llm = LLM.create(provider=LLMProvider.OPENAI, model_name="gpt-4o-mini")
        self.router = LLMRouter(llm_instance=llm, confidence_threshold=0.5)

        # Add content with descriptions
        choices = [
            (content['description'], {
                'title': content['title'],
                'url': content['url'],
                'type': content['type']
            })
            for content in content_library
        ]
        self.router.add_choices(choices)

    def recommend(self, user_interest, count=3):
        results = self.router.route_top_k(user_interest, k=count)

        recommendations = []
        for result in results:
            _, metadata = self.router.get_choice(result.selected_index)
            recommendations.append({
                'title': metadata['title'],
                'url': metadata['url'],
                'type': metadata['type'],
                'relevance': result.confidence_score
            })

        return recommendations

# Usage
content = [
    {
        'title': 'Getting Started with LLMs',
        'description': 'Beginner tutorial on using language models',
        'url': '/docs/quickstart',
        'type': 'tutorial'
    },
    {
        'title': 'Advanced Async Patterns',
        'description': 'Deep dive into asynchronous LLM operations',
        'url': '/docs/async',
        'type': 'advanced'
    }
]

recommender = ContentRecommender(content)
recs = recommender.recommend("I'm new to AI development", count=2)

for rec in recs:
    print(f"{rec['title']} ({rec['type']})")
    print(f"Relevance: {rec['relevance']:.0%}")
    print(f"URL: {rec['url']}\n")

Performance Optimization

1. Choose Efficient Models

Use cost-efficient models like gpt-4o-mini for routing - they're fast and accurate enough for classification tasks.

2. Cache Router Instances

Reuse router instances with the same choices instead of recreating them for each query.

3. Batch Similar Queries

Process multiple queries that share the same choice set together to minimize overhead.

4. Optimize Choice Descriptions

Write clear, concise choice descriptions - they help the LLM make better decisions faster.

Best Practices

Routing Guidelines

Clear Choices: Make choice descriptions distinct and descriptive
Appropriate Threshold: Set confidence threshold based on risk (higher for critical decisions)
Fallback Strategy: Always handle cases where no confident match is found
Monitor Performance: Track confidence scores and routing accuracy over time
Test Edge Cases: Verify routing works for ambiguous or unusual queries
Use Metadata: Store relevant context in choice metadata for downstream processing

Error Handling

from SimplerLLM.language.llm_router import LLMRouter
from SimplerLLM.language.llm import LLM, LLMProvider

try:
    # Create router
    llm = LLM.create(provider=LLMProvider.OPENAI, model_name="gpt-4o-mini")
    router = LLMRouter(llm_instance=llm, confidence_threshold=0.7)

    # Add choices
    choices = [("Option 1", {}), ("Option 2", {})]
    router.add_choices(choices)

    # Route query
    result = router.route("User query here")

    if result:
        choice, metadata = router.get_choice(result.selected_index)
        print(f"Matched: {choice}")
    else:
        print("No match found above threshold")

except ValueError as e:
    print(f"Configuration error: {e}")

except ConnectionError as e:
    print(f"LLM connection failed: {e}")

except Exception as e:
    print(f"Routing error: {e}")

Comparison with Traditional Routing

Why Use LLM Router?

Traditional Keyword Matching:

❌ Requires exact keyword matches
❌ Can't handle synonyms or paraphrasing
❌ Needs constant maintenance of keyword lists

LLM Router:

✅ Understands semantic meaning
✅ Handles natural language variations
✅ Provides confidence scores and reasoning
✅ Works out of the box with minimal setup

Next Steps

← Vector Databases

Learn about vector storage and search

Prompt Templates →

Build reusable prompt templates

🎯 Try It Live

Experience LLM Router in action with our interactive demo:

Try LLM Router Demo →

Edit this page on GitHub →