π An awesome & curated list of best LLMOps tools. But more about LLMOps.
Name | Stars | Release | Contributors | About |
---|---|---|---|---|
BentoML | Build Production-Grade AI Applications | |||
Dify | One API for plugins and datasets, one interface for prompt engineering and visual operation, all for creating powerful AI applications | |||
FastChat | An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena. | |||
Flowise | Drag & drop UI to build your customized LLM flow | |||
Haystack | π LLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots. | |||
LangChain | β‘ Building applications with LLMs through composability β‘ | |||
LiteLLM | lightweight package to simplify LLM API calls - Azure, OpenAI, Cohere, Anthropic, Replicate. Manages input/output translation | |||
LLaMa-Factory | Easy-to-use LLM fine-tuning framework (LLaMA, BLOOM, Mistral, Baichuan, Qwen, ChatGLM) | |||
LlamaIndex | LlamaIndex (GPT Index) is a data framework for your LLM applications | |||
Ludwig | Low-code framework for building custom LLMs, neural networks, and other AI models | |||
SkyPilot | SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed executionβall with a simple interface. |
Name | Stars | Release | Contributors | About |
---|---|---|---|---|
Flyte | Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. | |||
Kubeflow | Machine Learning Toolkit for Kubernetes | |||
Metaflow | π Build and manage real-life data science projects with ease! | |||
MLflow | Open source platform for the machine learning lifecycle | |||
ZenML | ZenML π: Build portable, production-ready MLOps pipelines. https://zenml.io. |
Name | Stars | Release | Contributors | About |
---|---|---|---|---|
ColossalAI | Making large AI models cheaper, faster and more accessible |
Name | Stars | Release | Contributors | About |
---|---|---|---|---|
AutoGPT | An experimental open-source attempt to make GPT-4 fully autonomous. | |||
MetaGPT | π The Multi-Agent Framework: Given one line Requirement, return PRD, Design, Tasks, Repo | |||
XAgent | An Autonomous LLM Agent for Complex Task Solving |
Name | Stars | Release | Contributors | About |
---|---|---|---|---|
DeepSpeed-MII | MII makes low-latency and high-throughput inference possible, powered by DeepSpeed. | |||
LMDeploy | LMDeploy is a toolkit for compressing, deploying, and serving LLMs. | |||
OpenLLM | Operating LLMs in production | |||
RayServe | Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads. | |||
SGLang | SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable. | |||
Triton Inference Server | The Triton Inference Server provides an optimized cloud and edge inferencing solution. | |||
Text Generation Inference | Large Language Model Text Generation Inference | |||
vLLM | A high-throughput and memory-efficient inference and serving engine for LLMs |
Name | Stars | Release | Contributors | About |
---|---|---|---|---|
AgentBench | A Comprehensive Benchmark to Evaluate LLMs as Agents | |||
lm-evaluation-harness | A framework for few-shot evaluation of language models. |
Name | Stars | Release | Contributors | About |
---|---|---|---|---|
deeplake | Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai |
Name | Stars | Release | Contributors | About |
---|---|---|---|---|
OpenLLMetry | Open-source observability for your LLM application, based on OpenTelemetry | |||
phoenix | ML Observability in a Notebook - Uncover Insights, Surface Problems, Monitor, and Fine Tune your Generative LLM, CV and Tabular Models |
Name | Stars | Release | Contributors | About |
---|---|---|---|---|
Instructor | structured outputs for llms | |||
Outlines | Structured Text Generation |