Speed up open source LLM-serving with llama-cpp-python

Name: Speed up open source LLM-serving with llama-cpp-python
Start: 2024-04-05T12:00:00+00:00
End: 2024-04-05T12:25:00+00:00
Location: SMK

Talk 25 min April 05, 2024 at 12:00

Isaac Chung

Large language models (LLMs) often require huge compute resources to serve. This is a common challenge for those who want to avoid sharing their data with cloud API providers, or to deploy their stack in air-gapped environments. We will take a look at how the open source llama-cpp-python library opens the door to lower hardware requirements and simplifies deployment significantly.