Understanding LLM Text Generation

Decoder-only Transformer models, like those in the GPT family, are designed to predict the next word based on an input prompt, making them highly effective for text generation. With sufficient training data, these models can also generate code, serving either as a coding assistant or a chatbot.

An example of a commercial AI pair programmer is GitHub Copilot. In contrast, Meta AI's Code Llama offers similar functionalities without any associated costs.

What is Code Llama?

Code Llama is a specialized family of large language models (LLMs) designed for coding tasks, developed by Meta AI and launched in August 2023. Not to be confused with its namesake animal, the Llama 2 model serves as the foundational model for Code Llama, which has been further trained on a vast dataset primarily consisting of code.

The Code Llama suite includes three versions, available in four different sizes, and is free for both research and commercial applications.

Code Llama Specialization Pipeline

The Code Llama models are specifically trained for code generation, employing an infill objective that enhances their ability to complete code snippets in Integrated Development Environments (IDEs). The Instruct versions are fine-tuned on instruction datasets, allowing them to respond to user queries similarly to ChatGPT.

The Python Variant

There’s also a version of Code Llama trained on an additional 100 billion tokens of Python code, specifically aimed at code generation tasks.

Implementing the LLM Chatbot

For our project, we will leverage the CodeLlama-7b-Instruct model, which is the smallest variant of the Instruct series. Despite being the smallest, it still boasts 7 billion parameters. When using 16-bit half-precision parameters, approximately 14 GB of GPU memory is required. However, by employing 4-bit quantization, we can cut this requirement down to about 3.5 GB.

Model Implementation

First, we will create a ChatModel class that loads the Code Llama model from Hugging Face and generates text based on a given prompt. We’ll utilize BitsAndBytesConfig for 4-bit quantization, AutoModelForCausalLM for model loading, and AutoTokenizer for generating token embeddings from the input.

import torch

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

class ChatModel:

def __init__(self, model="codellama/CodeLlama-7b-Instruct-hf"):

quantization_config = BitsAndBytesConfig(

load_in_4bit=True,

bnb_4bit_compute_dtype=torch.float16,

bnb_4bit_use_double_quant=True,

)

self.model = AutoModelForCausalLM.from_pretrained(

model,

quantization_config=quantization_config,

device_map="cuda",

cache_dir="./models",

)

self.tokenizer = AutoTokenizer.from_pretrained(

model, use_fast=True, padding_side="left"

)

We will also set up a fixed-length history list to store previous prompts and AI-generated responses, helping maintain context throughout the conversation.

System Prompt

To optimize the assistant's responses, we’ll define a default system prompt that guides its interactions.

self.DEFAULT_SYSTEM_PROMPT = """

You are a helpful, respectful, and knowledgeable assistant with expertise in coding and software design. Always provide constructive and safe responses. Avoid sharing harmful, unethical, or biased content. If a question lacks clarity or coherence, kindly explain why instead of providing incorrect information.

"""

Generating Responses

Next, we will implement the generate method to produce text based on user input. Each LLM operates with a specific prompt template, which we will adhere to for effective results.

def generate(

self, user_prompt, system_prompt, top_p=0.9, temperature=0.1, max_new_tokens=512

):

# Implementation here

Testing the ChatModel

Before we build the front-end application, let’s validate the ChatModel.

from ChatModel import *

model = ChatModel()

response = model.generate(

user_prompt="Write a hello world program in C++",

system_prompt=model.DEFAULT_SYSTEM_PROMPT

)

print(response)

This should output a simple C++ program that prints "Hello, World!" to the console.

Building the Front-End Application

We will utilize Streamlit to create a user-friendly chatbot interface. The Streamlit documentation provides a basic example that we can adapt for our needs.

import streamlit as st

from ChatModel import *

st.title("Code Llama Assistant")

@st.cache_resource

def load_model():

model = ChatModel()

return model

model = load_model() # load our ChatModel once and then cache it

Next, we’ll establish a sidebar for user input controls to adjust the model's parameters.

with st.sidebar:

temperature = st.slider("temperature", 0.0, 2.0, 0.1)

top_p = st.slider("top_p", 0.0, 1.0, 0.9)

max_new_tokens = st.number_input("max_new_tokens", 128, 4096, 256)

system_prompt = st.text_area(

"system prompt", value=model.DEFAULT_SYSTEM_PROMPT, height=500

)

Now, let’s create the message interface for the chatbot.

# Initialize chat history

if "messages" not in st.session_state:

st.session_state.messages = []

# Display chat messages from history on app rerun

for message in st.session_state.messages:

with st.chat_message(message["role"]):

st.markdown(message["content"])

# Accept user input

if prompt := st.chat_input("Ask me anything!"):

# Add user message to chat history

st.session_state.messages.append({"role": "user", "content": prompt})

# Display user message in chat message container

with st.chat_message("user"):

st.markdown(prompt)

# Display assistant response in chat message container

with st.chat_message("assistant"):

user_prompt = st.session_state.messages[-1]["content"]

answer = model.generate(

user_prompt,

top_p=top_p,

temperature=temperature,

max_new_tokens=max_new_tokens,

system_prompt=system_prompt,

)

response = st.write(answer)

st.session_state.messages.append({"role": "assistant", "content": answer})

You can run the Streamlit app via the command streamlit run app.py, which will launch it in your browser, allowing you to interact with the chatbot.

References

[1] B. Rozière et al.: Code Llama: Open Foundation Models for Code (2023), arXiv:2308.12950

Resources

Streamlit chat app example: Build a basic LLM chat app

Hugging Face Code Llama Gradio implementation: codellama-13b-chat