Understanding Fine-Tuning

Selecting a large language model (LLM) that returns suitable responses for your use cases is only one part of creating a high-quality GenAI app. You still need a way to tailor the LLM’s responses to incorporate recent data specific to your scenarios.

What is fine-tuning?

Creating an LLM from scratch is expensive. Even the BloombergGPT model, which was significantly smaller than OpenAI’s GPT-3, cost around $2.7 million to make.

That's why both companies don't build their own LLMs! It's much more cost-effective to use an LLM for its language generation capabilities and supplement it with context-specific data. This approach is almost always much faster and cheaper than building your own model and regularly re-training it.

One such approach to adding context is fine-tuning, which adjusts a pre-trained model to better fit your data. A full AI model training project requires a multi-phase training process that involves adjusting millions or billions of model parameters with each pass. By contrast, fine-tuning freezes some or most of these parameters, retraining only a subset of parameters for a use case-specific task.

Fine-tuning is a type of technique in machine learning and AI circles as transfer learning. Instead of retraining the model from scratch, it transfers knowledge gained from a prior task for use on a new task.

Approaches to fine-tuning

There are many different approaches to fine-tuning. What follows below is one useful way to think about these approaches.

Full fine-tuning. Also called instruction fine-tuning, this updates all of a model’s weights with new information. Full fine-tuning is the most expensive and time-consuming approach to fine-tuning, as it's essentially retraining the LLM. It's very computationally expensive, requiring large amounts of disk space and memory.

Parameter-efficient fine-tuning. Parameter-efficient fine-tuning, by contrast, updates only a subset of a model's billions of parameters. There are multiple approaches to parameter-efficient fine-tuning:

• Partial fine-tuning, as its name implies, updates only a portion of the parameters, leaving the rest frozen.
• Additive fine-tuning, instead of updating existing parameters as extra layers or presentations to the model on top of what currently exists.

One example of additive fine-tuning is prompt tuning. In the early days of AI models, this was implemented as prompt engineering, in which humans generated hand-tailored prompts guiding the LLM to correct answers using techniques such as one-shot learning. Over time, data scientists discovered they could replace these with automatically generated numeric prompts, which proved more efficient to scale.

This approach, which effectively uses LLMs to train LLMs, can generate hundreds of thousands of data points. This makes it more economical than obtaining new data.

Re-parameterization. This approach uses a technique known as low rank adaptation (LoRA), which updates a set of lower rank matrixes versus updating the entire parameter set.

Supervised fine-tuning. This approach —which can be implemented using either full fine-tuning or parameter-efficient fine-tuning — uses a high-quality data set to associate example inputs with desired outputs. This technique is best used for training the model on a highly specific task or for capturing stylistic nuances.

Use cases for fine-tuning

There are several solid use cases for fine-tuning:

• Task-specific training of a larger LLM. LLMs do a great job at human language generation and echoing back common knowledge. However, they're not trained to understand or recognize a user's intent for a specific use case. Due to the time required to train an LLM, their data is also usually several years out of date.To address this gap, many commercial LLMs, such as OpenAI, enable some sort of fine-tuning of parameters via API call. You can also freely fine-tune on most open-source LLMs.
• Mitigation of biases. LLMs trained on publicly available data —such as Internet forums — often reflect societal biases and prejudices. (Even proprietary training data can contain unintentional biases.) Fine-tuning can help compensate for these blind spots.
• Handle edge cases and failures. You can use fine-tuning to identify known errors in an LLM’s responses, such as the hallucination in the absence of data.
• Style customization. Fine-tuning is a great technique to tailor an LLM’s responses so that they fit your company's tone and messaging.
• Incorporation of proprietary data. Fine-tuning is one method you can use to import data you own into the more generalized framework of the LLM.

Benefits of fine-tuning

Fine-tuning brings multiple benefits to GenAI apps:

• Highly customizable. Done well, fine-tuning can give you fine-grained control over your LLM’s responses. It can also customize aspects, such as tone of voice, that are harder to customize with other LLM optimization techniques.
• Better accuracy. Providing context-relevant data expands an LLM’s ability to respond with relevant information to domain-specific queries.
• Higher performance. Fine-tuning data is turned into numeric embeddings that are incorporated directly into the LLM. This makes queries against the data faster than other methods of LLM optimization that require querying an external data source.

Challenges of fine-tuning

While there are benefits to fine-tuning, there are a number of challenges that lead developers to look at other methods. A few of these include:

• It’s still expensive. While cheaper than creating a new model from scratch, it still requires a lot of engineering effort and computing power.
• It’s time-consuming. Fine-tuning requires a lot of trial and error, combined with rigorous testing, to get right. Mistakes such as overfitting or misconfiguring parameters can even make your results worse. For example, it’s possible in some cases that fine-tuning can increase hallucinations.
• It’s hard to track down issues. LLMs are generally black boxes. They don’t tell you how they reached their decisions. That can leave you guessing as to how a model came to its conclusions, which makes troubleshooting issues difficult.
• It may not solve all your problems. You may still experience issues such as hallucinations even with fine-tuning applied.
• There are security concerns. Embedding your proprietary information directly in an LLM raises concerns about whether your data will be protected. Most major Large Language Model makers state explicitly in their terms of service that they will never use fine-tuning data uploaded by customers to train their own LLMs. However, that doesn’t mean information can’t accidentally leak or be coerced out of the LLM my a malicious actor.

Fine-tuning vs RAG

The effort involved in fine-tuning means most GenAI app developers look to other techniques to provide contextual data to LLMs.

The most popular is retrieval-augmented generation (RAG), which queries relevant information from a supplementary external data source and includes it as context in the LLM prompt. Data can include detailed product catalog information, past customer support chat logs, or other domain-specific data.

RAG is usually implemented using one of two techniques:

• Vector search, in which data is stored in high-dimensional vector embeddings and queried using a nearest-neighbor search; or
• Knowledge graphs, in which data is decomposed into nodes and edges and retrieved via tree reversal

The pros of RAG are that it:

• Takes less time to implement than fine-tuning
• Provides a faster and easier way to provide up-to-date context than fine-tuning
• Has proven highly effective at reducing LLM hallucinations

However, RAG’s convenience does come with some downsides, in that it:

• May not be able to provide finely tuned adjustments to facets of responses, such as tone and voice, that fine-tuning can provide.
• Can be more expensive per call. Commercial LLMs typically charge for usage by breaking requests down into tokens (typically one word = one token). The larger your prompt, the more you pay in LLM consumption.
• Can add additional latency to each call, as it requires calling another data source before calling the LLM.

The two approaches aren’t mutually exclusive. You might, for example, use fine-tuning to tailor an LLM’s tone and voice, while using RAG to provide relevant context.

Fine-tune your RAG apps with Astra DB and Langflow

Fine-tuning provides an option for high-performance tailoring of an LLM that provides more fine-grained control over the final result than other methods, such as RAG. While not every project requires it, it’s a good option to have in your toolbox when you need it.

For most GenAI use cases, you’ll want to start with RAG and then incorporate fine-tuning as needed. Using a highly-performant, serverless data store enables you to support RAG while adding minimal additional overhead to your GenAI app calls. Plus, with the right tools, you can get started using production-ready RAG, not in weeks or months, but in days.

Using Astra DB by DataStax in conjunction with Langflow, you can build highly scalable, RAG-powered GenAI apps in minutes. To learn more, check out our tutorial.