PSEUDO CHAIN OF THOUGHT INDUCED FINE TUNING FOR LARGE LANGUAGE MODELS

Abstract

The current state of Large Language Models is filled with hallucinations and miscalculations, since the probabilities of the initial prompt and generations are not completely aligned with the probability distributions of the hypothetical right answer.

Therefore, a chain of thought process allows the model probability to align more closely to that of the expected output by reiterating over internal thoughts. The framework for inducing chain of thought to Large Language Model is to, first fine tune the main LLM on specific data required and further a smaller adapter model or also called as Tiny Thought Model is obtained from two-step model distillation process to get a model in the order of 100M-1B parameters that is focused on creating thoughts and questions based on inputs.

Finally, the Main LLM and the Tiny Thought Model are joined together via a router that decides if a thought is necessary to reiterate over the model answer, hence generating internal thought. A synthetic question answer pairs of the order of 5000-10000 QnA’s are generated using an existing document such as a scientific or mathematical textbooks to train the Tiny Thought Model and also finetune the main LLM.

Further SAT Math and science datasets are used for generalized evaluations and test QnA’s dataset for document specific evaluation. Measures such as Rouge, BLEU and Accuracy are going to be used to determine the quality of the answer. The base LLM with finetuning and base LLM without fine tuning are to be used as control measures. Models such as Llama 3.1 8B, Mistral 7B v0.3, Gemma 2 9B, Llama 3.2 10B will be used for comparisons.