Problem statement

Financial question answering means multi-step table reasoning over SEC filings. A model has to find the right rows, pull the numeric values, and run a calculation chain before it answers. Base models invent figures that were never in the table and attach values to the wrong columns. Training on the gold answer alone doesn't teach the model which intermediate steps lead to the right final number, because it never sees the steps.

Stage 1. Supervised fine-tuning

Llama-3-8B is fine-tuned on the full FinQA training split. The data formatter converts each raw example (pre_text, table rows, post_text, gold program, gold answer) into a ChatML conversation. The assistant turn uses an explicit chain-of-thought format: a <thought> block containing the supporting fact indices and the reasoning program from the FinQA annotation, followed by an <answer> block with the executed numerical answer. This forces the model to produce verifiable intermediate steps rather than jumping to a final number.

SFT uses LoRA (r=16, alpha=32, targeting all 7 projection layers, following Hu et al. 2022 and Dettmers et al. 2023 on reasoning tasks), a cosine LR schedule, 5% warmup, paged_adamw_32bit, and gradient checkpointing to fit on a single RTX 4090 (24 GB). Gradient accumulation brings the effective batch size to 48. A 5% holdout split catches overfitting during SFT, which matters because an overfit SFT initialization degrades the DPO reference model (Gao et al. 2022). Training dynamics: loss 2.74 → 0.785 over 248 steps, token accuracy 44% → 80.8%.

Stage 2. RLAIF preference data

The trained SFT model generates two candidate responses per training example at temperatures 0.3 and 0.9. A separate model, Zephyr-7B-β (Tunstall et al. 2023), annotates each pair using the constitutional AI pairwise rubric (Bai et al. 2022) to decide which response wins. Pairs with cosine similarity above 0.85 are dropped, because a pair that close carries no usable DPO signal. What comes out is 1,800 (prompt, chosen, rejected) triples in NDJSON. Annotating with Zephyr and evaluating with Llama-3.3-70B keeps the 2 jobs on separate models and avoids the circularity Zheng et al. 2023 identified.

Stage 3. Direct Preference Optimization

DPO trains on the RLAIF preference dataset. The SFT adapter goes in as the trainable model, and TRL clones it into the frozen reference model automatically. NF4 4-bit quantization through BitsAndBytes. Beta=0.3 (Rafailov et al. 2023 recommend 0.1 to 0.5, and 0.3 is a safe midpoint at this dataset size). LR=5e-6, 1 epoch, 38 steps. Final DPO loss 0.058, final reward accuracy 97.7%.

Evaluation with an LLM judge

A two-pass evaluation pipeline runs 3 model variants (Base, SFT, SFT+DPO) against 100 FinQA test questions, judged by Llama-3.3-70B through Groq (following Zheng et al. 2023). The rubrics are Faithfulness to Context (1 to 5), Reasoning Accuracy (1 to 5), and Hallucination Rate (present or not, per response). In pass 1 each model loads in 4-bit, generates its responses, then unloads to free GPU memory before the next one loads. In pass 2 every response is scored on all 3 rubrics. Results go to CSV with partial checkpoints, so an interrupted run resumes without re-scoring the models it already finished.

SFT+DPO lifts Faithfulness from 1.17 to 1.29 and Reasoning Accuracy from 1.16 to 1.56 against the base model, and drops the Hallucination Rate from 96.0% to 86.7%. Those absolute scores sit low on a 1 to 5 scale, which says more about how hard multi-step SEC table reasoning is than about the training signal: DPO reward accuracy reached 97.7%, so the model is learning the preference structure. The honest read is that this task needs a larger base model, and the pipeline is ready for one.

Compute

The training scripts run on a single GPU (RTX 4090, RunPod) or across multiple GPUs with DDP through torchrun. The SFT script reads LOCAL_RANK from the environment and pins each process to its assigned GPU. A heartbeat thread prints progress every 10 minutes, which is what you want on a long cloud run. Adapters and the merged model are pushed to the Hugging Face Hub under G-Maxime-N/.