The performance assessment of Woollie models, developed from pretrained Llama models, demonstrated significant improvements in logic and medical benchmarks. Through stacked alignment and fine-tuning, fourteen Woollie models were evaluated across 11 standardized benchmarks, specifically targeting sizes of 7B, 13B, 33B, and 65B parameters. Metrics such as accuracy, F1 scores, and Matthews Correlation Coefficients (MCC) highlighted intentional focus on accuracy for comparative analysis due to its prevalence in leaderboard rankings. Woollie models outperformed baseline Llama models, particularly in medical domains like PubMedQA, achieving up to 0.81 accuracy. Notably, the 65B model performed exceptionally in conversation-centric tests. Additionally, the Woollie models maintained high accuracy while effectively mitigating catastrophic forgetting through stacked alignment methods. Further fine-tuning on oncology datasets enabled the Woollie MSK models to achieve an accuracy of 0.90 for cancer progression prediction, underscoring their practical applicability in clinical settings. Overall, Woollie’s advancements signal robust potential for enhancing medical AI applications.
Source link
