I recently explored OpenAI’s gpt-oss:20b, its first open-source model based on GPT-4 with a June 2024 knowledge cutoff. This model can also perform web searches, potentially enhancing its capabilities. I decided to test gpt-oss:20b against a real-world challenge, utilizing a UK 11+ practice test. Using my RTX 5080 GPU, which struggled with the model’s demands, I prompted the AI to solve the test questions.
The results were disappointing; it only correctly answered 9 out of 80 questions, often producing completely irrelevant responses. Despite demonstrating some reasoning skills in its output, the model frequently skipped problems or generated its own quiz instead of answering my prompts. Increasing the context length improved its performance slightly but still resulted in slow processing and irrelevant outputs. Overall, while my 10-year-old son outperformed the AI, the experiment yielded valuable insights into AI reasoning and limitations, despite the challenges posed by my hardware.
Source link
