I ran the code from my Fine-tuning “Connections” post using gpt-4o-mini.
I was hoping the results might be a bit better, which could motivate an effort to fine-tune the model.
I’m not sure where my original version of this code went, so I reconstructed a repo for it.
Once I was done, I ran 100 prompts through the model to get a sense of where its baseline performance was.
Correct: 2.00%
Incorrect: 98.00%
Total Categories Correct: 19.25%
Not great, and not much different from gpt-3.5-turbo.
With these kind of results, I wasn’t particularly motivated to put the effort in to do more fine tunes.
I read through the instructor, marvin and open-interpreter docs for the first time in a while.
It has been interesting to see these libraries grow and diverge.
I also read through how Jason has been structuring an evals repo for instructor.