@@ -29,16 +29,19 @@ Camelidae-8x34B | [🤗HuggingFace](https://huggingface.co/hywu/Camelidae-8x34B
2929## Performance
3030| Model | MMLU (5shot) | GSM8k (5shot) | MATH (4shot) | HumanEval (0shot) | MBPP (4shot) | HellaSwag (10shot) | TriviaQA (0shot) |
3131| ----------------------:| :------------:| :-------------:| :------------:| :-----------------:| :------------:| :------------------:| :----------------:|
32- | GPT3.5 | 70.0% | 57.1% | 34.1% | 48.1% | - | 85.5 | - |
33- | Camelidae-8x34B | 75.6% | 78.3% | 22.6% | 43.9% | 41.4% | 85.3% | 63.4% |
34- | SUSChat-34B | 76.4% | 72.3% | 22.0% | 11.6% | 40.2% | 83.9% | 56.1% |
35- | Mixtral-8x7B-instruct | 68.7% | 71.7% | 22.1% | 25.6% | 40.6 | 85 .5% | 57.7% |
32+ | GPT3.5 | 70.0% | 57.1% | ** 34.1%** | ** 48.1%** | - | 85.5% | - |
33+ | Camelidae-8x34B | 75.6% | ** 78.3%** | ** 22.6%** | ** 43.9%** | ** 41.4%** | 85.3% | ** 63.4%** |
34+ | SUSChat-34B | ** 76.4%** | 72.3% | 22.0% | 11.6% | 40.2% | 83.9% | 56.1% |
35+ | Mixtral-8x7B-instruct | 68.7% | 71.7% | 22.1% | 25.6% | 40.6% | ** 86 .5%** | 57.7% |
3636| LLaMA2-70B-chat | 63.8% | 59.3% | 10.4% | 32.3% | 35.6% | 84.8% | 63.0% |
3737| Camelidae-8x13B | 54.4% | 52.6% | 9.8% | 30.6% | 30.4% | 82.5% | 59.4% |
3838| LLaMA2-13B-chat | 54.6% | 37.1% | 5.2% | 18.9% | 27.2% | 81.9% | 55.0% |
3939| Camelidae-8x7B | 48.3% | 44.0% | 5.8% | 18.3% | 23.4% | 79.2% | 51.0% |
4040| LLaMA2-7B-chat | 48.3% | 26.3% | 3.9% | 12.2% | 17.6% | 78.6% | 46.4% |
4141
42+ We bold the highest scores for open-source models and all models separately.
43+ 44+ 4245## Usage
4346``` python
4447from transformers import AutoModelForCausalLM, AutoTokenizer
0 commit comments