Command Line¶

This described the is the command line interface for mteb.

mteb is a toolkit for evaluating the quality of embedding models on various benchmarks. It supports the following commands:

• mteb run: Runs a model on a set of tasks
• mteb available_tasks: Lists the available tasks within MTEB
• mteb available_benchmarks: Lists the available benchmarks
• mteb create_meta: Creates the metadata for a model card from a folder of results
• mteb leaderboard: Runs the MTEB leaderboard locally

In the following we outline some sample use cases, but if you want to learn more about the arguments for each command you can run:

mteb {command} --help

Running Models on Tasks¶

To run a model on a set of tasks, use the mteb run command. For example:

mteb run -m sentence-transformers/average_word_embeddings_komninos \
         -t Banking77Classification EmotionClassification \
         --output-folder mteb_output

This will create a folder mteb_output/{model_name}/{model_revision} containing the results of the model on the specified tasks supplied as a json file; {task_name}.json.

Listing Available Tasks¶

To list the available tasks within MTEB, use the mteb available-tasks command. For example:

mteb available-tasks # list _all_ available tasks

You can also use the multiple arguments for filtering:

mteb available-tasks --task-types Retrieval --languages eng # list all English (eng) retrieval tasks

Listing Available Benchmarks¶

To list the available benchmarks within MTEB:

mteb available-benchmarks # list all available benchmarks

Creating Model Metadata¶

Once a model is run you can create the metadata for a model card from a folder of results, use the mteb create-meta command. For example:

mteb create-meta --results-folder mteb_output/sentence-transformers__average_word_embeddings_komninos/{revision} \
                 --output-path model_card.md

This will create a model card at model_card.md containing the metadata for the model on MTEB within the YAML frontmatter. This will make the model discoverable on the MTEB leaderboard.

Running the Leaderboard¶

To run the MTEB leaderboard locally, use the mteb leaderboard command. For example:

mteb leaderboard

You can specify a custom cache path and other options:

mteb leaderboard --cache-path results --port 8080 --share

Available options: - --cache-path PATH: Custom path for model results cache - --host HOST: Host to run the server on (default: 0.0.0.0) - --port PORT: Port to run the server on (default: 7860) - --share: Create a public URL for the leaderboard - --rebuild: Force rebuild from full results repository, bypassing cached JSON

For more details on running the leaderboard, see the leaderboard documentation.