Spaces:

---

mteb

/

leaderboard

like 7.44k

Running on CPU Upgrade

The MTEB Leaderboard fails to load and throws a Runtime Error.

Exit code: 2. Reason: ect(benchmark.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mteb/mteb/leaderboard/app.py", line 804, in on_benchmark_select
_, summary_raw = apply_summary_styling_from_benchmark(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mteb/mteb/leaderboard/table.py", line 83, in apply_summary_styling_from_benchmark
summary_df = benchmark_instance._create_summary_table(benchmark_results)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mteb/mteb/benchmarks/benchmark.py", line 614, in _create_summary_table
joint_table = self._create_vidore_summary_table(benchmark_results)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mteb/mteb/benchmarks/benchmark.py", line 561, in _create_vidore_summary_table
model_metas.map(lambda m: int(m.embed_dim) if m.embed_dim else None),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mteb/.venv/lib/python3.12/site-packages/pandas/core/series.py", line 4719, in map
new_values = self._map_values(arg, na_action=na_action)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mteb/.venv/lib/python3.12/site-packages/pandas/core/base.py", line 925, in _map_values
return algorithms.map_array(arr, mapper, na_action=na_action, convert=convert)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mteb/.venv/lib/python3.12/site-packages/pandas/core/algorithms.py", line 1743, in map_array
return lib.map_infer(values, mapper, convert=convert)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pandas/_libs/lib.pyx", line 2999, in pandas._libs.lib.map_infer
File "/mteb/mteb/benchmarks/benchmark.py", line 561, in
model_metas.map(lambda m: int(m.embed_dim) if m.embed_dim else None),
^^^^^^^^^^^^^^^^
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'list'
make: *** [Makefile:82: run-leaderboard] Error 1