Training data and base model of d1-2B (research pre-registration)

#9
by clemsail - opened

We are pre-registering a confirmatory study (Zuse-2, Hypneum Lab; protocol on GitHub, hypneum-lab/zuse-2, docs/prereg/zuse-2/) that evaluates public decision models on held-out test splits of public benchmarks and codes, per (model, dataset) cell, whether the model's training data declared a training split of the same source. Cells we cannot resolve from the model card are kept as "unknown" and removed in a sensitivity analysis.

Could you tell us, for the datasets listed below, whether any split of them (or of a derived dataset) was part of the training or fine-tuning data of LiquidAI/d1-2B at revision current main? A short list, or "none of these", is all we need; we will cite your answer (this thread) in the protocol.

Specific question: the card does not list training sources. Which public datasets (and splits) were used, and which base model?

Datasets of the panel (100 text datasets, 20 per family; the Hub repo is the one we pin):

family dataset Hub repo config
knowledge arc_challenge Praveenrajus/jev-bench arc_challenge
knowledge arc_easy allenai/ai2_arc ARC-Easy
knowledge bertaqa_en HiTZ/BertaQA en
knowledge csqa2 tasksource/commonsense_qa_2.0
knowledge culturalbench_easy kellycyy/CulturalBench CulturalBench-Easy
knowledge geobench_ap daven3/geobench
knowledge head_qa_en dvilares/head_qa en
knowledge legalbench_citizenship nguha/legalbench international_citizenship_questions
knowledge medmcqa openlifescienceai/medmcqa
knowledge medxpertqa_text TsinghuaC3I/MedXpertQA Text
knowledge mmlu Praveenrajus/jev-bench mmlu
knowledge mmlu_cf microsoft/MMLU-CF
knowledge mmlu_pro TIGER-Lab/MMLU-Pro
knowledge qasc allenai/qasc
knowledge sanskriti 13ari/Sanskriti
knowledge scienceqa derek-thomas/ScienceQA
knowledge sciq allenai/sciq
knowledge supergpqa m-a-p/SuperGPQA
knowledge truthfulqa_mc1 truthfulqa/truthful_qa multiple_choice
knowledge wmdp cais/wmdp wmdp-bio,wmdp-chem,wmdp-cyber
nli anli facebook/anli plain_text
nli cad_snli tasksource/counterfactually-augmented-snli
nli chaosnli Praveenrajus/jev-bench chaosnli
nli chaosnli_snli earino/chaosnli snli
nli contract_nli kiddothe2b/contract-nli contractnli_a
nli defeasible_nli tasksource/defeasible-nli atomic + snli
nli flute ColumbiaNLP/FLUTE default
nli impli_gold tasksource/idioms-nli default
nli mnli Praveenrajus/jev-bench mnli
nli mnli_mismatched nyu-mll/multi_nli default
nli msci_nli sadat2307/MSciNLI default
nli propsegment_nli sihaochen/propsegment nli
nli pubhealth bigbio/pubhealth pubhealth_bigbio_pairs
nli scinli tasksource/scinli default
nli scitail allenai/scitail snli_format
nli sick RobZamp/sick
nli snli stanfordnlp/snli plain_text
nli wanli alisawuffles/WANLI default
nli wice tasksource/wice default
nli xnli_en facebook/xnli en
reading attributionbench osunlp/AttributionBench full_data
reading belebele_eng facebook/belebele eng_Latn
reading boolq Praveenrajus/jev-bench boolq
reading case_hold coastalcph/lex_glue case_hold
reading condaqa lasha-nlp/CONDAQA
reading cosmos_qa allenai/cosmos_qa
reading faithdial_begin McGill-NLP/FaithDial plain_text
reading fever_evidence Praveenrajus/jev-bench fever_evidence
reading fib r-three/fib
reading health_fact ImperialCollegeLondon/health_fact
reading maud theatticusproject/maud
reading multirc aps/super_glue multirc
reading onestopqa malmaud/onestop_qa
reading privacyqa alzoubi36/privacy_qa
reading pubmedqa qiaojin/PubMedQA pqa_labeled
reading quail textmachinelab/quail quail
reading race ehovy/race all
reading strategyqa_grounded Praveenrajus/jev-bench strategyqa_grounded
reading summedits Salesforce/summedits
reading vitaminc tals/vitaminc
routing banking77 Praveenrajus/jev-bench banking77
routing clinc150 Praveenrajus/jev-bench clinc150
routing congress_policy_area hheiden/us-congress-bill-policy-115_117
routing cuad_clause dvgodoy/CUAD_v1_Contract_Understanding_clause_classification
routing dbpedia_14 fancyzhx/dbpedia_14 dbpedia_14
routing fb_mtod_en edmdias/multilingual-task-oriented-dialog en
routing hwu64 FastFit/hwu_64
routing ibm_arg_topic ibm-research/argument_quality_ranking_30k argument_topic
routing ledgar coastalcph/lex_glue ledgar
routing lexglue_scotus coastalcph/lex_glue scotus
routing massive Praveenrajus/jev-bench massive
routing mtop_domain_en tasksource/mtop
routing natcs_intent splevine/dstc11-intent
routing snips_2017 benayas/snips
routing snips_builtin sonos-nlu-benchmark/snips_built_in_intents
routing taskmaster2_domain google-research-datasets/taskmaster2 flights,food-ordering,hotels,movies,music,restaurant-search,sports
routing topv2_domain WillHeld/top_v2
routing tweet_topic_single cardiffnlp/tweet_topic_single tweet_topic_single
routing twitter_financial_news_topic zeroshot/twitter-financial-news-topic
routing unfair_tos_cls nguha/legalbench unfair_tos
safety aegis_1 nvidia/Aegis-AI-Content-Safety-Dataset-1.0
safety aegis_2 nvidia/Aegis-AI-Content-Safety-Dataset-2.0
safety azstar_incivility civility-lab/incivility-arizona-daily-star-comments
safety civil_comments Praveenrajus/jev-bench civil_comments
safety conv_abuse tasksource/lewidi conv_abuse
safety davidson tdavidson/hate_speech_offensive
safety diasafety thu-coai/diasafety
safety disagg_offensiveness pia-p/disaggregated_offensiveness
safety do_not_answer_resp LibrAI/do-not-answer
safety edos tasksource/edos
safety gab_hate_corpus juliadollis/The_Gab_Hate_Corpus_ghc_test_original
safety hatecheck Paul/hatecheck
safety hatexplain Hate-speech-CNERG/hatexplain
safety mastodon_harmful ChaseLabs/Harmful-Texts-On-Mastodon
safety measuring_hate_speech Praveenrajus/jev-bench measuring_hate_speech
safety openai_moderation mmathys/openai-moderation-api-evaluation
safety prosocial_dialog allenai/prosocial-dialog
safety sbic allenai/social_bias_frames
safety toxic_chat lmsys/toxic-chat toxicchat0124
safety twitcivility incivility-UOH/TwitCivility

Vision (10 datasets, reported outside the primary analysis): mmmu_pro, cv_bench, blink, scienceqa_img, seed_bench, snli_ve, vsr, naturalbench, muirbench, pope (repos in docs/prereg/zuse-2/vision-axis-draft.md).

Thank you.

Sign up or log in to comment