Day 10 - Side Quest (Reproducibility)

September 2nd, 2025:

• Thoughts reading A Step Toward Quantifying Independently Reproducible Machine Learning Research (opens in a new tab):
  • Forgot why started reading this. Anyway, super happy to see this paper and start reading it, im kinda getting annoyed with papers not being reproducible, especially when papers make mention of the need for reproducibility. I thought the scientific method is what sorta unified the modern idea of “research” (i.e. we ask questions, hypothesize, experiment, analyze, and conclude) but seeing numbers of papers not being reproducible from different fields ranging from 60-70% is super disheartening. Like whats the point of publishing with all this jargon terminology and extra tests and results, if fewer than 30% of “research” papers are just not reproducible.
  • Ok, found out about the "reproducibility crisis”, im not alone. But this just makes me a lot more hesitant to even try and do research, there's just way too many uncontrollable factors, especially with computer science. Imma guess in 20 years, that 30% would drop to less than 10% being reproducible just cuz links to external platforms would probably change, people delete their github, hacks destroy sole source of datasets, etc. Like all this knowledge is just gonna be lost and inaccessible. TODO: read about historical similar cases where knowledge is lost. Anyway, in terms of yielding results, i cant deny lots of papers do but in terms of the lifetime of the 2025 knowledge we have, i think a lot of it will be lost completely in the next 100 years, if not a majority of it.
  • Honestly, im surprised by these journals not having way way more rigorous methods of ensuring results are reproducible. I understand, its basically money. There is a preference to publish and continue building on top of it to get more funding etc etc etc. Just seeing allllll of these pages and pages of policies and what not making me think it meant higher standards just makes me respect these journals way less, like all those pages and still 70% of papers not reproducible?!?! Like bro, i do not understand the underpinning of what research today even is. Honestly, this just makes me understand more how the Quran is divinely protected. Like, over a thousand years and its still the most timeless book and applies to today, what man can write any work WITHOUT errors. The “creme de la creme” of labs failing to reproduce the “creme de la creme” of research (opens in a new tab) is sad. Anyway, im biased and i barely finished reading a full page within the pages of policies although im sure there are probably policies and rules that encourage and enforce certain levels of reproducibility. Sad, i gotta think about if i wanna continue reading and eventually doing “research” or just try bare bone experimenting a bunch without giving a flying f about reproducibility and writing papers to fill in the knowledge gap. Makes me think the research field is just clout seeking, I should look and find researchers who dont care at all about clout but purely care about seeking to expand our knowledge in fields.
  • A thought came which is that i should give benefit of doubt and assume overwhelming majority of researchers are trying their best to make results as reproducible as possible. I think there is some lesson in this that most cant stand the test of time, and that research has a “lifespan” until it becomes unusable.
  • A thought came which is that researchers focus so much on running tests and experimenting to distinguish and determine the best from the worst and that in between, similarly Quran says so much about humans being put to test. In our case, humans are given the answer guide and following it through life’s tests helps us.
  • Another thought, work in arabic so it lasts longer and can make more impact long term.
• Found Peer Review in Scientific Publications: Benefits, Critiques, & A Survival Guide - PMC (opens in a new tab):
  • Seems relevant to understand the whole research process, not just from the researcher perspective.
  • Seems like lots more AI being used to review papers which give general reviews and not much specific
  • TODO: check out the auto-reviewing tools mentioned here (opens in a new tab) (ironically from ChatGPT) and read through the relevant papers
• Found ARR Responsible NLP Research checklist (opens in a new tab) which seems relevant to reproducibility, created by Jesse Dodge, how small this world is.
  • Another relevant paper to consider is Show Your Work: Improved Reporting of Experimental Results (opens in a new tab) whenever posting results.
  • Should take a looksy through these: https://reproml.org/proceedings/ (opens in a new tab)
  • I guess i should learn about what most researchers consider a paper ot be reproducible
  • oof went on tangent and saw the electrical costs of training/inference: AI Revolution: Unintended consequences for the environment | KTVU FOX 2 (opens in a new tab). Honestly, i feel like theres a lotta politics involved with trying to curb CO2 emissions
• I have way too many tabs, i gotta finish reading these tabs before trying to queue up new ones. Wish i had a better queue system