Some pretty good computer science got done before devops was gifted to the world.
And some pretty good science got done before computer scientists were gifted to the world.
I'm genuinely skeptical that modern software engineering practices are a good way of thinking about reproduction in science. Even in computer science. There's a lot that scientists can learn from software engineering (and in fact I've helped run workshops in the past on exactly this topic), but science is not engineering.
> Some pretty good computer science got done before devops was gifted to the world.
I'm happy to talk about this if you want. One of the most important aspects of this work was that people like Dijkstra started using notions that approached what real computers could read while remaining human-readable. This is some measure of classical "reproducibility". And work like McCarthy's was revolutionary in part because it was a definition of reproducibility as a result!
I can give examples of shockingly good papers that are struggling to see the light of day in their industry because they're written in ways that make them hard not only to understand, but to reproduce.
So don't presume to lecture me about this. Part of the reason the word2vec paper stands out is precisely because this is such a deviation from the norm to have a paper misrepresent its most fundamental component: the algorithm.
> if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.
This is the actually important definition of "reproducible".
"has an install.sh" is really nice, but it's far more important that an informed reader can recreate the artifact for themselves from the written description.
The "must be able to apt install or it's not real science" is a particularly dangerous path to go down. It's how you end up with e.g. shit loads of well-engineered LISP or FORTRAN code with no actual scientific insights or knowledge transfer. Which has actually happened in the past. A lot.
I don't think people in this thread are insisting on a downloadable package. Just that the algorithm published in the write-up accurately describe the work that was actually done to achieve the stated results. If even that threshold can't be met, IMO the authors are polluting the literature with bad work.
I am one of those people who actually did the extra weeks/months to properly test/review/document/release my code and data sets (you can apt-get install my "research artifacts").
In retrospect, it was a poor use of my time and a poor use of my sponsoring institution's time. "apt-get install" is NOT what we mean by "reproducible" in science.
High-quality or easy to install code is not necessary for a result to be reproducible. True reproduction would mean coding the algorithm from scratch by following the written description, and that's what it literally means in most other fields of science.
You can't download & install a Large Hadron Collider in an afternoon. Does that mean the LHC experiments are "not reproducible"? Of course not.
But that's not even the important point. The really important point is that, in most cases, high-quality code is not even sufficient for a result to be reproducible! See: the article we're discussing.
IMO, the "blindly rerun the code" definition of reproduction is actually a HUGE barrier to creating a true culture of reproducability in computer science. It results in super lazy reviewing where "public source code that's easy to install and puts the correct-sounding shit into STDOUT" becomes a stand-in for "paper actually describes a novel idea in enough detail that it can be truly reproduced".
> So what is the optimal outcome here?
An optimal allocation of scientists' time and effort.
As a scientist who has actually done that leg work, I don't think packaging code so that it runs with a single click is the best use of public money in science in 99.9999% of cases. That time is much better spent on writing and other dissemination explaining the ideas that make the code work (in some cases well-documented source code is the best description but in other cases prose is much more effective and illuminating). Or on coming up with new ideas that are even better than the old ones.
Which I guess is just another way of saying that scientists should spend their time on science, not engineering.
P.S. When shitting on "scientists" for not being good enough software engineers, please remember who's going to be doing the actual work you're demanding. It's mostly phd students who make $30K/yr. And they have to do this work in their free time because their 60 hr/wk day job is fully allocated to doing the actual science. I.e., treat scientists who maintain their code as you would treat FOSS contributors who are making 5x-10x+ less than you while working longer hours. Because maintaining high-quality code is something they are almost certainly doing in their free time.
>
High-quality or easy to install code is not necessary for a result to be reproducible. True reproduction would mean coding the algorithm from scratch by following the written description, and that's what it literally means in most other fields of science.
Which failed here. However, you're going to have a difficult time convincing many people that the attributes you described would be bad properties, just that they might not justify their expense.
> You can't download & install a Large Hadron Collider in an afternoon. Does that mean the LHC experiments are "not reproducible"? Of course not.
By the same token though, the LHC repeats experiments and solicits feedback on how to improve their methods, which they go to great lengths to publish and simulate, because they're aware of this problem.
> IMO, the "blindly rerun the code" definition of reproduction is actually a HUGE barrier to creating a true culture of reproducability in computer science. It results in super lazy reviewing where "public source code that's easy to install and puts the correct-sounding shit into STDOUT" becomes a stand-in for "paper actually describes a novel idea in enough detail that it can be truly reproduced".
Ah, yes. Yes. "If this code is TOO reproducible then people might reproduce it, and handwave handwave the quality of papers would decline.
That's certainly NOT the case in pure CS papers, which have only improved since the days when folks felt that "Lenses, Bananas and Barbed Wire" was how folks should go about writing papers.
Now, physics might be different. But there is surely a middle ground between, "I've shipped you a LHC just plug in in lol" and "This paper doesn't even remotely describe how we achived the results."
If you believe that wasting the time of scientists is bad, then surely you're for clear papers with accurate descriptions of the methods so that those who go and reproduce your work are not sent on wild goose chases?
> As a scientist who has actually done that leg work, I don't think packaging code so that it runs with a single click is the best use of public money in science in 99.9999% of cases.
No, we got that part. But surely someone does and maybe you can design your work to leverage that rather than reproducing and discarding scaffolding. My big concern here is that a lot of scientists (like you claim to be) are underqualified and unpracticed at software, and thus are surely seeing at least some aspect of their work distorted by software and hardware issues.
> Which I guess is just another way of saying that scientists should spend their time on science, not engineering.
Scientists are not going to be able to escape engineering. No one else is going to build what they need besides them.
> please remember who's going to be doing the actual work you're demanding. It's mostly phd students who make $30K/yr. And they have to do this work in their free time because their 60 hr/wk day job is fully allocated to doing the actual science.
Yeah, I'm aware. I suspect their lot would be better if your attitude wasn't that their work is disposable and unimportant.
Huh. The Verizon deal must be new. Not sure if that’s a good idea. The State Farm thing is a “marketing alliance” with Blue Cross. I don’t think State Farm sells health insurance.
It's telling that state GOPs typically support vouchers in the name of choice, but not actual choice within the public option.
Allowing any student to enroll at any public school regardless of their home zip code would be a much more meaningful form of choice than vouchers.
Why not just let anyone in a metro area attend high school at the richest/best high school in the area?
If you can answer that question, you can pretty well predict what systemic issues would be caused by a voucher system.
I'm generally skeptical of mixed public/private education systems with partial choice and subsidies for private options. We have exactly such a system for higher education and it's a fucking disaster.
You have 80% of the story but the last 20% is important.
Why not let anyone attend any public school? Because public schools are largely funded by local taxes. So a school in an expensive neighborhood with high taxes will naturally be a magnet for everyone else who can be a free-rider (in the economic theory sense). School vouchers fix this problem by carrying revenue along with the student instead of tying it to geography.
School vouchers with dollars attached to kids is effectively "choice, as long as you can pay for it". It's the status quo of zip code based schooling access, on steroids. That's exactly the "systemic issue" I'm referring to in my original post.
The claim that school choice would improve access to quality education for students in poor-performing schools is a complete farce.
I'm not saying that our current system works well or that it's particularly just. I'm just pointing out that vouchers make the tie between wealth and educational access even more explicit and codified than it already is. A voucher system would deepen, not alleviate, the inequities in our educational system.
I suspect we're just going to disagree on "as long as you can pay for it". In a publicly voucher system a large part of your ability to pay is provided by the government. This is not the case today. Vouchers take the ~$5-10k allocated for you that is "locked up" in your zip code and allows you to spend it anywhere you want. This doesn't fully equalize ability to pay but it's a large step toward equalizing it.
Secondly it fixes incentive structures. Good schools in high-tax areas can now see inbound students from elsewhere as partially subsidizing the cost of the school, rather than being a pure cost center. Again this doesn't perfectly equalize anything but it's a step in the right direction.
Allowing low income students more freedom in how they spend their government-allocated funds mitigates the problems of wealth disparity. This is already how the university system works so assertions that it is a complete farce should also justify the abolition of the FAFSA system in exchange for federal funds that can only be used at the university nearest your home.
I could consistently help even the poorest students move from below 50th percentile to 75th percentile. Moving from < 600 to mid 700s is totally doable with sufficient tutoring [1]. Even for pretty dumb students.
I think SAT/ACT are pretty good tests [1], but they're horrendously over-gamed at this point. I have very little faith in either as anything other than a demonstration of how badly the student wants to be admitted to a good school and has money for tutoring.
[1] edit: i.e., SAT/ACT are not easy to game wit short-term coaching, but sustained tutoring can substantially increase students' performance... see thread below for further elaboration and discussion of "coaching doesn't help" studies.
Nearly all of those studies focus on short-term and test-specific interventions; i.e., "coaching", not "tutoring".
I worked with students throughout the school year with a focus on the underlying content, and only switched to "coaching" the last few weeks before the exam. For many students, I tutored them weekly or biweekly, for 1-3 hours per week, for multiple years!
NONE of the studies on the effect of coaching consider the effect of this sort of longer-term individualized instruction.
I'm willing to believe that short-term coaching only has small effects, but sustained individual instruction has a huge impact on mathematical ability. And as I explicitly said in my original post, SAT/ACT do a good job of measuring that ability.
But claiming that sustained access to individualized high-quality teaching doesn't effect performance on subject-specific tests that require nontrivial content knowledge and practice is, on face, absurd. At the very least, the studies you're citing say absolutely nothing about this sort of sustained intervention.
(Also, College Board loves amplifying those studies. I wonder why...)
> For many students, I tutored them weekly or biweekly, for 1-3 hours per week, for multiple years!
This... is a feature, not a bug. The SAT is a tool to measure educational attainment, and you boosted scores by legitimately educating students. The SAT is not a test to measure natural ability (I can't believe this needs to be said, but so many people claim that no intervention should be able to boost SAT scores, and the only logical conclusion is that they want the SAT to measure some sort of unchangeable inborn ability? Of course, I think the actual problem is that they haven't realized that if you eliminate all environmental differences, all you're left with is the genetic lottery.)
But anyway, I think this is absolutely fine. Would you expect someone who hasn't gone to high school to do well on the SATs? Then why in the world would you think legitimate education shouldn't boost SAT scores?
There's a huge spectrum between "raw ability" and SAT/ACT. A good test would measure somewhere in the middle of that spectrum. It's not impossible to do, but is really hard.
The SAT (and other standardized tests) make a lot of sense when you're comparing people who have spent more-or-less the same amount of time and money preparing. They also make sense as one component of a holistic picture, weighed appropriately.
The the true value of these tests for predicting potential is a lot less useful otherwise.
The huge problem, from a predict-success perspective, is that you can't tell the difference between:
1. a brilliant person;
2. a kind-of-smart person who's very driven; and
3. an average person with no work ethic who was forced to sit with a tutor for many hours each weekend.
> Then why in the world would you think legitimate education shouldn't boost SAT scores?
It should. That's what the SAT is for. As I've said twice now, the SAT is a well-designed test. I don't think the SAT should change. I'm just now sure how useful it is, especially as a holistic measure.
To be really concrete about this: colleges should shy away from the SAT because I won't be holding those students hands forcing them to study and custom-designing their course of study at their first job!
At some point soon after graduating college, the hand holding disappears and you sink or swim. Academic preparation helps, but work ethic and the ability to learn on your own is really important. Colleges are, or at least should be, attempting to select people who are more likely to "swim".
If I were a college admissions officer, I'd probably weigh "good enough scores to know you're not an idiot, plus a compelling demonstration of grit and work ethic" WAY over "great scores with no demonstration of independent drive".
(FWIW I think we're now completely disconnected from the actual topic of the article, since that's not what the hardship score is measuring)
Oh okay, I think we mostly agree. People ragging on the SATs and other standardized exams for being teachable is a pet peeve of mine and I overreacted.
Back to the hardship score, I just don't think that the College Board should be in this business at all. Individual colleges certainly know where an applicant is coming from, and what high school they went to, and they have a lot more additional information not available to the College Board. So they have a much better idea of what hardships the applicant went through. Furthermore, different colleges want different things from their students which would and should lead to them weighing different kinds of hardships differently. Reducing all of this to a single number based on very coarse data is exactly the opposite of what holistic admissions is supposed to achieve.
Seems pretty clear to me: Google doesn't show the website link in the top 3 results but shows a lot of fact-checking sites in the top 10. It definitely should show the site at #1. The number of fact-checkers shown seems a little excessive but as long as they do something similar for equivalent left-wing sites, I don't have much of a problem with it - The Gateway Pundit publishes a lot of made up bullshit
Being able to learn these things on your own is exactly what a CS degree is supposed to teach.
In my career, I've taught myself how to make complex CLIs. Then CGI-based web apps. Then Java desktop apps. Then MVC frameworks. Then JS frameworks. Etc. Ditto for changes in development and deployment processes. Ditto for changes in DB technology. etc.
My employer rightfully expected that, for a six figure salary, I should be a professional who is able to keep up with best practices. That's why they pay me well. They let me attend conferences and I negotiate for some "experimentation" time whenever there's a big shift in the tech landscape, but it's not like I'm disappearing for months at a time every 5 years. That'd be insane.
I'm not very old.
Hell, I don't even work in web development.
It's just one of those skills I need to keep fresh because it comes in handy every once in a while.
The changes in my actual domain of expertise have been even larger and much more difficult to keep up with.
Coding bootcamps are expensive -- they often cost on the order of a single year of college. If you're going to need to go through a few of them in your career just to keep up with a single skill set -- building UIs -- that's more of a baseline competence than a real specialty, why not front-load and just get a good-quality degree that prepares you to teach yourself the easy stuff? Double major while you're at it.
Maybe at some point you will notice that once you have more responsibility in one area it will use up all our time and energy and you simply don't have time to learn other stuff on the side. I am in that situation right now and I think an 8 week bootcamp to focus on Machine Learning would be a good starting point. I could learn it myself but I would need some dedicated time to focus on it and not have to worry about my regular job.
We're commenting on an article about a coding camp founded by a professional TED talker who's now in law school.
The "they" pitching bootcamps and the "they" automating jobs away are two different "theys".
People who have been doing software for the last 20 years have always been saying "coding is hard" and "a few months in a coding bootcamp isn't sufficient preparation for a career in tech, especially when the winds of the market shift".
But no one wants to hear "high-paying jobs require spending years getting good at doing hard things".
FWIW I undowned your posts; internet karma is about communication/moderation, and sometimes has the opposite of the intended effect. I've been there.
Anyways, find some time to pet a dog/cat or fly a kite today.