Moreover, the students gain valuable machine learning experience via studying the papers and implementing the systems therein, and the community benefits from open source implementations of these machine learning systems. The latter would spur on faster research iterations and a faster incorporation of cutting edge systems into open source machine learning libraries.

Obviously, there are details to discuss and a couple of issues to work out. Will students also need to write a report/do a presentation on the research contained in each paper? Will these paper implementation projects replace or supplement the typical semester-long research project? In order to prevent a duplication of efforts, should professors/universities coordinate to prevent paper implementation overlap? If the choice for which papers are to be implemented is left up to the instructor, will that introduce an unwanted bias? Should the professors refrain from asking their students to implement papers from his/her research group? Should there be some central repository/archive of paper implementations? What about a list of papers that have yet to be implemented? Perhaps people should be able to upvote papers that they wish to see implemented, and the un-implemented list is ordered by the number of votes received?

A lot of stuff to consider.

]]>We’ll be able to pass this obstacle of papers being not using standards by training a deep learning model that can classify the papers to a number of similarity classes. ]]>

It’s a (bad) cognitive bias to underestimate the “engineering part” of the research as it will ultimately lead to the inefficient “reinvent the wheel” bad practice mentioned in the paper.

I think one possible strategy could consist of having the Community put some effort in “Standardization” : of tools (Dataset, Framework, Metrics, …) and communication (formal exposition of ideas and results) ]]>

I must say however that there is a substantial difference between the deep learning community and academia as in deep learning many aspects have not been explained nor have any mathematical proof, which make the hidden variables you mention more of an intrinsic feature rather than a real issue. One more reason why deep learning is not a pure academic subject. ]]>