I’ve recently read “The Game I Mean: Game Reference, Citation and Authoritative Access”, by Kaltman, Mason, and Wardrip-Fruin.

This paper discusses current and best practices for citing games in scientific articles. Although the paper is laser focused on the citation of digital games (and quite interesting in that regard!) It made me thing a lot about how we cite things in general, even in my own fields of Artificial Life, Artificial Intelligence, and Metaheuristic Optimization.

One thing that really caused an impression on me was the discussion on how to cite specific code in games. The article advocates for citations that include specific files and lines of code when discussing the implementation of certain algorithms or functions in games. In the optimization literature (and the Machine Learning Literature as well), a very large majority of our papers are discussing, criticizing, and building on algorithms from other researchers, and I think I’ve never seen a code citation!

Not only that, thinking about it, I think the vast majority of published algorithms (and these fields are all about algorithms) do not have a canonical implementation. You need to either hunt the internet in the hopes that one of the authors published an implementation somewhere (and even more rarely these implementations are tied to any particular papers), or try to reinvent the code from the paper discussion.

There is (or used to be?) an argument that the algorithm in general is more important than any particular implementation (in some sort of platonic ideal), but I find that very hard to defend, specially when the majority of evidence for these algorithms in the literature comes from empirical experiments, necessarily tied to particular implementations.

So we should really not only publish our code consistently (there are several papers on open code standards these days), but we should also make an effort to clearly tie particular versions of code to particular publications.

This paper also made me think about the difficulties of doing empirical AI research based on commercial games, where changes in version and availability of games can have an enormous effect on the replicability and reproducibility of an experiment. But that is another topic, to be discussed in some other occasion.