Evidence-Oriented Programming

Bringing randomized controlled trials to human factors decisions in programming language design. You know ... science.

Programming languages should be designed with human factors as a primary concern.

Traditional programming languages have been designed predominately with technical concepts and machines in mind. While such concerns are obviously critical, human beings ultimately use such tools in the broad development community. In evidence-oriented programming, human factors evidence takes a first-class seat in the language's design. All factors related to programming are considered, up for debate, and are subject to change if a community member shows rigorous evidence that another approach is better. This is true both for technical and human factors considerations. To our knowledge, Quorum is the first programming language to attempt this.

How does evidence-oriented programming work?

What evidence is there for Quorum already?

Programming language design is complicated, as is likely obvious. No amount of study will fix all usability or human factors concerns, or appease everyone, but the general goal is to make the greatest number of humans as productive as possible. So far, we have published and collaborated with a team of scientists, world-wide, on many systems, both in and out of Quorum. For example, the evidence on Quorum's syntactic choices is extensive. Similarly, we used all known empirical studies on the design of type systems when designing Quorum's. While this description barely scratches the surface, published works on the design are available for those that want more information.

What is the standard of evidence for core changes to Quorum?

Quorum requires a standard of evidence for language features that are commensurate with the change being requested. For example, core designs for the syntax have already been vetted in randomized controlled trials with human beings (e.g., see Stefik and Siebert) Thus, changes require more, or better, evidence. For core changes, we highly recommend familiarizing yourself with the What Works Clearinghouse standard. Anecdotal claims regarding core feature changes will almost certainly be rejected. Generally, the programming language wars appear to be too complex to solve with beliefs alone.

What is the standard of evidence for Quorum's standard library?

Submissions to add to the standard library require less evidence. We ask that the submission follows the coding standard, that the library actually works, and it provides some kind of useful functionality. If you want to replace an existing part of the standard library, the burden is on you to prove that the change benefits human beings. Again, claims must be actionable and anecdotes are not acceptable, unless the claim is really obvious (e.g., a submission that fixes a known bug).

Submitting a change or library to Quorum

The benefits of evidence-oriented design

Since Quorum is evidence based, the entire community has a buy-in toward adjusting and improving the language over time. To keep the language as stable as possible, core language features should be unchanged without solid empirical evidence. We feel this is especially important, given the long history of language designers using little evidence (if any) when making human factors decisions. Given that evidence does require the language to change, from time-to-time, developers are informed of important changes as soon as possible. This is why, for example, the Quorum designers started with syntax and type systems in controlled trials. The bar to adjust these features is, thus, extremely high.