Reproducible Research

Replication and the Reproducibility Spectrum

• Full replication is a huge challenge, and sometimes impossible, e.g.
  • rare phenomena, long term records, very expensive projects like space missions, etc
• Where the “gold standard” of full replication cannot be achieved, we have to settle for a lower rung somewhere on The Reproducibility Spectrum (Peng 2011)

Why work reproducibly?

Let’s start being more specific about our miracles… Cartoon © Sidney Harris. Used with permission ScienceCartoonsPlus.com

Why work reproducibly?

“Five selfish reasons to work reproducibly” (Markowetz 2015)

1. Its transparent and open - helping avoid mistakes or track down errors
2. It makes it easier to write papers - faster tracking of changes and manuscript updates
3. It helps the review process - reviewers can actually see what you did
4. It enables continuity of research - simplifying project handover (esp. past to future you!)
5. It builds reputation - showing integrity and gaining credit where your work is reused

Why work reproducibly?

Some less selfish reasons (and relevant for ecoforecasting):

6. It speeds progress in science by allowing you (or others) to rapidly build on previous findings and analyses

7. It allows easy comparison of new analytical approaches to older ones

8. It makes it easy to repeat analyses on new data, e.g. for ecological forecasting or LTER¹

9. The tools used are useful beyond Reproducible Research, e.g. building websites

10. Reproducible research skills are highly sought after!

• Skills are important should you decide to leave science…
• Within science, more and more environmental organizations and NGOs are hiring data scientists or scientists with strong data and quantitative skills

Barriers to working reproducibly

From “A Beginner’s Guide to Conducting Reproducible Research” (Alston and Rick 2021):

1. Complexity

• There’s a learning curve in getting to know and use the tools effectively
• One is always tempted by the “easy option” of doing it the way you already know or using “user-friendly” proprietary software

2. Technological change

• Hardware and software used in analyses change over time, making it very difficult to rerun old analyses etc
  • This should be less of a problem as more tools like contained computing environments become available

Barriers to working reproducibly

3. Human error

• Simple mistakes or poor documentation can easily make a study irreproducible.
• Most reproducible research tools are actually aimed at solving this problem!

4. Intellectual property rights

• Rational self-interest can lead to hesitation to share data and code via many pathways:
  • Fear of not getting credit; Concern that the materials shared will be used incorrectly or unethically; etc
• Hopefully most of these issues will be solved by better awareness of licensing issues, attribution, etc, as the culture of reproducible research grows

Reproducible Scientific Workflows

‘Data Pipeline’ from xkcd.com/2054, used under a CC-BY-NC 2.5 license.

Working reproducibly requires careful planning and documentation of each step in your scientific workflow from planning your data collection to sharing your results.

2. File and folder management

3. Coding and code management

Why write code?

“Point-and-click” software like Excel, Statistica, etc may seem easier, but you’ll regret it in the long run…

Why write code?

• Automation - reusing code is one click, and you’re unlikely to introduce errors
• A script provides a record of your analysis
• Uninterrupted workflows - scientific coding languages like Python or R allow you to run almost any kind of analysis in one scripted workflow
  • GIS, phylogenetics, multivariate or Bayesian statistics, etc
  • saves you manually exporting and importing data between softwares
• Most coding languages are open source (e.g. R, Python, JavaScript, etc)
  • Free! No one has to pay to reuse any code you share
  • Transparent - You (and others) can check the background code and functions you’re using, not just the software company
  • A culture of sharing code (online forums, with publications, etc)

Some coding rules

It’seasytowritemessyindecipherablecode!!! - Write code for people, not computers!!!

Check out the Tidyverse style guide for R-specific guidance, but here are some basics:

• use consistent, meaningful and distinct names for variables and functions
• use consistent code and formatting style - indents, spaces, line-breaks, etc
• modularize code into manageable steps/chunks
  • or separate scripts that can be called in order from a master script or Makefile
• use commenting to explain what you’re doing at each step or in each function
  • “notebooks” like RMarkdown, Quarto, Jupyter or Sweave allow embedded code, simplifying documentation, master/Makefiles, etc and can used to write manuscripts, presentations or websites (e.g. all my teaching materials)
• write functions rather than repeating the same code
• check for mistakes at every step!!! Do the outputs make sense?

Some coding rules continued…

• start with a “recipe”
  • outline the steps/modules before you start coding to keep you on track
  • e.g. a common recipe in R (using commented headers):

#Header indicating purpose, author, date, version etc

#Define settings and load required libraries

#Read in data

#Wrangle/reformat/clean/summarize data as required

#Run analyses (often multiple steps)

#Wrangle/reformat/summarize analysis outputs for visualization

#Visualize outputs as figures or tables

• avoid proprietary formats! i.e. use open source scripting languages and file formats
• use version control!!!

Version control

Version control tools can be challenging , but also hugely simplify your workflow!

The advantages of version control¹:

• They generally help project management, especially collaborations
• They allow easy code sharing with collaborators or the public at large - through repositories (“repos”) or gists (code snippets)
• The system is online, but you can also work offline by cloning the repo to your local PC. You can “push to” or “pull from” the online repo to keep versions in sync.
• Changes are tracked and reversible through commits
  • Any changes in a repo must be commited with a commit message. Each commit is a recoverable version that can be compared or reverted to
  • This is the essence of version control and magically frees you from duplicated files!

Version control continued…

• Users can easily adapt or build on each others’ code by forking repos and working on their own branch.
  • This allows you to repeat/replicate analyses or even build websites (like this one!)
• Collaborators can propose changes via pull requests
  • Repo owners can accept and integrate changes seamlessly by review and merge the forked branch back to the main branch
  • Comments associated with commit or pull requests provide a written record of changes and track the user, date, time, etc - all of which and are useful tracking mistakes and blaming when things go wrong
• You can assign, log and track issues and feature requests

This should all make more sense after the practical, but here are some pretty pictures to drive some of this home…

Version control

Artwork by @allison_horst CC-BY-4.0

Version control

Artwork by @allison_horst CC-BY-4.0

4. Computing environment

Sharing your code and data is not enough to maintain reproducibility…

Software and hardware change with upgrades, versions or user community preferences!

• You’ll all know MicroSoft Excel, but have you heard of Quattro Pro or Lotus that were the preferred spreadsheet software of yesteryear?

The simple solution is to carefully document the hardware and versions of software used so that others can recreate that computing environment if needed.

• In R, you can simply run the sessionInfo() function, giving details like so:

R version 4.2.2 (2022-10-31)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.0

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] compiler_4.2.2  fastmap_1.1.0   cli_3.6.0       tools_4.2.2    
 [5] htmltools_0.5.4 rstudioapi_0.14 yaml_2.3.7      rmarkdown_2.20 
 [9] knitr_1.42      xfun_0.36       digest_0.6.31   jsonlite_1.8.4 
[13] rlang_1.0.6     evaluate_0.20  

4. Computing environment cont.

Containers

A better solution is to use containers like docker or singularity.

These are contained, lightweight computing environments similar to virtual machines, that you can package with your software/workflow.

You set your container up to have everything you need to run your workflow (and nothing extra), so anyone can download (or clone) your container, code and data and run your analyses perfectly first time.

5. Sharing data, code, publication etc