Replicability and Reproducibility in Geospatial Research: A SPARC Workshop

ASU's Spatial Analysis Research Center (SPARC) hosted its first workshop, focused on the subjects of Replicability and Reproducibility, at Arizona State University on February 11 and 12, 2019. Please see below for workshop information, including video presentations from the workshop and position papers from participants.

About the workshop:

Replicability and reproducibility (R&R) have always been core requirements of scientific research. Recently, cases of failure to replicate previously published findings have received widespread public attention. As research grows more complex and increasingly reliant on data and software, concerns about replicability will grow rather than diminish. For example, different software packages may produce different results even when the same technique of spatial analysis is applied to the same data or analysis results cannot be reproduced by the same software due to the lack of proper metadata or provenance documenting the spatial processing and parameters used. Geospatial researchers may need to be especially concerned about replicability, such as when results from one geographic area fail to be replicated in other geographic areas.

This small and focused workshop addressed the following questions and related issues:

What forms of failure to replicate exist in the geospatial sciences? Can a formal framework be devised?
In what areas of geospatial research is the danger of non-replicability most severe?
What mechanisms can be used to avoid or minimize the danger of such failures?
Do we expect the results of model calibrations to be constant over space and if not, what are the implications for spatial analysis?
How should R&R be incorporated into the design and implementation of future spatial software?
How should students be made aware of these issues?
What follow-on activities might draw greater attention to these issues?

Read the full summary of the workshop

Participant presentations

Song Gao, University of Wisconsin Madison

 

Yingjie Hu, The University at Buffalo

 

 

Bandana Kar, Oak Ridge National Laboratory

 

Peter Kedron, Arizona State University

 

Daniel Nüst, University of Münster

 

Nathan Piekielek, Pennsylvania State University

 

Alex Sorokine, Oak Ridge National Laboratory

 

Jason Tullis, University of Arkansas

 

John Wilson, University of Southern California

 

Dawn Wright, Esri

Position papers

Note: These materials are available at http://doi.org/10.17605/OSF.IO/GVP3Q. Please use the digital object identifier when citing these materials.

Mark Gahegan - The University of Auckland, New Zealand
"An eScience perspective on Replicability and Reproducibility"

Song Gao - University of Wisconsin, Madison
"A Provenance-Based Model for Facilitating Replicability and Reproducibility in Spatial Analysis"

Yingjie Hu - University at Buffalo
"Building benchmarking frameworks for supporting replicability and reproducibility: spatial and textual analysis as an example"

Bandana Kar - Oak Ridge National Laboratory
"Multi-User and Collaborative Framework for Reproducibility and Replicability"

Daniel Nüst - Institute for Geoinformatics, University of Münster
"Reproducibility in Geospatial Research"

Vaclav Petras - North Carolina State University
"Components of an Open Science Publication"

Nathan Piekielek - Pennsylvania State University
"Position Paper on Replicability and Reproducibility"

Alex Sorokine - Oak Ridge National Laboratory
"Replicability and Reproducibility in High-Performance and Cloud Geocomputations"

Jason Tullis - University of Arkansas
"Multiuser GIS and Practical Workflow Replicability"

John Wilson - University of Southern California
"Digital Terrain Modeling"

Participants

Vanessa Bastos, Arizona State University

Ling Bian, The University at Buffalo

Kevin Butler, Esri

Dylan Connor, Arizona State University

Mark Gahegan, University of Auckland

Song Gao, University of Wisconsin Madison

Mike Goodchild, Arizona State University and UC Santa Barbara

Stewart Fotheringham, Arizona State University

Amy Frazier, Arizona State University

Yingjie Hu, The University at Buffalo

Bandana Kar, Oak Ridge National Laboratory

Peter Kedron, Arizona State University

Mike Kuby, Arizona State University

Wenwen Li, Arizona State University

Alan Murray, University of California Santa Barbara

Soe Myint, Arizona State University

Trisalyn Nelson, Arizona State University

Daniel Nüst, University of Münster

Vaclav Petras, North Carolina State University

Nathan Piekielek, Pennsylvania State University

Alex Sorokine, Oak Ridge National Laboratory

Daniel Sui, University of Arkansas

Daoqin Tong, Arizona State University

Matthew Toro, Arizona State University

Jason Tullis, University of Arkansas

Shaowen Wang, University of Illinois at Urbana-Champaign

Elizabeth Wentz, Arizona State University

John Wilson, University of Southern California

Dawn Wright, Esri

 

Organizing committee

The R&R Workshop is possible thanks to the hard work of the organizing committee, including: Mike Goodchild, Arizona State University and UC Santa Barbara; Stewart Fotheringham, Arizona State University; Wenwen Li, Arizona State University; and Peter Kedron, Arizona State University. Logistical planning for the workshop was made possible thanks to the work of Nick Ray, assistant director of operations to the School of Geographical Sciences and Urban Planning and executive assistant to the Spatial Analysis Research Center.