Blog posts

This week, the o2r team was on tour.
We put our o2r tasks aside for a few days to interact with and contribute to the awesome Open Science/publishing/research community.

Markus and Daniel were two of the fortunate few who were invited to participate in the eLife Innovation Sprint 2019.
Thanks eLife!
eLife is a non-profit Open Access publisher with a mission to innovate and push scholarly communication, peer review, and publication of reproducible articles to new heights.
The #eLifeSprint is a two-day event and brings together scientists, developers, designers, architects, thinkers, community leaders, publishers, and early career researchers to come up with relevant challenges and promising ideas for the scientific community.
It took place for the second time in Cambridge, UK, where eLife’s headquarter is located, in the welcoming Cambridge Junction.
Just like last year, the event was excellently organised and run by eLife staff.

After introductions and pitching ideas, the participants formed into project groups and spent ~1.5 days on realising a first prototype.
You can learn about the results in the “time to shine” presentation and on social media under #eLifeSprint #timetoshine: an Open Science card game, a user interface for generating citation files for software, extracting data from text such as the used instruments, a prototype for discovering preprints from authors with underrepresented backgrounds, or a template project for running a journal on GitHub, to name just a few.
Daniel and Markus really enjoyed the event and contributed with their developer skills (containers, UI development, eating cake) to several projects.

#TimeToShine: Ankit, Stephen, Daniel have made a UI prototype on GitHub and Docker Hub, worked on UI development for Binder, written case studies and more principles for CODECHECK, as well as helping others with Docker projects at the #eLifeSprintpic.twitter.com/YJWrsvqWkN

While it was a little disappointing that Markus’ idea of a JavaScript image comparison library (hopefully more on that soon!) did gain attention but did not end up in a team, the sprint was a great occasion to give back to the community, to broaden the horizon beyond the o2r project, to make new acquaintances, and to get to know potential collaborators.
And we did all that!

After the #eLifeSprint, Daniel hopped on a plane to Oslo, Norway, to participate in a Binder/BinderHub/MyBinder.org/JupyterHub event generously organised by Simula.
The event allowed long-term collaborators to meet in person, some for the first time, for some effective joint work.
Participants happily hacked away on their own or formed discussion groups on specific topics for a few hours before taking on a new challenge.
Ten to twelve developers of diverse backgrounds filled a hotel meeting room and turned coffee and delicious catering into pull requests, issues, and hackpads with new ideas and solutions in the Binder/Jupyter universe.
It was a great experience to get to know the friendly faces and delightful personalities behind GitHub usernames.
Daniel enjoyed participating in the discussions and picking the brains of the core developers of BinderHub and repo2docker, and the maintainers of mybinder.org.
He was able to contribute a fewpullrequest to repo2docker and enjoyed the discussions on future directions of the core tool in the Binderverse, such as a new user interface (a must to make BinderHub even more like magic), pinning the repo2docker version (a must for reproducibility) and re-enabling composability for all supported configurations (a must for many users).

Thanks to all participants for making the meeting so much fun and educational.
Daniel’s participation will surely help to pave the way for a Binder-powered scalable infrastructure for the o2r pilots and for CODE CHECK.
You can learn more about the numerous tasks tackled in the sprint in this HackMD pad: https://hackmd.io/N-uffNhvRdOgt1OvTuoq5w?view

In 2016, Dottori et al. published a paper about a flood damage model. The model calculates the damage costs caused by a flood event, e.g., for repairing buildings or cleaning. This model is based on a number of parameters, such as flow velocity and flood duration. In the paper, the authors discuss a scenario in which a flood has a velocity of 2m/s and a duration of 24 hours. The resulting damage costs are shown in a figure and also alternative values are discussed in the text. This is where the paper format, i.e. a PDF file, is limited. A mere format change does not help - a static HTML rendering has the same issues. Describing within the article text how changes to the parameter set affect the damage costs might be possible possible but is surely a daunting and time-consuming task. Authors need to find the right words to briefly describe these changes, and readers need to imagine how the results change.

A binding describes which source code lines and data subsets were used to produce an individual computational result, such as a figure, table, or number in the text. A binding explicitly refers to single parameters in the code which influence the result. By specifying a user interface widget (e.g. a slider) for a parameter, a binding can then be used to create an interactive figure.

Ok, cool, that sounds just awesome, but how does it look like? Let’s check both perspectives, the author who creates a binding, and the reader who uses the interactive figure. Just four steps are needed to create a binding for an interactive figure:

Specify the result, e.g. “Figure 3”

Mark the plot function in the code that creates the figure. From that plot function we automatically extract all relevant code lines, at least we plan to do so since this feature is currently under development.

Mark the parameter that should be made interactive, e.g. “duration” or “velocity”

Configure the user interface widget, e.g. a slider
That’s how authors can create interactive figures easily. Please note that we did not yet fully implement the functionality for specifying data subsets.

The readers’ view was integrated into the implementation discussed in an article published last year (“In-depth examination of spatio-temporal figures”) and the previous blog post.
The left side shows the static version of the paper. On the right side, readers can use the slider to change the two parameters velocity and duration. The changes are immediately reflected in the figure. Since it might be difficult to spot differences, we also implemented a simple view to compare two figures created through parameter manipulation by the reader.

Such explorable papers are the next generation of scholarly communications. Being able to provide interactive figures is beneficial for authors, who can explain visually how changes to the parameters affect the figure, and for readers, who better understand complex models. They are also a sign of quality for the analysis workflow, because they demonstrate that all pieces (data, software) needed to create the figure are encapsulated in the Executable Research Compendium (ERC) on which the bindings are based.

By the way, we also presented the paper at the Engineering Interactive Computing Systems Conference 2019 in Valencia. Of course, the slides are available online on Zenodo.

Discovery

By having all materials available in a publicly accessible way, we obtain additional capabilities to search for scientific papers which go beyond today’s keyword-based search engines. The materials underlying a paper include a bunch of information which can be extracted automatically (see o2r-meta) and put on display (see geospatial data science badges) to improve discovery. You were wondering how to use a specific software library in your R code in practice? Just search for papers with computations based on that library. Spatial information, temporal properties, models, parameters - this all becomes searchable which is good for readers, and findable which is good for the impact of authors.

Inspection

Once researchers found a suitable paper, they can continue with inspecting it. Parallel to reading the actual text of the paper, they can inspect the underlying source code and data. This is of particular interest for reviewers who want to check how the authors achieved the results reported in the paper. By the way, more and more reviewers reject papers reporting on computational results that do not contain code or data - Think about it! Again, this step is not only beneficial for readers and reviewers but also for the authors who can make their research workflows more reusable resulting in a higher research impact.

Manipulation

Many results in scientific papers are based on computational analyses. These calculations often include parameters which were set in a specific way by the author of the article. For example, a model that computes the damage costs caused by a flood strongly depends on the flow velocity (see Dottori et al., 2016) of the water. It is difficult to show in static papers, how changes to the flow velocity affect the final damage costs. One idea to solve this issue is an interactive figure. Readers and Reviewers can, for example, use a slider to change the parameter value interactively.

Substitution

Finally, other researchers can substitute, for instance, the original dataset by an own compatible dataset. This opportunity not only makes other researchers’ life easier as they can reuse existing materials, but might also bring citations, co-authorships, and cooperations for the original author.

+1

So who is this workflow steps’ +1?

It’s understanding.

In the paper, we argue that each of the steps contribute to a reader’s understanding in a better way than traditional papers could do. Already during the inspection phase, researchers get to know about spatio-temporal properties, used functions and so on. During inspection, they can see how the authors produced a specific figure, experience the data from the analysts perspective, and finally understand how the authors came to their conclusions. By manipulating parameters, readers and reviewers can comprehend better how the model actually works. Substituting datasets provides insights into the applicability to other settings and evaluates robustness of an approach. A key requirement for the realization of understanding is being able to compare, for example, the original figure with one resulting from parameter manipulation.

You think that this was nice to read but difficult to realize? Correct, it is. And that is why the o2r team works hard to make the five incentives easier to achieve and received funding for two more years.

On the first day, Daniel contributed to the organisation of the third workshop in the “Reproducible Research @ AGILE” series of workshops.
Adjusting the scope of the workshop after the firsttwo iterations, the participants learned first about the basics of reproducibility before being split up into a “beginners” and “advanced” group.
The former continued with practical experiences in reproducing a tailored small manuscript with data and code, while the latter took on real world papers in a reproduction sprint.
Starting only with a DOI, the participants skimmed real articles for practical instructions and shared how far they got after only 30 minutes.
The results were mixed, as it could be expected, but the lessons that could be drawn were already very educational and could be connected directly with concrete steps towards preproducibility.

After lunch, the groups joined again for getting to know the AGILE Reproducible Paper Guidelines.
The guidelines were developed in online collaboration and a recent expert meeting at TU Delft (see report).
They require authors to be transparent about the underlying building blocks of their work by adding a Data and Software Availability section.
Beyond this minimal requirement of transparency, the guidelines intent to nudge authors towards higher degrees of reproducibility with concrete steps and recommendations for both data and software.
The steps are illustrated by examples from the GI Science domain.
Leave your feedback about the guidelines OSF!
The ensuing discussion about the challenges, opportunities, and ethics of reproducible research made clear the participants were serious on their way to becoming experts in RR.
They continued on this path in the final session, in which both groups took on the role of an author and applied practices of Open Science and reproducible research.
Find all workshop material online at https://osf.io/d9kcr/.

Besides the workshop, the RR@AGILE team advertised and sought feedback on the guidelines in many small discussions and with a dedicated poster.
The feedback will be incorporated into a first release of the guidelines in the coming weeks, just in time for the call for papers for the next AGILE conference in Chania, Crete!
The RR@AGILE team is proud that the AGILE council and next year’s organising team support a transformation towards reproducible research publications and looks forward to working with authors, reviewers and organisers to making the move a success.

On the second day of the conference, Daniel presented the short paper “Guerrilla Badges for Reproducible Geospatial Data Science”.
The paper is based on the work of a project seminar at the Institute for Geoinformatics from 2017, which explains the long list of co-authors.
The article demonstrates how and what kind of novel badges can be created based on executable research compendia (ERC) and how they can be distributed on the web.
The full postprint of the peer-reviewed article is available on EarthArXiv and it contains links to the related software projects.

As always, AGILE was a delightful conference with many engaging discussions which may have started more collaborations to foster reproducible research.
Daniel also continued the text analysis of all AGILE papers for this year’s conference.

Obviously this needs advertising, as no one spotted the error in the "trends" analysis: Increase in "reproducibility" keywords can be largely awarded to mine and @pjkedron's papers. Still lots of data, algorithms, and processing going on - clear need for reproducibility! pic.twitter.com/W4FMhtm3as

A small rise in “reproducibility” terms can be traced to a couple of articles on the topic.
Yet the stronger trend prevails: AGILE papers talk about data, processing, and algorithms - so the transformation for more transparency and reproducibility continues to be relevant.

Now that we have two more years to work on open reproducible research (see our last blog post), there is also some space for an exchange with related projects and to explore potential new collaborations. We were thus very happy to receive an invitation from the Conquaire project at the University of Bielefeld for the workshop on data quality and reproducibility (03.04.2019). Conquaire started about the same time as o2r and strives for similar goals, i.e. assisting scholars in making their research results reproducible and reusable. The workshop was located at the Center for Interdisciplinary Research in a very nice room that looked a bit like the United Nations headquarter - so it was good practice for the bigger goals we have in mind.

Prof. Dr. Philipp Cimiano gave the first talk of the day. He presented the general Conquaire approach which focuses on storing all materials in a GitLab repository and running checks with the help of continuous integration based on Jenkins. Researchers can thus create an incremental publication where each git commit triggers and automatic validation process. They also had a promising number of use cases. However, similar to us, they struggled a bit with the amount of effort needed from authors to make research reproducible.

Christian Pietsch then gave a quick introduction into versioning tools such as GitLab and which benefits users get. I particularly liked his answer to the question from the audience if the Conquaire approach is also feasible with licensed software: Use free open source software! It’s that easy.

Afterwards, Conquaire team member Fabian Herrman talked about their validation approach by using continuous integration (Slides). They check, for example, if all files are available (including readme and license) and convey the result in two ways: First, by assigning a badge to the repository and second, by emailing the author of the repository.

The following talks were about Findable, Accessible, Interoperable, and Reusable data principles (by Silvia Wissel and Amrapali Zaveri) and the Jupyter Notebook, which was used in the context of history science by Malte Vogl. One benefit of Jupyter notebooks he mentioned stuck with us: it is also readable when the base software does not exist anymore. This is also one of the essential advantages of the Dockerfiles and R Markdown documents used in our executable research compendia (ERCs).

Last but not least, we were allowed to present our approach and what we plan to achieve in the next two years. The slides are available online: https://zenodo.org/record/2628278.

On April 1st 2019 the o2r team started into a new phase (“o2r2”).
In the next 30 months we plan to put our prototypes to the test with real articles, of course not without considerably improving them beforehand.

As detailed in the University of Münster’s press releases (English, German), we arefortunate to collaborate with publishers to achieve the following objectives:

Use ERCs for actual scientific publications in pilot studies with original research manuscripts in a scholarly peer review

Eliminate barriers for using ERCs as part of a publishing process

Evaluate concept and pilots with user studies and monitoring to understand the costs and benefits of ERC-based authoring, publishing, and reading

As the official press statement was edited for brevity, we’d like to use the opportunity to extend on it here:

The project “Opening Reproducible Research II” (o2r2) will be supported by the German Research Foundation (DFG) under the umbrella of the programme for Library Services and Information Systems (LIS).
Building on the results of the predecessor project, it will test, evaluate, and further develop solutions for improving reproducibility of research results in practice over the next 30 months.
The o2r project will conduct three pilot studies.
The Open Access publisher Copernicus Publications and a large commercial publisher could be won for the project to conduct two pilot studies in the form of special issues for scientific journals, in which classic articles will be enriched with interactive and transparent analyses.
In a third pilot, an open source software for the publication of scientific journals, Open Journal Systems by the Public Knowledge Project (OJS by PKP), which is widely used in the scientific community, will be connected to the o2r reproducibility service and piloted at the ULB Münster together with researchers and students from the geosciences.
Developments for OJS will be contributed to the global and national communities, including OJS-de.net.
These pilots will be accompanied by various evaluations: with the help of authors, reviewers, and students, the transformation potential of reproducible scientific articles will be investigated.
A focus lies on analyses based on geospatial data and the programming language R.
Futhermore, the operation provides relevant insights into the costs and efforts for contemporary publishing of and interaction with data-based scientific research.
All specifications and tools of o2r2 are published under free licenses and where possible are realised as contributions to existing Open Source projects.
The project’s developments and results provide building blocks and concepts for a future infrastructure for enhanced scholarly communication and academic publications.

We realise these goals are ambitious, but look forward confidently to work with the Open Science and Open Source communities to make them reality.

As a first action, o2r team member Daniel attended the EGU General Assembly in Vienna last week to start the conversation about the pilot with EGU journals with journal editors.
With the support of Copernicus staff, we distributed flyers about the planned Virtual Special Issue to journal editors of the Copernicus journals.
You can read more about the plans on the Pilots page.
Since we are very early in the project, it comes only as a little setback that Daniel could only speak to a handful of editors.
We plan to intensify our outreach to journals and editors later this year, when the first working prototypes can tell the story much more convincing than a paper leaflet can - isn’t that what executable interactive publications are all about?

The o2r project website’s first entry Introducing o2r was published 1132 days ago.
Since then we’ve published short and long reports about events the o2r team participated in, advertised new scholarly publications we were lucky to have accepted in journals, and reported on results of workshops organised by o2r.
But there has also been some original content from time to time, such as the extensive articles on Docker and R, which received several updates over the last years (some still pending), on the integration of Stencila and Binder, or on writing reproducible articles for Copernicus Publications.
These posts are a valuable output of the project, and contribute to the scholarly discussion.
Therefore, when it came to writing a report on the project’s activities and outputs, it was time to consider the preservation of the project website and blog.
The website is built with Jekyll (with Markdown source files) and hosted with GitHub pages, but GitHub may disappear and Jekyll might stop working at some point.

So how can we archive the blog post and website in a sustainable way, without any manual interference?

Today’s blog post documents the steps to automatically deposit the sources, the HTML rendering, a PDF rendering, and the whole git repository in a new version of a Zenodo deposit with each new blog post using Zenodo’s DOI versioning.
The PDF was especially tricky but is very important, because the format is established for archival of content, while using the Zenodo API was pretty straightforward.
We hope the presented workflow might be useful for other websites and blogs in a scientific context.
It goes like this:

The Makefile target update_zenodo_deposit starts the whole process with make update_zenodo_deposit. It triggers several other make targets, some of which require two processes to run at the same time:

Remove previously existing outputs (“clean”).

Build the whole page with Jekyll and serve it using a local web server.

Create a PDF from the whole website from the local web server using wkhtmltopdf and the special page /all_content, which renders all blog entries in a suitable layout together with an automatically compiled list of author names and all website pages, unless they are excluded from the menu (e.g. manual redirection/shortened URLs) or excluded from “all pages” (e.g. the 404 page, blogroll, or publications list).

Create a ZIP archive with the sources, HTML rendering and PDF capture.

Run a Python script to upload the PDF and ZIP files to Zenodo using the Zenodo API, which includes several requests to retrieve the latest version metadata, check that there really is a new blog post, create a new deposit, remove the existing files in the deposit, upload the new files, and eventually publish the record.

Kill the still running web server.

For these steps to run automatically, the Travis CI configuration file, .travis.yml, installs the required software environment () to conduct all above steps during each build.
It also includes a encrypted environment variable with a Zenodo API key, so the build system provided by Travis CI can manipulate the record, the first version of which was created manually and included the record description, authors, tags, etc.

So what is possible now?
As pointed out in the citation note at the bottom of pages and posts, the Digital Object Identifier (DOI) allows referencing the whole website or specific posts (via pages in the PDF) in scholarly publications.
Manual archival from a local computer is still possible by triggering the same make target.
As long as Zenodo exists, readers have access to all content published by the o2r research project on its website.

There are no specific next steps planned, but there’s surely room for improvement as the current workflow is pretty complex.
The post publication date is the trigger for a new version, so changes in a page such as About or in an existing post requires a manual triggering of the workflow (and commenting out the check for a new post) or wait for the next blog entry.
The created PDF could be made compliant with PDF/A.
The control flow could also be implemented completely in Python instead of using multiple files and languages; a Python module might even properly manage the system dependencies.
Though large parts of the process are not limited to pages generated with Jekyll (the capturing and uploading), it might be effectively wrapped in a standalone Jekyll Jekyll plugin, or a combination of a Zenodo plugin together with the (stale?) jekyll-pdf plugin?
Your feedback is very welcome!

The capabilities of containerisation and the concept of the Executable Research Compendium form the basis for o2r’s reproducibility service.
But the demonstration how latest technology may support a more open and transparent scholarly publication alone is only one half the battle.
Breakthroughs in tools and infrastructure must be accompanied by outreach activities to highlight the need for opening reproducible research to all stakeholders (scientists, editors, publishers, funding agencies).
That is why I was extremely glad to join some of the most renowned researchers of geography and GI Science at the “Replicability and Reproducibility Workshop” in Tempe, Arizona, on February 11 and 12, 2019.
The event was organised by the Spatial Analysis Research Center (SPARC) at Arizona State University (ASU).

Daniel Nust from the University of Munster presenting at our Replicability and Reproducibility in Geospatial Research: Open is not enough for reproducibility! pic.twitter.com/PwpVmwdQ7b

The events featured four full talks.
I was invited to go first and report on the activities of o2r as well as the Reproducible AGILE conference series and initiative for developing new submission and reviewing guidelines.
The expectations were high, but after three years of intense work by the o2r team, the allotted time was easily filled.
After introducing challenges which disrupt scholarly publication practices, I reported on observations made by our own surveys and others on reproducibility, including the two reproduction campaigns let be the o2r team.
The painted picture unsurprisingly left a lot of room for improvement, for which the talk provided technical as well as organisational approaches.

In the second talk, ASU’s own Peter Kedron took a step back and surveyed the existing definitions of replicability and reproducibility.
Peter then excellently connected and extended these terms with the intricacies and specifics of geospatial sciences.
The technical and theoretical groundwork was layed, so the discussion following the first talks set the bar for the remainder of the workshop quite high.
Many critical and thoughtful comments were made and viewpoints shared.
One of the main take-home messages for me was that while geography/GI Science/related disciplines may take advantage of the hard lessons learned in other domains (which faced a ‘replication crisis’), the uniqueness of geography as a science that always had to deal with uncertainty and context may also contribute a unique perspective on replicability and reproducibility.
It was great to see that the topic of reproducibility is widely acknowledged as a relevant challenge and the interest to initiate improvement was unilateral.

The first day continued with lightning talks.
As could be expected, the diverse backgrounds (eScience, ecology, political geography, …) let to a very useful diversity in topics and perspectives.
Afterwards the participants split up into three groups to tackle technical, organisational, and institutional aspects of replicability and reproducibility, which gave input for yet another thoughtful debate in the assembly to conclude day one.
The discussions continued between old colleagues and new friends during a delightful evening reception and dinner.

Day two kicked of in a similar fashion with talks by Daniel Sui, University of Arkansas, and Esri’s Dawn Wright and Kevin Butler.
Again two very different takes on the topic, with valuable new ideas.
The following discussion was lively and included potential venues for the newly formed group to continue the collaboration, most importantly to increase the awareness of the topic across all communities working with spatial data.
The whole meeting was nicely guided and framed by contributions from ASU’s Mike Goodchild and Stewart Fotheringham.
All participants were united in their interest to advance transparency and openness and a realisation that there is a need for action from many different angles, including education and evaluation, if a ‘crisis’ shall be avoided.
Despite some concerns how the topic might be received by critics, the meeting ended in a positive mood of newfound mutual support and of acknowledging the value of the work ahead.

My personal opinion is that the disruptions in science are more pressing than a traditional scholarly approach (organising a special issue for 2020, writing an editorial) can answer.
Yet the old-school way may be able to bridge across the divide and different skill-set/mindset/needs between computational/junior/young/technical and theoretical/senior researchers, and is as such worth pursuing.
For future discussions, I plan to frame reproducibility as an ideal that is worth striving for and worth to reward (e.g. in evaluations, during reviews, using badges, in funding schemes), but to be careful with too simple checklists and dos/don’ts, because there will always be corner cases and limitations for specific circumstances.
This is a core difference between reproducibility and openness - you can not be a little or partially open, but being almost reproducible is still an important achievement.
Reproducibility and replicability will not be helped by whataboutism nor by pointing fingers, but the positive spirit of preproducibility.

Luckily there is no need to echo all insights by talks and during the discussions: the sessions were recorded on video they will be published together with slides and position papers in an OSF project soon.
This post will be updated then. Followus on Twitter to not miss it.
Until then you can take a look at my position paper on GitLab, even the speaker notes in the presentation source file if you dare.

This post would be incomplete without a big Thank You to the sponsoring and excellent organisation provided by Esri and the hosting School of Geographical Sciences and Urban Planning (SGSUP).
I am confident this workshop may spark new collaborations and be able to put replicability and reproducibility on the map for more researchers in geography and related disciplines.

Contemporary science faces many challenges in publishing results that are reproducible.
This is due to increased usage of data and digital technologies as well as heightened demands for scholarly communication.
These challenges have led to widespread calls for more research transparency, accessibility, and reproducibility from the science community.
This article presents current findings and solutions to these problems, including recent new software that makes writing submission-ready manuscripts for journals of Copernicus Publications a lot easier.
While it can be debated if science really faces a reproducibilitycrisis, the challenges of computer-based research have sparked numerous articles on new goodresearchpractices and their evaluation.
The challenges have also driven researchers to develop infrastructure and tools to help scientists effectively write articles, publish data, share code for computations, and communicate their findings in a reproducible way, for example Jupyter, ReproZip and research compendia.

Recentstudiesshowed that the geosciences and geographic information science are not beyond issues with reproducibility, just like other domains.
Therefore, more and more journals have adopted policies on sharing data and code.
However, it is equally important that scientists foster an open research culture and teach researchers how they adopt more transparent and reproducible workflows, for example at skill-building workshops at conferences offered by fellow researchers, such as the EGU short courses, community-led non-profit organisations such as the Carpentries, open courses for students, small discussion groups at research labs, or individual efforts of self-learning.
In the light of prevailing issues of a common definition of reproducibility, Philip Stark, a statistics professor and associate dean of mathematical and physical sciences at the University of California, Berkeley, recently coined the term preproducibility: “An experiment or analysis is preproducible if it has been described in adequate detail for others to undertake it.”
The neologism intends to reduce confusion and also to embrace a positive attitude for more openness, honesty, and helpfulness in scholarly communication processes.

In the spirit of these activities, this article describes a modern workflow made possible by recent software releases.
The new features allow the EGU community to write preproducible manuscripts for submission to the large variety of academic journals published by Copernicus Publications.
The new workflow might require hard-earned adjustments for some researchers, but it pays off because of an increase in transparency and effectivity.
This is especially the case for early career scientists.
An open and reproducible workflow enables researchers to build on others’ and own previous work and better collaborate on solving the societal challenges of today.

Reproducible research manuscripts

Open digital notebooks, which interweave data and code and can be exported to different output formats such as PDF, are powerful means to improve transparency and preproducibility of research.
Jupyter Notebook, Stencila and R Markdown let researchers combine long-form text of a publication and source code for analysis and visualisation in a single document.
Having text and code side-by-side makes them easier to grasp and ensures consistency, because each rendering of the document executes the whole workflow using the original data.
Caching for long-lasting computations is possible, and researchers working with supercomputing infrastructures or huge datasets may limit the executed code to purposes of visualisation using processed data as input.
Authors can transparently expose specific code snippets to readers but also publish the complete source code of the document openly for collaboration and review.

R Markdown supports different programming languages besides the popular namesake R and is a sensible solution even if you do not analyse data with scripts nor have any code in your scholarly manuscript.
It is easy to write, allows you to manage your bibliography effectively, can be used for websites, books or blogs, but most importantly it does not fall short when it is time to submit a manuscript article to a journal.

The following code snippet and screen shot demonstrate the workflow.
Lines starting with # are code comments and explain the steps.
Code examples provided here are ready to use and only lack the installation commands for required packages.

# load required R extension packages:library("rticles")library("rmarkdown")# create a new document using a template:rmarkdown::draft(file="MyArticle.Rmd",template="copernicus_article",package="rticles",edit=FALSE)# render the source of the document to the default output format:rmarkdown::render(input="MyArticle/MyArticle.Rmd")

The commands created a directory with the Copernicus Publications template’s files, including an R Markdown (.Rmd) file ready to be edited by you (left-hand side of the screenshot), a LaTeX (.tex) file for submission to the publisher, and a .pdf file for inspecting the final results and sharing with your colleagues (right-hand side of the screenshot).
You can see how simple it is to format text, insert citations, chemical formulas or equations, and add figures, and how they are rendered into a high-quality output file.

All of these steps may also be completed with user-friendly forms when using RStudio, a popular development and authoring environment available for all operating systems.
The left-hand side of the following screenshot shows the form for creating a new document based on a template, and the right-hand shows side the menu for rendering, called “knitting” with R Markdown because code and text are combined into one document like threads in a garment.

And in case you decide last minute to submit to a different journal, rticles supports many publishers so you only have to adjust the template while the whole content stays the same.

Sustainable access to supplemental data

Data published today should be published and properly cited using appropriate research data repositories following the FAIR dataprinciples.
Journals require authors to follow these principles, see for example the Copernicus Publications data policy or a recent announcement by Nature.
Other publishers required, or still do today, to store supplemental information (SI), such as dataset files, extra figures, or extensive descriptions of experimental procedures, as part of the article.
Usually only the article itself receives a digital object identifier (DOI) for long-term identification and availability.
The DOI minted by the publisher is not suitable for direct access to supplemental files, because it points to a landingpage about the identified object.
This landing page is designed to be read by humans but not by computers.

Main takeaways

Authoring submission-ready manuscripts for journals of Copernicus Publications just got a lot easier.
Everybody who can write manuscripts with a word processor can learn quickly R Markdown and benefit from a preproducible data science workflow.
Digital notebooks not only improve day-to-day research habits, but the same workflow is suitable for authoring high-quality scholarly manuscripts and graphics.
The interaction with the publisher is smooth thanks to the LaTeX submission format, but you never have to write any LaTeX.
The workflow is based on an established Free and Open Source software stack and embraces the idea of preproducibility and the principles of Open Science.
The software is maintained by an active, growing, and welcoming community of researchers and developers with a strong connection to thegeospatialsciences.
Because of the complete and consistent notebook, you, a colleague, or a student can easily pick up the work at a later time.
The road to effective and transparent research begins with a first step - take it!

Acknowledgements

The software updates were contributed by Daniel Nüst from the project Opening Reproducible Research (o2r) at the Institute for Geoinformatics, University of Münster, Germany, but would not be able without the support of Copernicus Publications, the software maintainers most notably Yihui Xie and Will Pearse, and the general awesomeness of the R, R-spatial, Open Science, and Reproducible Research communities.
The blog text was greatly improved with feedback by EGU’s OliviaTrani and Copernicus Publications’ Xeniavan Edig.
Thank you!

A few weeks ago, a new journal article written by o2r team member Markus got published.
In our last article, we talked about the reproducibility of papers submitted to the AGILE conference.
We checked if the papers had materials attached and if these materials were complete.
The results were rather unfortunate.
In our newest article, we took one further step and tried to re-run the analyses of articles which had code and data in the supplements.

Recreating scientific data analysis is hard, but important.
To learn more about the state of reproducibility in geosciences, we conducted several studies.
We contacted over 150 geoscientists who publish and read articles based on code and data.
We learned that as readers they often would like to have access to these materials, but as authors they often do not have the time or expertise to make them available.
We also collected articles which use computational analyses and tried to execute the attached code.
This was not as easy as it sounds! We describe these numerous issues in a structured way and our experiences in this publication.
Some issues were pretty easy to solve, such as installing a missing library.
Others were more demanding and required deep knowledge of the code which is, as you might imagine, highly time consuming.
Further issues were missing materials (code snippets, data subsets) and flawed functionalities.
In some cases, we contacted the original authors who were, and this was a positive outcome, mostly willing to help.
We also compared the figures we got out of the code with those contained in the original article.
Bad news: We found several differences related to the design of the figures and results that deviated from those described in the paper.
OK, this is interesting, but why is it important?
We argue, a key advantage of open reproducible research is that you can reuse existing materials.
Apparently, this is usually not possible without some significant effort. Our goal is not to blame authors.
We are very happy that they shared their materials.
But they did that with a specific purpose in mind, i.e. making code and data available and reusable for others to build upon that.
One incentive in this context is an increased number of citations, one of the main currencies for researchers.
To facilitate that, we suggest some guidelines to avoid the issues we encountered during our reproducibility study, such as using Executable Research Compendia (ever heard of them? :)).

This article reports on a project, integrating Stencila and Binder, which started at the eLife Innovation Sprint 2018. It has been cross-posted on multiple blogs (eLife Labs, Stencila, Jupyter). We welcome comments and feedback on any of them!

eLife, an open science journal published by the non-profit organisation eLife Sciences Publications from the UK, hosted the first eLife Innovation Sprint 2018 as part of their Innovation Initiative in Cambridge, UK:
“[..] a two-day gathering of 62 researchers, designers, developers, technologists, science communicators and more, with the goal of developing prototypes of innovations that bring cutting-edge technology to open research communication.”
One of the 13 projects at the excellently organised event was an integration of Binder and Stencila.

This article reports on the project’s inception, building blocks, achievements at the sprint weekend, and work conducted in the months following the sprint.
Today, Binder has first class Stencila support.
You can open Stencila documents from any online code repository on mybinder.org with the click of a single button. Just try out the example below:

The idea and the sprint team

The eLife Innovation Sprint started with brief introductions by all participants.
Some of them prepared pitches for projects ideas, which quickly got little group discussions going.
One table at the sprint attracted a few people with an interest in containerisation technology for research applications.
Many ideas were floated and a helpful exchange around existing solutions and tools took place.
When it was time to find a concrete task, two of the sprinters identified a worthwhile technological problem as their challenge for the next 1.5 days and the project “Jupyter+DAR compatibility exploration” started.
Min from the Simula Research Laboratory, Norway, is a core developer of Binder and related tools.
He was interested to get to know the Stencila project and explore the possibilities of having alternative user interfaces on Jupyter Hub.
Daniel from the o2r project at the Institute for Geoinformatics, Germany, works on reproducible computations in the geosciences and had a keen interest in learning more about the Binder platform.
They were joined remotely by Nokome, the initiator and one of the developers of Stencila.

The building blocks

Stencila Desktop is an office suite for reproducible research documents.
It allows scientists to use languages like R and Python within familiar and intuitive word processor and spreadsheet user interfaces.
By doing so, it aims to lower the barriers to reproducible research for those with little or no software development skills.
At the same time, Stencila aims to make it easy for researchers versed in software development to collaborate with their colleagues without having to switch from R or Python.
Stencila Desktop is built upon Texture, an editor for scientific content, which uses the Dar file format. Dar is an extension of the JATS publishing format which has been designed for reproducible research publications. It aims to serve researchers using computational methods for data, and publishers using digital workflows for publication and preservation of scholarly journals.

Binder makes it simple to generate reproducible computing environments from code repositories.
The online service mybinder.org is the most prominent example for a platform based on the Binder project, a part of Project Jupyter.
A user can run a Jupyter Notebook and other environments for their research projects, which are published in online repositories (e.g. GitHub or GitLab, see binder examples).
In the spirit of the Unix philosophy, Binder combines several Open Source tools to achieve this goal: repo2docker, for generating Dockerfiles and building Docker images from software projects, JupyterHub for executing a Docker image and user-facing web portal in a cloud environment, and BinderHub for gluing the above together.

A Dockerfile is a human- and machine-readable recipe for setting up a computational environment, which is just fancy words for saying “installing and configuring software”.
Dockerfiles are used by the popular Docker container software.
They can be built into an executable image, which is portable between host computers.
These properties make containers very interesting for capturing and sharing research involving data and software.

While containers have become a commodity for developers, researchers still struggle to grasp and control the complexity of computational environments.
This is where the two building blocks join:
Running Stencila as part of a Binder helps researchers to communicate their work openly, to collaborate effectively with other scientists, and to ensure a high quality and transparency of their workflow and findings.

The challenge

Their goal was “[..] to connect them so that users can edit reproducible documents (DAR files) as part of a Binder project” with the following objectives: (i) understanding DAR [Dar Format], (ii) launching Stencila Editor on Binder (potentially not launching anything else, i.e. w/o the Jupyter Notebook start page), and (iii) repo2docker support for DAR files.
The project was also part of the Mozilla Global Sprint 2018, see mozilla/global-sprint#317.

The solution

It took more than just the 1.5 days in Cambridge to really fulfil this challenge.
First we describe the crucial breakthroughs that were actually made at the sprint, then the updates that happened until today.

Sprint breakthrough

Min and Daniel started by taking a close look at an existing solution, namely the integration of RStudio based on nbrsessionproxy, i.e. the “Notebook R session proxy”.
They learned two things:

a Jupyter notebook extension can be used to add a menu item to the Jupyter UI

a component is needed to route the traffic between the browser-based user interface and the server-side software

The first attempts utilised Binder’s feature of manually defining a bespoke Dockerfile (see a first attempt) and later also a postBuild script to install and configure all software.
It was Daniel’s first task to transfer the first finding for Stencila.
After setting up a local development environment and learning Jupyter/Binder, it just needed small adjustments to selected files from nbrsessionproxy to achieve this (see commit from the second day):

Min took on the second task while at the same time figuring out what parts of Stencila we really needed, and how to glue them together.
He wrote a hard-wired proxy using Python and added some JavaScript/HTML files to serve Dar files and the Stencila UI itself.

Connecting Stencila to Jupyter kernels

Stencila has “execution contexts” (the equivalent of Jupyter’s “kernels”) for R, Python, SQL, Javascript (in the browser), and Node.js. Execution contexts differ from kernels in a number of ways including code dependency analysis and returning execution results as data values. Both of these are necessary for the reactive, functional execution model of Stencila.

We could install these execution contexts in the Docker image.
However, Stencila also has a JupyterContext which acts as a bridge between Stencila’s API and Jupyter kernels.
So, since the base jupyter/minimal-notebook image already has a Jupyter kernel for Python installed, we decided to use that.
This did mean however, that some of the reactive aspects of the Stencila UI won’t work as expected.

We included the stencila-node Node.js package in the Docker image which provides the JupyterContext as well as a NodeContext (for executing Javascript) and a SqliteContext (for executing SQL) .

We first used Stencila’s development build to run the JavaScript app using node make -w -s -d /our/own/dir, but struggled a bit to configure the file storage, i.e. the dar-server, to use the directory we want to, and to run it in a full path configured by us instead of make.js starting the dar-server relative to __dirname.
Eventually we ended up implementing our own minimal JavaScript module (i.e. an npm package) that run (i) the dar-server and (ii) a static file server for the app using the distribution files (i.e. the dist directory).
This gave us control of the paths and let us get rid of complex development features (e.g. substance-bundler).

We also made our own version of app.js, removing the virtual file storage (vfs, used to seamlessly integrate examples) and instead defaulting to a file system (fs) storage, because that is what is needed for Jupyter.
In the same line, we built or own index.html (based on example.html) to serve as the entry page.
This allowed us to directly render a single Dar document instead of a listing of examples and to use our own app.js.
Relevant path configurations comprised the local storage path as well as the URLs used by the client, accessing the dar-server through the nbserverproxy.

At the end of the first day, the wiring was all there so we could open a repository and the Stencila document was shown!
But the interactive execution of code cells did not work yet :-/.

Thanks to an international time-zone-difference-powered “overnight” contribution, Min and Daniel got a big surprise on Friday morning:
Nokome added the Stencila Node.js host for Jupyter execution context support, so that Python cells could be executed by connecting to the Jupyter Kernel (which of course was already there in the container).
In doing so, he returned the “surprise” he had when learning about the project.
The added “host” provides the single gateway for code cell contents to be forwarded to the respective execution contexts.
Nokome showed everything works with the obligatory screenshot:

Since you can run any commit in a Binder, you can also try out that particular state from the repository yourself:

The second day of the sprint involved many iterations of improvements, including changes to repo2docker.
These updates could not simply be thrown upon mybinder.org, so Min set up a test server for the demonstrations at the sprint’s final day.
Daniel continued his work on supporting R code cells, but albeit small contributions to the Stencila codebase, he could not complete this task in time.

The sprint ended with presentations by all projects, some of which are still continuing today, for example Appstract, Citation Gecko, PREreview, or Octopus.
The results were truly awesome, ranging from ambitious concepts, case studies, design concepts, completely new tools with great UX design, to technical demonstrators.
It’s an easy guess where on the spectrum our project can be placed…
You’re invited to catch a glimpse of the sprint, its results, and the people behind all of it on Twitter under the hashtag #eLifeSprint and read the project roundup.

The following screencast and Binder link show the status at the end of the sprint: a Stencila document could be opened on a bespoke Binder deployment and the contained Python code could be interactively edited.
The code is re-run on the server and the figure updated.

You can view the Python example document by appending ?archive=py-jupyter to the URL of Stencila in the Binder, e.g. https://hub.mybinder.org/.../stencila/?archive=py-jupyter.

Consolidation

A couple of weeks after the sprint, a second less intensive development period started. Daniel continued his work on adding support for the R context, and also managed to get plain Python cells running (see pull requests #15 and #16).
Min restructured the whole project and gave it the name it still bears: nbstencilaproxy - a Jupyter notebook server extension and proxy for Stencila.

The projects GitHub repository holds a Python module with the Jupyter notebook server and “non-server” extensions of the same name, and a bundledJavaScript module (of the same name).

The Python module allows proper versioned installation, dependency management, and installation from an established software repository.
It takes care of the plumbing between the user interface and the services in the background, so that the binder is viewable over one port in the browser, while the many different background components run on their own ports.
The “no server” extension adds the “Stencila session” menu entry and conveniently lives in the same directory structure as the server extension.

The JavaScript module manages the required JavaScript dependencies and provides an well-defined structure for the code files.
It serves the Dar document and provides access to the Stencila host (see above).

While complex at first sight, this modularity hopefully makes maintenance for future developments and new collaborators easier.
For now, the JavaScript module and its installation are bundled with the Python module instead of being published independently, because the code and configuration is very much specific to the Jupyter integration.

Min also extended repo2docker with automatic detection of Dar documents (as part of a “build pack”), so that no configuration is required for most common use cases.
As with most Binder repositories, a user could simply open a Dar document on Binder and trust the required environment to provide all required software.

On July 20th the nbstencilaproxywas published on PyPI and on August 1st, the new developments made it into repo2docker.
Soon after Stencila was available for all users on mybinder.org, which was a great achievement for a little project started at a community sprint.
However, the big announcement was still not made, since some things were still hard-wired and, for example, to use R, the author of a repository had to manually add a configuration file although the information that R is needed is already part of the Stencila document.

The last mile

In October, Daniel took on the final tasks of writing this blog post and fixing the R installation, including the automatic detection of the required execution contexts of a given Dar document.
This included some housekeeping in nbstencilaproxy and more importantly new contributions to repo2docker (thanks to Tim for review and help) to (i) properly detect the languages used in a Stencila document, (ii) extend the R build pack to install R if it is used in a Stencila document, and (iii) add documentation and tests.
repo2docker now detects Dar documents based on their manifest.xml files and uses the location of the first discovered one as the base directory to start Stencila.
If a Dar manifest is found, then nbstencilaproxy is installed and the languages are extracted from code cells from the document.
Authors can install extra dependencies using the repo2docker’s existing mechanisms.

Daniel also created a few example repositories to provide a starting point for users.
Thankfully the binder team generously welcomed the changes to mybinder.org and and the examples to the binder examples organisation on GitHub.
The following repositories contain single or multiple Stencila documents with code chunks in different programming languages.

Summary and outlook

Thanks for reading so far!
This blog post is a long planned write-up of the history of the tool and decisions mostly relevant to developers, but also an demonstration of the power that the Open Source and Open Science community can foster.
Many people are working together on the (technological) challenges of science today towards full research transparency and reproducibility, even if we use computers to an unprecedented level.
Many small contributions on “side projects” such as these can make a difference, and connecting these two great projects hopefully helps to solve some problem in science down the road.

What’s next?
While there are no concrete plans, there are of course some ideas listed on the project’s issue tracker, such as an automatic Jupyter notebook to Dar conversion when there is no Dar archive in a repository.
In any case you can keep an eye out on GitHub for projects being tagged stencila and binder and join the public Stencila and binder chats to stay in touch or get help.
We look forward to see scientists using nbstencilaproxy for communicating their work and new challenges that come with it.

Screenshot 1: o2r reference implementation listing of published Executable Research Compendia. The right-hand side shows a metadata summary including original authors.

The papers were originally published in Journal of Statistical Software or in a Copernicus Publications journal under open licenses.
We have created an R Markdown document for each paper based on the included data and code following the ERC specification for naming core files, but only included data, an R Markdown document and a HTML display file.
The publication metadata, the runtime environment description (i.e. a Dockerfile), and the runtime image (i.e. a Docker image tarball) were all created during the ERC creation process without any human interaction (see the used R code for upload), since required metadata were included in the R Markdown document’s front matter.

The documents include selected figures or in some cases the whole paper, if runtime is not extremely long.
While the paper’s authors are correctly linked in the workspace metadata (see right hand side in Screenshot 1), the “o2r author” of all papers is o2r team member Daniel since he made the uploads.
You can find all publications on his author page (this is the link you definitely want to try out!):

Screenshot 2: o2r reference implementation ERC detail page for compendium [SLVlQ](https://o2r.uni-muenster.de/#!/erc/5LVlQ). The link "Article" in the top left corner leads to the original article, the "magnifying glass" button takes you to a core feature: the reproduction result.

You can get to the original publication by clicking the “Article” button in the top left corner (see Screenshot 2).
The workspaces demonstrate a variety of issues and are a great source for future work on architecture and implementation.
Here are some examples of the power of a reproducible research service and publishing platform:

The ERC for “Tidy Data” by Hadley Wickham completes the reproduction successfully, so no differences between the uploaded and reproduced HTML file were found! You can even download the image tarball (just bear with our demo - not production - server it it takes some time).

A demo ERC with randomised output shows how things can really go wrong. Feel free to click “Run Analysis” and see how the differences changes with each execution.

If you want to go through the creation process yourself, register on the platform (this requires a short manual interaction by us) and upload one of selected workspaces, which you can find in our public demo share at https://uni-muenster.sciebo.de/s/G8vxQ1h50V4HpuA (just look for zip files starting with corpus_..).
Please take care to choose appropriate licenses and be aware that we might remove compendia from the demo platform without prior notice.

More and more research use data and algorithms to answer a question.
That makes it harder for researchers to understand a scientific publication, because you need more than just the text to understand what is really going on.
You need the software and the data to be able to tell if everything is done correctly, and to be able to re-use new and exciting methods.
We took a look at the existing guides for such research and created our own criteria for research in sciences using environmental observations and maps.
We used the criteria to test how reproducible a set of papers from the AGILE conference actually are.
The conference is quite established and the papers are of high quality because they were all suggested for the “best paper” awards at the conference.

The results are quite bad!
We could not re-create any of the analyses.
Then we asked the authors of the papers we evaluated if they had considered that someone else might want to re-do their work.
While they all think the idea is great, many said they do not have the time for it.

The only way for researchers to have the time and resources to work in a way that is transparent to others and reusable openly is either to convince them of the importance or to force them.
We came up with a list of suggestions to publishers and scientific conference organisers to create enough reasons for researchers to publish science in a re-creatable way.

Last week o2r team member Daniel co-organised a workshop at the 21st AGILE International Conference on Geographic Information Science in Lund, Sweden.
The workshop went very well and Daniel together with his colleagues was able to spread the word about reproducible research and Open Science.
They are pretty sure they convinced some new scientists to reconsider their habits!

The workshop series will probably be continued at the next AGILE conference in Limassol, Cyprus.
For o2r participating in such a workshop is a great way to stay in touch with users of reproducibility tools and practices, and to give back to the communities not only with technology but with education.

Last week EGU General Assembly (GA) 2018 took place in Vienna, Austria, and it was packed with interesting sessions and inspiring presentations.
The o2r team humbly tried to contribute to a massive conference: 15075 participants from 106 countries gave 17323 presentations in 666 sessions (it’s been reported the programme committee briefly discussed adding a session…), taught 68 short courses, and collaborated in 294 side events.
Let’s go through the events with o2r participation in chronological order.

Image courtesy of EGU website.

On Monday, Daniel joined the first ever EarthArXivtownhall meeting.
He was happy to share his stickers with a small but engaged crowd and experienced an open-minded discussion about the young community-led EarthArXiv (already over 300 pre- and postprints after a little over 6 months), preprints, postprints, and the bigger picture of Open Access in the context of the two large conferences in the geosciences, EGU GA and the AGU Fall Meeting. AGU’s cooperation with ESSOAr on abstracts and poster publications was presented at the meeting by Brooks Hanson.
The event went really well and it was fun to meet fellow Open Science enthusiasts Friedrich Hawemann and David Fernandez-Blanco (the mastermind behind the gif-loaden entertaining tweets by @EarthArXiv).

On Tuesday the evening events continued with the always enjoyable OSGeotownhall meeting.
Its theme was “Open Science demystified” and organiser Peter Löwe nicely connected the spirit and goals of an Open Source organisation with Open Science.
As usual, it did not take long until newcomers could be helped with concrete advise on software, development, and transformation to FOSS for organisations by the attending mix of FOSS users, developers, contributors, and old-timers.

On Wednesday Daniel had to shift his attention to the early morning.
In the PICO session IE4.4, “R and the benefit of low-cost solutions - democratic participation to face challenges in Earth science”, he demonstrated the usefulness of rocker/geospatial for science with a number of showcases in a PICO presentation slot packed with exciting projects and software presentation.

In the same session, Edzer presented “R vector and raster data cubes for openEO”, his latest work to continue the evolution for spatial data handling in R and connecting it to openEO.
Both o2r team members could welcome many interested scientists at their PICO screens and had to stay until the very end answering questions and discussing the depths of the respective implementations.

On Thursday afternoon Daniel was joined by Markus and good friends of o2r from New York, Vicky and Remi from NYU Center for Data Science and ReproZip, to continue the collaboration on teaching tools for Open Science and Reproducible Research at EGU.
They welcomed a large audience (70+ people) to the short course“Writing reproducible geoscience papers using R Markdown, Docker, and GitLab”.
The course was hands-on, so participants worked with their own laptops.
Hopefully most of them arrived safely home with a working research environment for R and git.
The course’s contents and speed are adjusted to accommodate the diverse previous knowledge from a multidisciplinary conference such as EGU.
Inspired by the great Carpentry courses, but considerably shorter and more dense, all conveners/teachers were active at the same time to lead through the instructions and help fixing the many little issues during software installations.
We tried hard to leave no one hanging behind and albeit being confronted with an estimated number of 6 operating systems the RStudio-based instructions stood their ground excellently and we are glad to have received numerous positive feedbacks from participants.

We sincerely thank the attendees for the useful questions and the positive atmosphere at the course!
People were helping each other and showed patience when little breaks had to be taken to solve individual issues.
We welcome comments, ideas and suggestions in the GitLab repository of the course.
We hope it’s not the last time we can use the material ourselves but also invite everybody to use it.
It contains numerous links to more detailed courses and we thank the R and Open Science communities for the breadth of existing tutorials and the inspiration they provide.

The evening belonged to yet another townhall meeting: “Research Software Engineers in the Geosciences”.
Daniel initiated this meeting to bring together researchers developing software, or software developers doing research, to get to know the existing national chapters and initiatives as well as each other.
A diverse group came together from different countries and scientific divisions to share their experiences and to discuss how to improve the situation for RSEs in the geosciences (see de-RSE’s objectives).
A more detailed report from the townhall will follow in due course on the de-RSE Blog, until then see Daniel’s full thread on Twitter.

On Friday it was time for PICOs and posters. Daniel and Markus presented “Open Environmental Data Analysis” and “Reproducible research bindings” respectively in the session“Open Data, Reproducible Research, and Open Science”.
Again we enjoyed fruitful discussions and missed out on the interesting other presentations as we were fortunate enough to be visited by interested people throughout the whole viewing time.

We are happy to thank the great people from Copernicus, the organisers of the conference and a great supporter of Open Science as well as a partner of o2r, who we got to meet and catch up with.
The conference has been great and here we only scratch the surface of fun, entertaining and educational experiences where o2r team members presented or convened and lack the many times we met new people and communities, colleagues, and friends to talk science.

First we talked about how digitisation is a familiar topic for computer scientists professionally (digital data, algorithms), but also something we encounter as citizens.
Next I explained the importance of reproducibility in science and when asked if that was not the case in the past, I outlined the new challenges of a completely digital research workflow.
I summarise the idea of the o2r project and use the term “digital laboratory” as a metaphor for our Executable Research Compendium, which collects all research artefacts and opens them up in a transparent way and allows collaboration.
We then briefly touch on the fact that the concepts and ideas are relevant for all sciences, but how o2r (and geoinformatics) focuses on geoscience applications.
Looking ahead I mention our plans to bring our idea of a reproducible article into the publishing workflow of regular scientists.
Is digitisation a blessing or a curse? It’s both, because it creates new challenges and exciting applications in geoinformatics, but the amount of information in large datasets is not always clear and requires critical dealing.

What was it like?

Shooting the interview was fun and a great experience.
Having two cameras pointed at you and 4 people standing critically observing in the background could have been intimidating, but it wasn’t - big thanks to the team!

The film shooting took only about 40 minutes (some technical preparations, two full takes, a little break and small talk) and was prepared with a one-hour conversation some weeks ahead.
I wrote down answers to some questions I expected - but the actual questions were not shared, for the better I think because the spontaneity makes it a conversation and less of a lecture.
The video published online is from the second, shorter take.
I wish they would have used the first one, because it was longer (around 10 minutes, so double the target time) and I could make more good points and feel like I got the message across better.
Researchers can go on for hours about topics they care about, and I hope my enthusiasm about Open Science and Reproducible Research does come across.
Though I am partly unhappy with the content, I hope the brevity spikes interest by fellow researchers and students at University of Münster.
Next time you feel like cutting down your paper from 6000 to 5000 words is hard, try bringing it down to 2 minutes of talking to a non-expert :-).
A worthwhile excercise by the way.

Although in this case, a non-expert could very well be an experienced scientists!
The lack of a common terminology for reproducible/replicable etc. became very apparent during the preparations.
For the next video I’ll make every spectator read “Terminologies for Reproducible Research” first …

“Digitalisierung” is a very hot topic in Germany, both in politics and economy. It regularly makes it into national news and more and more also into small talk.
There is so much connected with digitisation I did not touch on, like artificial intelligence (AI). How can we ensure research transparency and reproducibility when we don’t even know how something works?
The one thing I regret not saying in the interview is the fact that having studied computer science, I rarely grasp the difficulties non-programmers must have with digitisation.
While I do sometimes have to “explain computers” to friends and family, I don’t do it often enough, and must show more patience when I get the chance.
AI, web services, cloud computing - it is complex stuff!
Let’s help non-techies close to us more in understanding them (reading recommendation: “Tubes: A Journey to the Center of the Internet” by Andrew Blum) !

Formulating my view on digitisation in research and its impact on research reproducibility in “plain language” was a worthwhile challenge and I can only recommend every researcher to participate in public relations workshops et cetera to try it out.
You got to take every chance you can get to reach out to non-scientists (research resolution: Make sure all papers have non-specialist summaries).
I applaud all bloggers and podcasters out there who do!

It’s the time of the year to make resolutions and to see beyond one’s own nose.
For o2r team member Daniel, this meant to explore what he could do with his brand new senseBox:home and the awesome BinderHub instead of putting it on the back burner.

Building on a deep stack of Open Hardware, Free and Open Source Software, and Open Data, he created a fully open analysis of particulate measurements at New Year’s Eve in Münster.
With just a few clicks you can open the exact computational environment which he utilized to retrieve historic sensor data from the openSenseMap API, and to analyse and visualise it with R.
And all that without installing any software to your computer, all you need is a web browser.

The following screenshots show the RStudio and Jupyter Notebook renderings of the workflow.

And of course he worried about reproducibility and put in several layers of backup! Learn all about it at the GitHub repository:

Open Science, R, and FOSS sessions at EGU General Assembly 2018

The European Geophysical Union’s General Assembly (EGU GA) takes place once more in April in Vienna - #EGU18.
The deadline for abstracts is 10 Jan 2018, 13:00 CET, so don’t delay, submit your abstract today to one of the following sessions:

The half day workshop attempts to provides a hands-on introduction to reproducible research by reproducing a provided real-world publication. Together with the instructors they create a reproducible document from text, code, and data of a scholarly publication and publish it in a data repository.

The only efforts besides a few commands on your computer is registering a client application with ORCID to be able to log in, because there is no other way to authenticate within the o2r platform and microservices.
You may also get an access token from Zenodo to “ship” your completed ERC.
Eventually this repository will be the basis for a citable package of our software.

Introduction

Today badges are widely used in open source software repositories. They have a high recognition value and consequently provide an easy and efficient way to convey up-to-date metadata. Version numbers, download counts, test coverage or container image size are just a few examples. The website Shields.io provides many types of such badges. It also has an API to generate custom ones.

Now imagine similar badges, i.e. succinct, up-to-date information, not for software projects but for modern research publications. It answers questions such as:

When was a research paper published?

Is the paper openly accessible?

Was the paper published in a peer reviewed journal?

What is the research’s area of interest?

Are the results reproducible?

These questions cover basic information for publications (date, open access, peer review) but also advanced concepts: the research location describes the location a study is focusing on. A publication with reproducible results contains a computation or analysis and the means to rerun it - ideally getting the same results again.

We developed a back-end service providing badges for reproducible research papers.

Overview of badges for research

We are however not the first nor the only ones to do this: ScienceOpen is a search engine for scientific publications. It has badges for open access publications, content type, views, comments and the Altmetric score as displayed in Figure 1.

Figure 1: ScienceOpen badges in a search result listing.

These are helpful when using the ScienceOpen website, but they are not available for other websites. Additional issues are the inconsistent style and missing information relevant for reproducible geosciences, e.g. reproducibility status or the research location.

A study by Kidwell et al. [1] demonstrates a positive effect by the introduction of open data badges in the journal Psychological Science: After the journal started awarding badges for open data, more articles stating open data availability actually published data (cf. [2]). They see badges as a simple yet effective way to promote data publishing. The argument is very well summarized in the tweet below:

Peng [3, 4] reports on the efforts the journal Biostatistics is taking to promote reproducible research, including a set of “kite marks”, which can easily be seen as minimalistic yet effective badges. D and C if data respectively code is provided, and R if results were successfully reproduced during the review process (implying D and C). Figure 3 shows the usage of R on an article’s title page (cf. [5]).

Figure 3: Biostatistics kite mark R rendering in the PDF version of the paper.

The Association for Computing Machinery (ACM) provides a common terminology and standards for artifact review processes for its conferences and journals, see their policies website section on Artifact Review Badging. The have a system of three badges with several levels accompanied by specific criteria. They can be independently awarded:

Artifacts Evaluated means artifacts were made available to reviewers and awarded the level Functional or Reusable

Artifacts Available means a deposition in a repository ensures permanent and open availability (no evaluation)

Results Validated means a third party successfully obtained the same results as the author at the levels Results Replicated (using, in part, artifacts provided by the author) or Results Reproduced (without author-supplied artifacts)

Although these examples are limited to a specific journal, publisher, or institution, they show the potential of badges. They also show the diversity, limitations, and challenges in describing and awarding these badges.

For this reason, our goal is to explore sophisticated and novel badge types (concerning an article’s reproducibility, research location, etc.) and to find out how to provide them independently from a specific journal, conference, or website.

An independent API for research badges

Advanced badges to answer the above questions are useful for literature research, because they open new ways of exploring research by allowing to quickly judge the relevance of a publication, and they can motivate efforts towards openness and reproducibility. Three questions remain: How can the required data for the badges be found, ideally automatically? How can the information be communicated? How can it be integrated across independent, even competitive, websites?

Some questions on the data, such as the publication date, the peer review status and the open access status can already be answered by online research library APIs, for example those provided by Crossref or DOAJ.
The o2r API can answer the remaining questions about reproducibility and location: Knowing if a publication is reproducible is a core part of the o2r project. Furthermore, the location on which a research paper focuses can be extracted from spatial files published with an Executable Research Compendium [6]. The metadata extraction tool o2r-meta provides the latter feature, while the ERC specification and o2r-muncher micro service enable the former.

How can we integrate data from these different sources?

o2r-badger is a Node.js application based on the Express web application framework. It provides an API endpoint to serve badges for reproducible research integrating multiple online services into informative badges on scientific publications. Its RESTful API has routes for five different badge types:

executable: Information about executability and reproducibility of a publication

licence: licensing information

spatial: a publication’s area of interest

releasetime: publication date

peerreview: if and by which process the publication was peer reviewed

The API can be queried with URLs following the pattern /api/1.0/badge/:type/:doi. :type is one of the aforementioned types, and :doi is a publication’s Digital object identifier (DOI).

The badger currently provides badges using two methods: internally created SVG-based badges, and redirects to shields.io.
The redirects construct a simple shields.io URL.
The SVG-based badges are called extended badges and contain more detailed information: the extended license badge for example has three categories (code, data and text, see Figure 5), which are aggregated to single values (open, partially open, mostly open, closed) for the shields.io badge (see Figure 6).

Extended badges are meant for websites or print publications of a single publication, e.g. an article’s title page. They can be resized and alternatively provided pre-rendered as a PNG image. In contrast, the standard shields.io badges are smaller, text based badges. They still communicate the most important piece of information:

They excel at applications where space is important, for example search engines listing many research articles. They are generated on the fly when a URL is requested (e.g. https://img.shields.io/badge/licence-open-44cc11.svg) which specifies the text (e.g. licence and open) and the color (44cc11 is a HTML color code for green).

Let’s look at another example of an executable badge and how it is created.
The badge below is requested from the badger demo instance on the o2r server by providing the DOI of the publication for the :doi element in the above routes:

This URL requests a badge for the reproducibility status of the paper “Global Air Quality and Pollution” from Science magazine identified by the DOI 10.1126/science.1092666. When the request is sent, the following steps happen in o2r-badger:

The badger tries to find a reproducible research paper (called Executable Research Compendium (ERC) via the o2r API. Internally this searches the database for ERC connected to the given DOI.

If if finds an ERC, it looks for a matching job, a report of a reproduction analysis.

Depending on the reproduction result (success, running, or failure) specified in the job, the badger generates a green, yellow or red badge. The badge also contains text indicating the reproducibility of the specified research publication.

The request is redirected to a shields.io URL link containing the color and textual information..

The returned image contains the requested information, which is in this case a successful reproduction:

If an extended badge is requested, the badger itself generates an SVG graphic instead.

Badges for reproducibility, peer review status and license are color coded to provide visual aids. They indicate for example (un)successful reproduction, a public peer review process, or different levels of open licenses.
These badges get their information from their respective external sources: the information for peer review badges is requested from the external service DOAJ, a community-based website for open access publications. The Crossref API provides the dates for the releasetime badges. The spatial badge also uses the o2r services. The badger service converts the spatial information provided as coordinates into textual information, i.e. place names, using the Geonames API.

Spread badges over the web

There is a great badge server, and databases providing manifold badge information, but how to get them displayed online? The sustainable way would be for research website operators to agree on a common badge system and design, and then incorporate these badges on their platforms. But we know it is unrealistic this ever happens.
So instead of waiting, or instead of engaging in a lengthy discourse with all stakeholders, we decided to create a Chrome extension and augment common research websites. The o2r-extender automatically inserts badges into search results or publication pages using client-side browser scripting. It is available in the Chrome Web Store and ready to be tried out.

For each article display on these websites, the extender requests a set of badges from the badger server. These are then inserted into the page’s HTML code after rendering the regular website as shown exemplary in the screenshot in Figure 7.

When the badger does not find information for a certain DOI, it returns a grey “not available” - badge instead. This is shown in the screenshot above for the outermost license and peer review badges.

The extender consists of a content script, similar to a userscript, adjusted to each target website. The content scripts insert badges at suitable positions in the view. A set of common functions defined in the Chrome extension for generating HTML, getting metadata based on DOIs, and inserting badges are used for the specific insertions. A good part of the extender code is used to extract the respetive DOIs from the information included in the page, which is a lot trickier than interacting with an API. Take a look at the source code on GitHub for details.

But the extender is not limited to inserting static information. The results of searches can also be filtered based on badge value and selected badge types can be turned on or off directly from the website with controls inserted into the pages’ navigation menus (see left hand side of Figure 8).

Figure 8: Filtering search results on DOAJ. Results not matching the filter or articles where the DOI could not be detected are greyed out.

The extender is easily configurable: it can be enabled and disabled with a click on the icon in the browser toolbar. You can select the badge types to be displayed in the extension settings. Additionally it contains links to local info pages (“Help” and “About”, see Figure 9).

Figure 9: extender configuration.

Outlook: Action integrations

The extender also has a feature unrelated to badges. In the context of open science and reproducible research, the reproducibility service connects to other services in a larger context as described in the o2r architecture (see section Business context).

Two core connections are loading research workspaces from cloud storage and connecting to suitable data repositories for actual storage of ERCs.
To facilitate these for users, the extender can also augment the user interfaces of the non-commercial cloud storage service Sciebo and the scientific data repository Zenodo with reproducibility service functionality.

When using Sciebo, a button is added to a file’s or directory’s context menu. It allows direct interaction with the o2r platform to upload a new reproducible research paper (ERC) from the current file or directory as shown in Figure 10.

Figure 10: Sciebo upload integration.

When you are viewing an Executable Research Compendium on Zenodo, a small badge links directly to the corresponding inspection view in the o2r platform (see Figure 11):

Figure 11: Link to inspection view and tag "ERC" on Zenodo.

Discussion

The study project Badges for computational geoscience containers initially implemented eight microservices responsible for six different badges types, badge scaling and testing. A microservice architecture using Docker containers was not chosen because of the need for immense scaling capabilities, but for another reason: developing independent microservices makes work organization much easier. This is especially true for a study project where students prefer different programming languages and have different skill sets.

However, for o2r the microservices were integrated into a single microservice for easier maintainability. This required refactoring, rewriting and bug fixing.
Now, when a badge is requested, a promise chain is executed (see source code example). The chain reuses functions across all badges where possible, which were refactored from the study project code into small chunks to avoid callback hell.

A critical feature of extender is the detection of the DOI from the website’s markup. For some websites, such as DOAJ.org or ScienceOpen.com, this is not hard because they provide the DOI directly for each entry.
When the DOI is not directly provided, the extender tries to retrieve the DOI from a request to CrossRef.org using the paper title (see source code for the DOI detection). This is not always successful or may find incorrect results.

The Chrome extension supports nine different websites. If there are changes to one of these, the extender has to be updated as well. For example, Sciebo (based on ownCloud) recently changed their URLs to include a “fileid” parameter which resulted in an error when parsing the current folder path.

As discussed above, in an ideal world the Chrome extension would not be necessary. While there are a few tricky parts with a workaround like this, it nevertheless allows o2r as a research project to easily demonstrate ideas and prototypes stretching beyond the project’s own code to even third party websites.
Moreover, the combination of extender client and badger service is suitable for embedding a common science badge across multiple online platforms. It demonstrates a technical solution how the scientific community can create and maintain a cross-publisher, cross-provider solution for research badges. What it clearly lacks is a well-designed and transparent workflow for awarding and scrutinizing badges.

Future Work

One of the biggest source of issues for badger currently is the dependence on external services such as Crossref and DOAJ. While this cannot be directly resolved, it can be mitigated by requesting multiple alternative back-end services, which can provide the same information (e.g. DOAJ for example also offers licence information at least for publications), or even by caching.
Furthermore, the newness of the o2r platform itself is another issue: licence, executable, and spatial badges are dependent on an existing ERC, which must be linked via DOI to a publication. If a research paper has not been made available as an ERC then a users will get a lot of “n/a” badges.

The extender is only available for Google Chrome and Chromium. But since Firefox is switching to WebExtensions and moving away from their old “add-ons” completely with Firefox 57, a port from a Chrome Extension to the open WebExtensions makes the extender available for more users. The port should be possible with a few changes due to only minor differences between the two types of extensions.

Other ideas for further development and next steps include:

Interactive badges can provide additional information when hovering over them or when the badges are clicked, most importantly why and by who the badge was assigned.

Provide the information behind the badges via an API.

Create a common design for extended badges.

Conduct a user study on extended and basic badges within a discovery scenario.

Evaluating usage of badges in print applications and for visually impaired people (cf. COS badges)

Daniel presented a new R extension package, containerit, in the Data reproducibility session.
It can automatically create a container manifest, i.e. a Dockerfile, from different sources, such as sessions or scripts.

If you want to learn more about containerit, read this blog post and take a look at Daniel’s presentation (also on Zenodo).

Fortunately the presentation was very well-attended and assured our understanding that the importance of reproducibility is wide-spread in the R community.
The interest in using containers for this challenge is growing, as shown by the numerous questions Daniel received after the session and the remainder of the conference.

Unlike the workshop’s title, Docker was not the only container solution. Singularity made some important appearances, especially with the different groups working with clusters of thousands of nodes in HPC (high performance computing) and GPGPUs (general processing on graphical processing units). Further topics included deep learning, statistical reports by governments, using containers for teaching, scientific workflows in the cloud, virtual clusters and “best before” dates for software.

Daniel had the hard job of giving the final presentation. After all the previous talks, which comprises many different aspects of reproducible research also somehow part of o2r, this was a threatening task and felt a bit like like “imposters syndrome”. However, the commonalities in motivation, challenges, and ideas are also a sign of the increasing popularity for using containers across diverse domains. Eventually it is a very positive fact an event such as C4RR took place in Europe and had more than 50 people in attendance!

The workshop was a great experience and very well organized by the Software Sustainability Institute. We learned about both related and quite similar projects, but also acknowledged that o2r’s focus on “Desktop-sized” data and computing as well as supporting the geosciences domain does set us apart.

1. Introduction

Even though R is designed for open and reproducible research, users who
want to share their work with others are facing challenges. Sharing
merely the R script or R Markdown document should warrant
reproducibility, but many analyses rely on additional resources and
specific third party software as well. An R script may produce
unexpected results or errors when executed under a different version of
R or another platform. Reproduciblility is only assured by providing
complete setup instructions and resources. Long-term reproducibility can
be achieved by either regular maintenance of the code, i.e. keeping it
always working with the latest package versions from CRAN. It can be
supported by packages such as
packrat and platforms such as
MRAN, which provide means to capture a
specific combination of R packages. An alternative to updating or
managing packages explicitly is providing the full runtime environment
in its original state, using virtual
machines or software
containers.

The R extension package containerit aims to facilitate the latter
approach by making reproducible and archivable research with containers
easier. The development is supported by the DFG-funded project Opening
Reproducible Research (o2r, https://o2r.info). containerit relies on
Docker and automatically generates a container
manifest, or “recipe”, with setup instructions to recreate a runtime
environment based on a given R session, R script, R Markdown file or
workspace directory. The resulting
Dockerfile can
not only be read and understood by humans, but also be interpreted by
the Docker engine to create a software container containing all the R
packages and their system dependencies. This way all requirements of an
R workflow are packaged in an executable format.

The created Dockerfiles are based on the
Rocker project (Rocker on
Docker Hub,
introduction).
Using the stack of version-stable Rocker images, it is possible to match
the container’s R version with the local R installation or any R version
the user requires. containerit executes the provided input workspace
or file first locally on the host machine in order to detect all
dependencies. For determining external software dependencies of attached
packages, containerit relies (a) on the sysreqs
database and makes use of the corresponding
web API and R package, and (b) on internally defined rule sets for
challenging configurations.

The Dockerfile created by containerit can then be used to build a
Docker image. Running the image will start an R session that closely
resembles the creating systems runtime environment. The image can be
shared and archived and works anywhere with a compatible Docker version.

To build images and run containers, the package integrates with the
harbor package and adds a few
convenience functions for interacting with Docker images and containers.
For concrete details on reading, loading, or installing the exact
versions of R packages including their system dependencies/libraries,
this project focuses on the geospatial domain. containerit uses the
package
futile.logger
to provide information to the user at a configurable level of detail,
see futile.logger
documentation.

In the remainder of this vignette, we first introduce the main usage
scenarios for containerit and document current challenges as well as
directions for future work.

2. Creating a Dockerfile

2.1 Basics

The easiest way to generate a Dockerfile is to run an analysis in an
interactive R session and create a Dockerfile for this session by
loading containerit and calling the dockerfile()- method with
default parameters. As shown in the example below, the result can be
pretty-printed and written to a file. If no file argument is supplied
to write(), the Dockerfile is written to the current working directory
as ./Dockerfile, following the typical naming convention of Docker.

When packaging any resources, it is essential that the R working
directory is the same as the build context, to which the Dockerfile
refers. All resources must be located below this directory so that they
can be refered to by relative paths (e.g. for copy instructions). This
must also be considered when packaging R scripts that use relative
paths, e.g. for reading a file or sourcing another R script.

2.3 Packaging an external session

Packaging an interactive session has the disadvantage that unnecessary
dependencies might be added to the Dockerfile and subsequently to the
container. For instance the package futile.logger is a dependency of
containerit, and it will be added to the container because it was
loaded into the same session were the analyses was executed. It cannot
be removed by default, because other packages in the session might use
it as well (even unintentionally in case of generic methods). Therefore,
it is safer not to tamper with the current session, but to run the
analysis in an isolated vanilla session, which does not have
containerit in it. The latter will batch-execute the commands in a
seperate instance of R and retrieves an object of class sessionInfo.
The session info is then used as input to dockerfile(). This is also
how dockerfile() works internally when packaging either expressions,
scripts or R markdown files.

The following code creates a Dockerfile for a list of expressions in a
vanilla session.

2.4 Packaging an R script

R scripts are packaged by just supplying the file path or paths to the
arguement from of dockerfile(). They are automatically copied into
the container’s working directory. In order to run the R script on
start-up, rather than an interactive R session, a CMD instruction can be
added by providing the value of the helper function CMD_Rscript() as
an argument to cmd.

2.5 Packaging an R Markdown file

Similarly to scripts, R Markdown files can be passed to the from
argument. In the following example, a vignette from the Simple Features
package sf is packaged in a container. To render the document at
startup, the Dockerfile’s CMD instruction must be changed. To do this,
the cmd argument passed to dockerfile() is constructed using the
function CMD_Render. Note that, as shown in the Dockerfile, the GDAL
library has to be build from source for sf to work properly, because a
quite recent version of GDAL is required. This adaptation of the
installation instruction is based on an internal ruleset for the package
sf.

2.6 Packaging a workspace directory

A typical case expected to be interesting for containerit users is
packaging a local directory with a collection of data and code files. If
providing a directory path to the dockerfile() function, the package
searches for the first occurence of an R script, or otherwise the first
occurence of an R markdown file. It then proceeds to package this file
along with all other resources in the directory, as shown in the next
section.

3. Including resources

Analyses in R often rely on external files and resources that are
located located in the workspace. When scripts or R markdown files are
packaged, they are copied by default into the same location relative to
the working directory. The argument copy influences how dockefile()
behaves in this matter. It can either have the values script (default
behaviour), script_dir (copies the complete directory in which the
input file is located), or a custom list of files and directories inside
the current working directory

Including R objects works similar to resources, using the argument
save_image. The argument can be set to TRUE to save all objects of
the current workspace to an .RData file, which is then copied to the
container’s working directory and loaded on startup (based on
save.image()).

4. Image metadata

Metadata can be added to Docker images using Label
instructions.
Label instructions are key-value pairs of arbitrary content. A dublicate
key overwrites existing ones. Although it is up to the user how many
labels are created, it is recommended to bundle them into one Label
instruction in the Dockerfile. Each use of the Label() function
creates a seperate instruction in the Dockerfile.

As shown in section 2, the maintainer label is set by default to the top
as the dockerfile and contains the username of the current host system.
The maintainer can be changed with the maintainer argument of
dockerfile():

Labels can be applied to the existing Dockerfile object using the
addInstructions() function, which adds any newly created instructions
to the end of the Dockerfile but before the CMD statement. The Label()
constructor can be used for creating labels of arbitrary content and
works similar to creating named lists in R.

5. Further customization

The dockerfile() function allows further customization regarding the R
version or the used base image (cf. Rocker stack). Note that while
choosing an R version for the Dockerfile explicitly is possible, the
session to generate the required information (i.e. which packages are
attached etc.) is still running the R version of the generating machine.

The following examples show usage of these options and the respective
FROM statements in the Dockerfile.

7. Challenges

We encountered several challenges during containerit’s development.
First and foremost, a well known limitation is that R packages don’t
define system dependencies and do not provide explicit versions for R
package dependencies. The sysreqs package is a promising approach
towards handling system requirements, but so far lists package names but
does not provide version information. The
shinyapps-package-dependencies
demonstrate a (currently system dependent) alternative. The high value
of R might well lie in the fact that “packages currently on CRAN” should
work well with each other.

An unmet challenge so far is the installation of specific versions of
external libraries (see
issue). A
package like sf relies on well-tested and powerful system libraries,
see sf::sf_extSoftVersion(), which ideally should be matched in the
created container.

And of course users may do things that containerit cannot capture from
the session state “after the analysis is completed”, such as detaching
packages or removing relevant files, and unknown side-effects might
occur.

All software is presumed to be installed and run on the host system.
Although it is possible to use deviating versions of R or even create
Dockerfiles using sessionInfo-objects created on a different host, this
may lead to unexpected errors because the setup cannot be tested
locally.

8. Conclusions and future work

containerit alows to create and costumize Dockerfiles with minimal
effort, which are suitable for packaging R analyses in the persistant
runtime environment of a software container. So far, we were able to
reproduce complete R sessions regarding loaded and attached packages and
mitigate some challenges towards reproducible computational research.

Although we are able to package different versions of R, we still do not
fully support the installation of specific versions of R packages and
external software libraries, which R itself does not support. This
should be tested in the future by evaluating version-stable package
repositories like MRAN and GRAN or utility packages such as packrat –
see the GitHub
issues for the
status of these plans or provide your own ideas there.

Related to installing specific versions is support for other package
repositories, such as Bioconductor, git, BitBucket, or even local files.
For now, it is recommended that users have all software up-to-date when
building a software container, as the latest version are installed from
CRAN during the image build, to have matching package versions between
the creation runtime environment and the container. All Dockerfiles and
instructions are adjusted to the Rocker image stack and assume a
Debian/Linux operating system. As we are not yet supporting the build of
Docker images from scratch, we are restricted to this setup.

The package is a first prototype available via GitHub. While a
publication on CRAN is a goal, it should be preceded by feedback from
the user community and ideally be accompanied by related packages, such
as harbor, being available on
CRAN, too. The prototype of containerit was developed and tested only
on Ubuntu/Linux, which should be extended before releasing a stable
version on CRAN.

As part of the o2r project, it is planned to integrate containerit in
a web service for creating archivable
research in form of Executable Research Compendia
(ERC). Making containerit
itself easier to use for end-users is a secondary but worthwhile goal, for example by
building a graphical user interface for metadata creation. Country
locales are also not supported yet. We may want to support other
container OS (e.g. windows container or other Linux distributions) or
even containerization solutions such as
Singularity or the Open Container
Initiative’s (OCI) Image
Format.

Thanks to the valuable questions and comments by our external partners, the project tasks were assessed and refocussed. On top of it, we agreed to collaborate even further and sketched first ideas for putting o2r’s prototypes into real-world settings.

This workshop was only possible thanks to our partner’s commitment, enthusiasm, and continued interest in the project. Our heartfelt thanks go to

As last year, the full day meeting took place in the countryside at the lovely Wersehaus. Unlike last year, we skipped lightning talks and profited from the existing understanding of the project. Instead we dove right into the project’s significant progress: survey results which motivated our design decisions, a critical view on the project schedule and completed/open tasks, the specification for executable research compendia (ERC), our architecture, the API, and most importantly the Open Source reference implementation and its integration with Sciebo and Zenodo.

Just as intented these topics were merely started as presentations and led to an active discussion. They were evaluated and connected to the partners perspectives, not the least by putting more ambitious goals (“let’s completely change the way scholarly publishing works!”) into perspective and defining concrete steps ahead to (i) spread understanding of reproducible research, and (ii) show the potential for enhancements by computational reproducibility with ERC. Many valuable insights will keep the o2r team busy in the following weeks.

In the blog post of the first workshop, we included some statements on what will we understand in two years time that we do not know now?, and here is the original (left) and updated version:

We have a good understanding of how far the process of creating research compendia can be automated, and what efforts remain for authors or preservationists that must be counterbalanced with incentives.

Our understanding is consolidated in specifications, in well-defined user workflows, and is demonstrated by a reference implementation. On the topic of incentives, the need for a cultural change (_"it takes a generation"_) was re-stated at the workshop but we can better communicate o2r's actual contributions.

We know the potential of user interface bindings as the connecting entity of research compendia.

By conducting a survey and interviews with geoscientists, we identified promising use cases for UI bindings. e.g. change an analysis variable and update a diagram. The conceptual description (an ontology) underlying these use cases is in progress. It is an open question if we can realise a generic solution to generate UI bindings automatically, and how much effort by the author is required.

We show the improvements in discovery and understanding of research when all aspects of research are explicitly linked in a meaningful way.

Thanks to feedback by last year's workshop and continued interaction with other researchers at conferences and workshops, we decided to concentrate on these challenging topics first: easily packaging research into ERC, integrating with data repositories, and interacting with ERC. Therefore discovery is a topic for the second half of 2017, including a recently started master thesis.

We get to know the common language as well as points of contact for the involved parties as we create a closer connection between research, preservation, and publication communities.

Success! The prototypes are received well by all of the parties. They provide unifying concepts and workflows and are even seen as ready for pilot studies.

We hope to have another inspirational meeting like this in 2018! To keep in touch, follow us on Twitter or GitHub.

Daniel will also present a poster on o2r titled “An Architecture for Reproducible Computational Geosciences”. Please visit the AGILE 2017 poster session tomorrow at 15:00 and discuss with us how our ERC fits into the geosciences landscape.

Completely different is the scale of this weeks conference: unlike EGU general assembly, AGILE is a small conference with an informal feeling. While the attendees represent diverse topics, the common connection to GI Science is strong and while the programme is packed at times (5 parallel tracks - hard to choose!), there is ample room to focus, for example in the single track keynote or poster sessions, but also to chat and learn, which we hope to do by spreading questions on reproducibility of the works presented at AGILE.

It was a great experience tomeet fellow scientists interested in, and worried about, reproducibility of scholarly works.
We got useful feedback on our practical work and are encouraged again to continue spreading the word on the general topic of reproducibility alongside our research.

The short was divided into two parts: a practical introduction to selected tools supporting computational reproducibility, and talks by stakeholders in the scientific publication process followed by a lively panel discussion.

In the first part, Daniel and Vicky began with sharing some literature on reproducible research (RR) with the roughly 30 participants. After all, the participants should take home something useful, so a reading list seems reasonable for RR newcomers but also for researchers writing about the reproducibility aspects in upcoming papers.

Then Daniel fired up a console and took a deep dive into using containers to encapsulate environments for reproducible computational research. He started with a very quick introduction to Docker and then demonstrated some containers useful to researchers, i.e. Jupyter Notebook and RStudio.

The material presented by Daniel is a starting point for an Author Carpentry lesson, which is currently developed on GitHub, so he highly appreciates any feedback, especially by shourt course attendees. We were surprised to learn a good portion of the participants had already some experience with Docker. But even better was realizing a few actually hacked along as Daniel raced through command-line interface examples! This “raw” approach to packaging research in containers was contrasted in the second section.

Under the title “ReproZip for geospatial analyses”, Vicky and Rémi showcased ReproZip, a tool for automatically tracing and packaging scientific analyses for easily achieved computational reproducibility. The resulting file is a ReproZip package (.rpz), which can be easily shared due to it’s small size, and contains everything necessary to reproduce research (input files, environmental information etc.) across different operating systems. They demonstrated their various unpackers and showed how these .rpz files can be used for reproducibility and archiving. They also demoed they brand new user interface for the first time in Europe.

The practical demonstrations paved the way for the second part of the short course, which was more abstract yet proofed to excellently demonstrate the breadth of reproducible research. Selected speakers provided their perspectives on the topic of reproducing scientific papers in the broader context of the scientific publication cycle. In short talks they wore a specific role of the scholarly publication process and shared their experience as as researcher, infrastructure provider, publisher, reviewer, librarian, or editor. The speakers:

David Ham shared the priorities of the journal Geoscientific Model Development (GMD) where he is editor, when it comes to reproducibility and the issues they face. Proper provenance and citations are examples for the former, the ephemerality of code and data for the latter.

Xenia van Edig lead us through the stages of Open Access that Copernicus went and is going through as a publisher: from public data (1.0) via interactive articles and public peer review (2.0) to the future of open science and executable papers (3.0)

Vicky Steeves advertised the expertise of librarians worldwide in supporting research in all aspects, including reproducibility, writing grants, or data management plans, but also pointed out the necessity to support scientists with proper tools and teach the required skills.

All speakers touched on the topic of scientific culture, which was seen in a process of changing towards more openness, but with still quite some way to go. The cultural aspects and larger scale challenges were a recurring topic in the panel discussion after the short talks. These aspects included resistance to share supplemental material, so that journals cannot make sharing everything mandatory, for example because of unwillingness (fear of stealing) or because authors might not be allowed to do so. A member of the audience could share that in their experience as a publisher, requiring data and software publication did not result in a decrease in submissions when accompanied by transparent and helpful author guidelines. Such guidelines for both data and code are lacking for many journals but are a means to improve the overall situation - and make the lives of editors simpler.
When the progress of the last years on Open Data was pointed out as largely a top down political endeavour, the contrast to Open Source as a bottom-up grassroots initiative became clear. Nevertheless, the hope was phrased that with the success of Open Data, things might go smoother with Open Source in science.

A further topic the discussion covered for some time was creditation, and the need to update the ways researchers get and give credit as part of grant-based funding and publishing scholarly articles. Though it was pointed out that RR is also about “doing the right thing”. Credit and culture were seen as closely linked topics, which can only be tackled by improving the education of scientists, both as authors and reviewers(!), and spreading the word about the importance of reproducibility for all of science, not least in the light of the marches for sciences taking place just a few days before the short course.

While one could say we were mostly preaching to the choir, it was great to see an interest in the topic of reproducible research amongst EGU attendees. This workshop being the first of its kind at the EGU general assembly hopefully was a step towards even higher visibility and interest for RR as a crucial topic in today’s research.

We thank the short course attendees and invited speakers for turning the first afternoon of EGU 2017 into an instructive and diverting few hours. Will there be a reproducible research short course next year at EGU? We don’t know yet, but please do get in touch if you would like to support the planning. It could be worth providing a longer course targeted as early career scientists, giving the next generation the tools to work reproducibly.

Due to the overall topic of our project, we felt the Open Science Conference (#osc2017) taking place this week in Berlin would be a great chance to share our ideas and meet like-minded folks. We were happy about the notification that our poster was accepted and even made it into the top ten (of altogether 57 submissions), which allowed o2r team member Markus to give a three-minute lightning talk and present a project poster. Both days included interesting talks given by international speakers (see full programme) and in this post Markus reports on the trip.
The first day covered several topics related to o2r, for example, data infrastructures (see European Open Science Cloud). Speakers also mentioned social challenges such as a new reward system and incentives required to motivate scientists to conduct Open Science – a key issue in the Executable Research Compendium-concept as well. In times of fake news and the credibility crisis, keynote speaker Prof. Johannes Vogel strongly encouraged in his opening talk to set a good example in the field of Open Science and convincingly put scientists in charge of the issue.

A few people I talked to liked the idea of making the dataset the actual publication and the paper being “only” the supplementary material. It might be interesting to play around with some thoughts on that: Will institutes focus on publishing datasets instead of papers? Is “data collector” a new job title?

The lightning talks and the poster session were a success. Several visitors were keen to ask questions and to get explanations on technical and conceptual details. I hope that I was able to answer all of them in sufficient detail. If you think I didn’t, please don’t hesitate to ask me or in case of doubts, my colleagues Docker Daniel and Metadata Marc. You should also take a look at Conquaire, an interesting project in the context of reproducible research.

In the evening, we had a wonderful dinner next to dinosaurs (I am not talking about the scientists 😉) organized by Museum für Naturkunde Berlin, a museum of natural science. In this impressive atmosphere, we were able to network a bit and to continue discussions.

The second day was rather education-driven. However, we do also want to enhance and extend the understanding of scientists when examining a paper by using our ERC. Why not addressing students, too? We still dream of a reproducible and interactive atlas.

It was interesting to see that the great majority of guests and speakers focused on open data when discussing challenges in Open Science. Mentioning source-code was rather the exception although reproducibility was perceived as being part of Open Science. For this reason, I think that our contribution to the conference was relevant as we treat (open) code and software as being equally important. I mentioned this aspect in my lightning talk, too, and tried to highlight the importance of source code during the poster presentation. One might argue that open code is implicitly included in open data or open methodology. However, we should not rely on vague interpretations and make explicit what is required to rerun analyses. In the future, submitting, for example, analysis scripts should be as mandatory as it is demanded for datasets.

To conclude, here a few take home messages:

Rewards and incentives that motivate to conduct Open Science are key issues

We have to engage people from society to increase trust in scientific results (tackle credibility crisis)

Problems are social – not technical. BUT: we have to provide scientists with working examples, otherwise they don’t know why to use it and how.

Open Science strongly focuses on data and educational aspects.

P.S. The next time you read about guidelines, recommendations on open data, try to replace it by source code. The argument still works, right?

The short course Reproducible computational research in the publication cycle (SC81) at the EGU general assembly was accepted by the short course programme group and scheduled Monday, April 24th, 2017 in the afternoon. Thanks!

We are grateful for the change to share our work on reproducible research together with members of the ReproZip team in a practical, hands-on short course.
Afterwards we welcome a number of esteemed speakers to share their views on reproducibility as researcher, reviewer, editor, publisher, and preservationist.

With this half day workshop we want to introduce the topic of reproducible research to the AGILE conference series, the most prominent and long-standing GIScience and GIS conference in Europe.
The 3-day conference is accompanied by 13 workshops on diverse topics.

The workshop is co-organized by o2r team members and Frank Osterman from ITC, Enschede. Contributions and a public peer review are done via GitHub and supported by a great programme committee of distinguished researchers. Please share this information with potentially interested parties (and retweet). Thanks!

Docker and R: How are they used and could they be used together?
That is the question that we regularly ask ourself.
And we try to keep up with other people’s work! In this post, we are going to share our insights with you.

Dockerising R

Several implementations of besides the one by R-core exist today, together with numerous integrations into open source and proprietary software (cf. English and German Wikipedia pages).
In the following we present the existing efforts for using open source R implementation with Docker.

With a big choice of pre-build Docker images, Rocker provides optimal solutions for those who want to run R from Docker containers.
Explore it on Github or Docker Hub, and soon you will find out that it takes just one single command to run instances of either base R, R-devel or Rstudio Server.
Moreover, you can run specific versions of R or use one of the many bundles with commonly used R packages and other software, namely tidyverse and rOpenSci).

Images are build monthly on Docker Hub, except devel tags which are build nightly.
Automated builds are disabled, instead builds are triggered by CRON jobs running on a third party server (cf. GitHub comment).

Bioconductor

If you come from bioinformatics or neighboring disciplines, you might be delighted that Bioconductor provides several images based on Rocker’s rocker/rstudio images.
See the help page, GitHub, and Open Hub for more information.
In short, the Bioconductor core team maintains release and devel images (e.g. bioconductor/release_base2), and contributors maintain image with different levels of pre-installed packages (each in release and devel variants), which are based on Bioconductor views (e.g. bioconductor/devel_proteomics2 installs the views Proteomics and MassSpectrometryData).

Image updates occur with each Bioconductor release, except the devel images which are build weekly with the latest versions of R and Bioconductor based on rocker/rstudio-daily.

MRO

Microsoft R Open (MRO) is an “enhanced R distribution”, formerly known as Revolution R Open (RRO) before Revolution Analytics was acquired by Microsoft.
MRO is compatible with main R and it’s packages.
“It includes additional capabilities for improved performance, reproducibility, and platform support.” (source); most notably these are the MRAN repository a.k.a. CRAN Time Machine, which is also used by versioned Rocker images, and the (optional) integration with Intel® Math Kernel Library (MKL) for multi-threaded performance in linear algebra operations (BLAS and LAPACK).

o2r team member Daniel created a Docker image for MRO including MKL.
It is available on Docker Hub as nuest/mro, with Dockerfile on GitHub.
It is inspired by the Rocker images and can be used in the same fashion.
Please note the extended licenses printed at every startup for MKL.

R Client adds to MRO by including a couple of “ScaleR” machine learning algorithms and packages for parallelisation and remote computing.

Renjin

Renjin is a JVM-based interpreter for the R language for statistical computing developed by BeDataDriven.
It was developed for big data analysis using existing R code seamlessly in cloud infrastructures, and allows Java/Scala developers to easily combine R with all benefits of Java and the JVM.

While it is not primarily build for interactive use on the command line, this is possible.
So o2r team member Daniel created a Docker image for Renjin for you to try it out.
It is available on Docker Hub as nuest/renjin, with Dockerfile on GitHub.

pqR

pqR tries to create “a pretty quick version of R” and fixing some perceived issues in the R language.
While this is a one man project by Radford Neal, it’s worth trying out such contributions to the open source community and to the discussion on how R should look like in the future (cf. a recent presentation), even if things might get personal.
As you might have guess by now, Daniel created a Docker image for you to try out pqR: It is available on Docker Hub as nuest/pqr, with Dockerfile on GitHub.

[WIP] FastR

Also targeting performance, FastR is “is an implementation of the R Language in Java atop Truffle, a framework for building self-optimizing AST interpreters.” FastR is planned as a drop-in replacement for R, but relevant limitations apply.

While GraalVM has a Docker Hub user, no images are published probably because of licensing requirements, as can be seen in the GitHub repository oracle/docker-images, where users must manually download a GraalVM release, which requires an Oracle Account… so the current tests available in this GitHub repository, trying to build FastR from source based on the newest OpenJDK Java 9.

Dockerising Research and Development Environments

So why, apart from the incredibly easy usage, adoption and transfer of typical R environments, would you want to combine R with Docker?

Ben Marwick, Associate Professor at the University of Washington, explains in this presentation that it helps you manage dependencies.
It gives a computational environment that is isolated from the host, and at the same time transparent, portable, extendable and reusable.
Marwick uses Docker and R for reproducible research and thus bundles up his works to a kind of Research Compendium; an instance is available here, and a template here.

A new solution to ease the creation of Docker containers for specific research environments is containerit.
It creates Dockerfiles (using Rocker base images) from R sessions, R scripts, R Markdown files or R workspace directories, including the required system dependencies.
The package was presented at useR!2017 and can currently only be installed from GitHub.

While Docker is made for running tools and services, and providing user interfaces via web protocols (e.g. via a local port and a website opened in a browser, as with rocker/rstudio or Jupyter Notebook images), several activities exists that try to package GUI applications in containers.
Daniel explores some alternatives for running RStudio in this GitHub repository, just for the fun of it.
In this particular case it may not be very sensible, because RStudio Desktop is already effectively a browser-based UI (unlike other GUI-based apps packages this way), but for users with reluctance to a browser UI and/or command line interfaces, the “Desktop in a container” approach might be useful.

Running Tests

The package dockertest makes use of the isolated environment that Docker provides: R programmers can set up test environments for their R packages and R projects, in which they can rapidly test their works on Docker containers that only contain R and the relevant dependencies.
All of this without cluttering your development environment.

The package gitlabr does not use Docker itself, but wraps the GitLab API in R functions for easy usage.
This includes starting continuous integration (CI) tests (function gl_ci_job), which GitLab can do using Docker, so the function has an argument image to select the image run to perform a CI task.

In a completely different vein but still in the testing context, sanitizers is an R package for testing the compiler setup across different compiler versions to detect code failures in sample code.
This allows testing completely different environments on the same host, without touching the well-kept development environment on the host.
The packages’ images are now deprecated and superseded by Rocker images (rocker/r-devel-san and rocker/r-devel-ubsan-clang).

Dockerising Documents and Workflows

Some works are dedicated to dockerising R-based documents.

The package liftr (on CRAN) for R lets users enhance Rmd files with YAML-metadata (example), which enables rendering R Markdown documents in Docker containers.
Unlike containerit, this metadata must be written by the author of the R Markdown document.

liftr is used in the DockFlow initiative to containerise a selection of Bioconductor workflows as presented in this poster at BioC 2017 conference.
Liftr also supports Rabix, a Docker-based toolkit for portable bioinformatics workflows.
That means that users can have Rabix workflows run inside the container and have the results integrated directly into the final document.

The Bioconductor package sevenbridges (see also above) has a vignette on creating reproducible reports with Docker.
In recommends a reproducible script or report with docopt respectively R markdown (parametrised reports).
The cloud-based Seven Bridges platform can fulfill requirements, such as required Docker images, within their internal JSON-based workflow and “Tool” description format (example), for which the package provides helper functions to create Tools and execute them, see this example in a vignette.
Docker images are used for local testing of these workflows based on Rabix (see above), where images are started automatically in the background for a user, who only uses R functions.
Automated builds for workflows on Docker Hub are also encouraged.

RCloud is a collaborative data analysis and visualization platform, which you can not only try out online but also host yourself with Docker.
Take a look at their Dockerfiles or try out their image rcl0ud/rcloud.

Control Docker Containers from R

Rather than running R inside Docker containers, it can be beneficial to call Docker containers from inside R.
This is what the packages in this section do.

The harbor package for R (only available via GitHub) provides all Docker commands with R functions.
It may be used to control Docker containers that run either locally or remotely.

A more recent alternative to harbor is the package docker, also available on CRAN with source code on GitHub.
Using a DRY approach, it provides a thin layer to the Docker API using the Docker SDK for Python via the package reticulate.
The package is best suited for apt Docker users, i.e. if you know the Docker commands and life cycle.
However, thanks to the abstraction layer provided by the Docker SDK for Python, docker also runs on various operating systems (including Windows).

dockermachine provides a convenient R interface to the docker-machine command, so you can provision easily local or remote/cloud instances of containers.

Selenium provides tools for browser automation, which are also available as Docker images.
They can be used, amongst others, for testing web applications or controlling a headless web browser from your favorite programming language.
In this tutorial, you can see how and why you can use the package RSelenium to interact with your Selenium containers from R.

googleComputeEngineR provides an R interface to the Google Cloud Compute Engine API.
It includes a function called docker_run that starts a Docker container in a Google Cloud VM and executes R code in it.
Read this article for details and examples.
There are similar ambitions to implement Docker capabilities in the analogsea package that interfaces the Digital Ocean API.
googleComputeEngineR and analogsea use functions from harbor for container management.

R and Docker for Complex Web Applications

Docker, in general, may help you to build complex and scalable web applications with R.

If you already have a Shiny app, then Cole Brokamp’s package rize makes you just one function call away from building and viewing your dockerised Shiny application.

If you want to get serious with Shiny, take a look at ShinyProxy by Open Analytics.
ShinyProxy is a Java application (see GitHub) to deploy Shiny applications.
It creates a container with the Shiny app for each user to ensure scalability and isolation and has some other “enterprise” features.

Mark McCahill presented at an event of the Duke University in North Carolina (USA) how he provided 300+ students each with private RStudio Server instances.
In his presentation (PDF / MOV (398 MB)), he explains his RStudio farm in detail.

The platform R-hub helps R developers with solving package issues prior to submitting them to CRAN.
In particular, it provides services that build packages on all CRAN-supported platforms and checks them against the latest R release.
The services utilise backends that perform regular R builds inside of Docker containers.
Read the project proposal for details.

The package plumber (website, repository) allows creating web services/HTTP APIs in pure R.
The maintainer provides a ready to use Docker image trestletech/plumber to run/host these applications with excellent documentation including topics such as multiple images under one port and load balancing.

The call for abstracts is now open. The abstract submission deadline is 11 Jan 2017, 13:00 CET. So there is plenty of time to contribute, prepare an abstract and share your experience of reproducible research.

This session will showcase papers that focus on big data analysis and take reproducibility and openness into account. It is open to members of all programme groups and scientific disciplines to present how they conduct data-based research in a reproducible way. They are welcome to share practical advice, lessons learned, practical challenges of reproducibility, and report on the application of tools and software that support computational reproducibility.

The session is co-organized as part of the Interdisplinary Event “Big Data in the Geosciences” (IE 3.3), and the division on Earth & Space Science Informatics (ESSI ESSI4.11). “Using computers” is the unifying feature of many a researcher in the scientific divisions, so we look forward to meet a diverse group of people next year in Vienna. In the session description the conveners point out that…

[c]omputational reproducibility is especially important in the context of big data. Readers of articles must be able to trust the applied methods and computations because [..] data are also unique, observed by a single entity, or synthetic and simulated. Contributions based on small datasets are of special interest to demonstrate the variety in big data. Topics may include, but are not limited to, reproducibility reports and packages for previously published computational research, practical evaluations of reproducibility solutions for a specific research use case, best practices towards reproducibility in a specific domain such as publishing guidelines for data and code, or experiences from teaching methods for computational reproducibility.

The Open Access movement has improved the foundation for research reproducibility in that it has greatly advanced the accessibility of research data and text. This year’s theme for the International Open Access Week is “Open in Action”. The o2r team joins in by creating local awareness for what may come beyond Open Access.

To transform access into action, the o2r team is working towards the implementation of a simple technical solution. A “one click reproduce” button is one of the extremes within the continuum of reproducibility.
It enables the user to recreate the original results of a study with only a mouse click. In order to realize that, a new format for the publication of research findings has to be created and integrated into the publication cycle.

In o2r we envision a container format that implements the executable research compendium (ERC) to encapsulate any information relevant to constitutating a complete set of research data, code, text and UI. This includes any necessary specification of the working and run time environments.

Towards the other end of the continuum of reproducibility we find examples of published code and data that are openly accessible and yet fail to be rebuild easily by another scholar. By being dependent on other software, vanished packages and specific versions or environments, such cases leave it to the user to reconstruct the individual computational dependency architectures.
This strongly increases the efforts to rebuild, run, or compile the code and thus effectively blocks Open Action.

With the use of ERCs such obstacles can be resolved: The orginial analysis underlying a scientific publication becomes fully reproducible for independent researchers and anyone interested.
Opening reproducibility is where we see the biggest need for Open Action in science.

The workshop was a great event with contributions from very diverse backgrounds, ranging from computer science to library technology, and use cases, from big data to metadata interoperability or microscopic experiments.

The presentations were followed by a general discussion session, which touched, for example, the topics of publication bias/negative results, education having a higher potential than yet another infrastructure (“software data carpentry works”, says Carole Goble), and the necessity to communicate better about reproducible research. The latter lead to the idea of “five stars of reproducibility” inspired by the tremendously useful 5 ★ Open Data and also the FAIR principles .

It was great for us to share our ideas of an Executable Research Compendium with the workshop attendees. The discussions and feedback was very helpful. We especially realized that we need to sharpen the distinctive aspects of our project when we talk about it. We’re now working hard to implement this in a paper draft we’re working on.

o2r team member Daniel Nüst recently participated in the worlds largest conference for geospatial open source software. The FOSS4G 2016 was hosted by the Open Source Geospatial Foundation (OSGeo) and took place close to home, namely in Bonn. Therefore Daniel was extremely happy that his talk “An overview of Docker images for geospatial applications” was voted to be presented by the OSGeo community. Daniel presented an evaluation into the existing containers for FOSS4G software. After an introduction into Docker and some live demos, the takeaway was that everybody should use Docker more, and many different application scenarios (development, demos, training, cloud deployment) exist.

The presentation was very well attended (~ 120 people), albeit taking place in the first session on Friday morning after the conference dinner the night before. Reactions on Twitter were also quite positive, several good questions were asked, and great discussions followed throughout the day.

The main part of the work is published in the OSGeo wiki: a comprehensive list of Docker containers published by projects or third parties to use a large variety of tools, libraries, or Desktop applications in Docker containers. Check out the list at https://wiki.osgeo.org/wiki/DockerImages. Contributions are welcome!

How is this related to the o2r project? The expertise build up around Docker should be shared with the communities we know. And more concretely, many applications in the geospatial world are build upon services and APIs, so scientific work building upon these APIs will require to archive such services, too. This is a topic we will experiment on in the second year of o2r.

As some popular projects surprisingly did not have Docker images yet, Daniel started a new independent project on GitHub to provide a place for FOSS4G-related containers and to expand the knowledge and application of containers for geospatial applications: geocontainers. Inspired by Biodocker, geocontainers is intended to be a place to experiment and collaborate on containers without any initial rules or guidelines.

The conference was excellently organized in a great venue which includes the former Plenary Chambers of the Bundestag. Indeed a very special place to meet the people behind the projects of Free and Open Source Software for Geospatial.

With the University in a rather calm state during summer, the o2r team continues to work on the first prototypes for testing and demonstrating our ideas. This is the first post on a technical topic, and we will occasionally write about topics that are not related to the scientific work but either kept us busy for some time or might be useful to others.

Last week o2r team member Jan struggled with the implementation of the login feature for a Node.jsmicroservice. Why would we bother with that?
Because we want to share our prototypes publicly and invite you to try them out, but at the same time not have to worry about one of your most valuable possessions: your password.

Therefore we decided early on to rely on three legged OAuth 2.0 for handling user authentication. We opted for ORCID as the authorization server because it is the most widespread identification for researchers today1, and because of the potential for useful integrations in the future2.

So in the end, a full day of work to figure out two missing lines of code, but still many days saved on bullet-proofing standalone authentication and password storage.

The used libraries would allow us to quickly add more authorization services, such as Google or GitHub. ↩

Wouldn’t you like to have a research container be automatically added to your publication list? ↩

In a nutshell, the passReqToCallback option must be enabled when creating the OAuth4Strategy and the used callback function must include 6 arguments. Only then the function with the largest number of arguments is used and the content of the accessToken-request answer, which includes the ORCID id and user name, is accessible in your own code. They can be found in the params parameter of the function, not as part of profile as one is used to with other OAuth servers. This seems to be a slight deviation from the standard by the ORCID folks. ↩

A couple of weeks ago the o2r team met with a group of experts to discuss the project’s outline and scope. Being a few months into the project, the team members were eager to get feedback on their plans, which they created based on the original project proposal, the first practical evaluations, and extensive reviews of research literature. To give this feedback, we invited a group of external partners to a full day meeting at the Wersehaus, a small boathouse in the countryside next to the Werse river.

We started the day with talks by the external partners and project grantees. With such a select group, we were not surprised to get an excellent discussion rolling from the first talk on! You can download the talks’ slides below if available, just click on the person’s name.

Edzer set the context of the project and took a look back at the motivation for the project (among which is personal annoyance!)

Werner discussed the products of research (hypotheses, software, data, narratives) and provided some claims on these that fueled a lively discussion.

Tomi approached reproducibility from the question “How science works?” and connected it to his work on Linked Open Science (see also the original Prezi).

Holger introduced the interests and role of the university library in the project.

Xenia presented different levels of open access publication workflows and shared experiences from the publication domain and enforcement of openness.

Hylke talked about content innovation’s relation to reproducibility and showed a variety of work around interactivity in the article of the future.

Chris related reproducible research to ifgi’s vision of an open geoinformatics platform, and critically discussed benefits and challenges.

We continued the day with intensive discussions on the project’s schedule for the first year, stretching across all aspects such as preservation metadata, usability and user interaction, and compendium specification. After lunch these areas were explored more deeply in an Open Space setting prepared by the projects full-time employees Marc, Markus, and Daniel. Afterwards we drilled deeper to identify potentials risks and their mitigations, as well as answering the crucial question: Where can the project have the largest impact?

We found that a narrow focus is crucial for the project to succeed. Since we’re not going to change the publishing landscape in one step and we want to make an impact in the community we know best, geoinformatics, we see these high priority goals for the foreseeable project’s future:

New means of interaction with and exploration of scientific spatio-temporal data, analyses, and visualisations based on linked research compendia contents.

Automatic (bordering on magical) creation of executable research compendia based on typical science workspaces for R-based geosciences.

Specification of an executable research compendium rooted firmly in users’ requirements, preservation requirements, the currently dominating procedures in scientific publications, and reality of highly diverse scientific workflows.

New ways for searching scientific work based on the integrated and linked parts of a research compendium (text, code, data, user interface bindings).

So what will we understand in two years time that we do not know now?

We have a good understanding of how far the process of creating research compendia can be automated, and what efforts remain for authors or preservationists that must be counterbalanced with incentives.

We know the potential of user interface bindings as the connecting entity of research compendia.

We show the improvements in discovery and understanding of research when all aspects of research are explicitly linked in a meaningful way.

We get to know the common language as well as points of contact for the involved parties as we create a closer connection between research, preservation, and publication communities.

What do you think? Ambitious goals, or nothing new? Give the new discussion feature below this post a try!

We thank again our guests for their valuable inputs. Having their backgrounds in research as well as scientific publishing, their critical evaluation helps us to shape a clear direction for our work. To keep in touch, follow us on Twitter or GitHub.

In the last two days o2r team member Daniel participated in a workshop organized by the project “Data and Software Preservation Open Science” (DASPOS) at the University of Notre Dame, USA.
It was organized in an excellent fashion and in perfect amenities by the Center for Research Computing (CRC at the University of Notre Dame.

The workshop title “Container Strategies for Data & Software Preservation that Promote Open Science” fits perfectly with our own project goals, so Daniel was not surprised learn about a lot of great initiatives and projects.
A great group of researchers and librarians, mostly from the US, presented diverse topics almost all of which had one connection or another with o2r.
The general connecting feature between all participants were (Docker) containers and the interest in preservation.
Different approaches were presented in hands-on sessions, for example ReproZip, Umbrella and the NDS Dashboard.
You can check the full schedule and participant list for details.

A few highlights from Daniel’s perspective were the shared understanding that a common language and terms would be needed going forward when containerisation is applied more widely for openness and transparency of research.
But at the same time, and certainly at the current point in time of implementations, diversity is good and some amount of re-doing existing “features” is unavoidable.

Daniel would like to thank the DASPOS team for the invitation and the excellent filming and transcribing.
It was a great experience to meet the leaders in the field.
The workshop was an awesome opportunity to share the o2r vision and to learn about other projects, ideas and concepts at this stage of our project.

o2r team members Edzer Pebesma and Daniel Nüst published a short blog article on r-spatial about the project in general, and more specifically about the poster presented at EGU General Assembly a couple of weeks ago.

Next week the largest European geosciences conference of the year will take place in Vienna: the European Geophysical Union General Assembly 2016. It takes place in the Austria Center Vienna for a full week and expects to welcome over thirteen thousand scientists from all over the world. A vast variety of research across all disciplines of the Earth, planetary and space sciences will be presented in a meeting programme featuring workshops, lectures, talks, and posters.

Welcome to the new website of the research project Opening Reproducible Research.

You can learn the basics of the project and get to know the participants on the About page.

In short, we will develop new methods to make geosciences research reproducible. We will create open source tools and standards to compile text, data, and code (both sources and binary executables) into research compendia. These compendia will be easy to create for non-developers, executable in a web-based infrastructure, and allow exchanging of data and methods between compatible compendia.

Website pages

About

Goals

Open access is not only a form of publishing such that research papers become available to the large public free of charge, it also refers to a trend in science that the act of doing research becomes more open and transparent when it comes to data and methods.
Increasingly, scientific results are generated by numerical manipulation of data that were already collected, and may involve simulation experiments that are entirely carried out computationally.
Reproducibility of research findings, the ability to repeat experimental procedures and confirm previously found results, is at the heart of the scientific method.

As opposed to the collection of experimental data in labs or nature, computational experiments lend themselves very well for reproduction.
Some of the reasons why scientists do not publish data and computational procedures that allow reproduction will be hard to change, e.g. privacy concerns in the data, fear for embarrassment or of losing a competitive advantage.
Others reasons however involve technical aspects, and include the lack of standard procedures to publish such information and the lack of benefits after publishing them.
We aim to resolve these two technical aspects.

We propose a system that supports the evolution of scientific publications from static papers into dynamic, executable research documents and aim for the main aspects of open access: improving the exchange of, facilitating productive access to, and simplifying reuse of research results that are published over the internet.

Building on existing open standards and software, this project develops standards and tools for executable research documents, and will demonstrate and evaluate these, initially focusing on the geosciences domains.
Building on recent advances in mainstream IT, o2r envisions a new architecture for storing, executing and interacting with the original analysis environment alongside the corresponding research data and manuscript.
o2r bridges the gaps between long-term archiving, practical geoscientific research, and publication media.
The o2r team collaborates with publishers to achieve the following goals:

Open Source

For Open Science and reproducible research, we see not alternative to Open Source software.
All scripts, code, and libraries supporting a computational analysis must be open for scrutiny by fellow scientists.
We publish current code online, instead of holding back until publication of a paper, to profit from interaction with the Free and Open Source Software (FOSS) community.
Even the software of supported workflows (i.e. R) and underlying technologies (i.e. Docker) are published under FOSS principles.
Already in the project proposal, we set a clear agenda on the question of software licenses:

All software developed by project staff will be distributed under a permissive open source license that allows reuse, modification and integration in commercial systems (e.g., Apache 2.0). Development happens openly at GitHub and all developments are visible directly instead of after the end of the project.

See our results page for more information about all software projects.

People

o2r team members, supporting university staff, and external advisory board members in alphabetical order.

External partners

The o2r project is connected to external partners since its inception, and the group has been extended since then.
They come from different disciplines and provide valuable feedback on project plans and decisions.