Skip to contents

CollaboratoR Workflow

collaborator_workflow.Rmd

Overview

As stated on the project main README, this package is to assist data synthesis research by automating data validation and aggregation processes for researchers with different skill sets. to use a set of CSV formatted data file, or optionally employ user-friendly Google Sheets, as a collaborative data and meta- data entry tool. It features automated validation of the data using simple data definition table and easy to write standardized rules, and finally the workflow tracks changes to data using git.

Components

  • (optionally) Google Sheets to entry of data and meta-data
    • other systems could be used with manual export of CSVs, this package has capability of automatically pulling from Google Sheets
  • meta-data and data store in common tabular data formats
    • CSV is basis for using version control
    • this package enables use of user-friendly Google Sheets for editing these
  • R Readr package for data column name and data type validation
  • R Validate package for data content validation
  • git to track changes to data and meta-data via CSV text file
  • (optionally) Git service to share results via the web

Setup/Configuration

  • Install this package from github using renv as renv::install('IBEEM-MSU/CollaboratoR')
  • Information on how to set-up google workspace in the vignette for Google Sheets API setup.
  • To use git features for sharing, the package can work with github (others such as gitlab may be added later) which requires a github account.
  • This package relies on configuration values in the .Renviron file for connecting automatically to a github repository.
  • See also the CollaboratoR Example for ideas on organizing your git repository for this workflow.

Workflow Summary

  1. Write your data entry protocol (outside of this system but crucial for organization)
  2. Setup Git repositories to hold data as it flows through
  3. Create metadata and configuration
  • column definitions for main data sheet (CSV or Google sheets)
  • create validation rules (YAML format)
  • data lists such as species lists in CSV or
  • list/table for tracking data entry sheets
  • CollaboratoR configuration in .Renviron
  1. Enter data into tabular form
  • CSV or Google Sheets
  • add entry to list of URIs (Google sheet URLs or CSV file paths)
  1. Use CollaboratoR package to accumulate metadata
  • read metadata: column specification, list of data sheet URIs
  • reads in list of data sheets CSV filename or list of googlesheets URLs
  • validation rules written in YAML and stored a RData file
  1. automated validation and report
  • read data files (from list of URIs in CSV or Google Sheets)
  • data format against schema (from column spec)
  • category columns are checked against lists (i.e. species lists)
  • data values against rules (from YAML file)
  • report errors printed for user

  1. update local repository with git and upload changes to shared repository

  2. aggregate

Workflow

Write your data entry protocol

This is beyond the scope of this documentation and package, but crucial for successful data synthesis projects. The process for finding, extracting and formatting every data point is crucial. This protocol will the determine the columns in the datsheet and column definitions in metadata

Create metadata and configuration

Create a table describing the columns in the each to data files based on the protocol. This table must have columns as follows:

Column Description
col_name exact name of column as it will appear in the datasheet
col_description human reable description based on your data collection protocol
col_type name of the datatype, see datatype list below
example human readable example for guidance
  • create validation rules (YAML format)
  • data lists such as species lists in CSV or
  • list/table for tracking data entry sheets
  • CollaboratoR configuration in .Renviron

Enter data into tabular form:

  • CSV or GoogleSheets
  • add entry to list of URIs (Google sheet urls or CSV file paths)

Use CollaboratoR package to accumulate metadata

  • read metadata: column specification, list of data sheet URIs
  • reads in list of data sheets CSV filename or list of googlesheets URLs
  • validation rules written in YAML and stored a RData file

6. Automated validation and report

Classes of validation:

  1. structural, based on the “data dictionary” or field specifications created above. The package ensure the data sheets have the same column names and data types per column.

  2. Constraints or Rules created using Validates format

Order of automated validation:

  1. check the data format against the column specification
  2. category columns are checked against lists (i.e. species lists)
  3. data values validated against constraints using validate::confront(), accumulating validation messages

Reporting

A report of all errors is provided to the user. If there validation errors at any point, the next step is not automatically initiated. However the user can run inititate the git commit anyway for those sheets that have passed.

7. Update data repository with git

This workflow uses git, which tracks changes when files are changed.
Changed CSVs are ‘check-in’ or ‘committed’ to the repository with git.
A git ‘commit’ is a set of changes across included files, a message, and a unique id number. Team members can review the commit messages and changes to see what changed between updates. It’s not necessary to put a list of files changed in the message since that can be examined.

There are two ways to update the validated data set.

auto save valid data

If all validation checks pass, the default action is to ‘check in’ ( also known as “commit”) all csvs in the set to the git repository. This updates the local repository on the computer in which it’s run.

user initiated save of validated sheets

When there are validation errors, the user much initiate a check-in using a simple function . Those sheets that passed validation are “committed” with a message

manual commits.

Because this system is a collection of “helpers,” a user who knows how to use git may commit one or more files at any time simple using git commands or using the features of Rstudio/Positron to selectively commit sheets. One should always add a message indicating why the commit was made

It’s very important that no invalid data sheets be added to L0 as CSV Files and checked into git manually.

If so then data aggregation or common analysis will fail

Upload or “push” using git

To share data, the workflow is to “push” to remotes. This can be done automatically via the pipeline or manually using git commands. In either case the connection to the git repository must be configured correctly. Detailed instructions for using git are beyond the scope of these instructions.

For automatic pushing, some configuration is requires

8. Aggregation

Data rows should be assigned a category, and/or a unique ID value that allows them to be aggregated and still be identified.

The aggregate function peforms simple data frame combination, but then if a “primary_e