Skip to content
cpfaff edited this page Jul 6, 2020 · 42 revisions

Welcome to the BEFdata documentation wiki

BEFdata is a web based research database and data management system for biodiversity related research consorita. The software makes it easy to store, harmonize and share closely realted data amongst scientists. It also comes with a built in integration into the R statistical environment. The R package allows to exchange data and metadat with the portal software. This enables a project to access data hosted online, to share, document, and analyse the data. In this wiki you find all information required to setup and use BEFdata.

Setup

The following sections about the setup will detail how to get a BEFdata instance up and running. It is divided between a description for development environments installed on a local computer and production systems serving the application to the public via a web-server.

Development

This sections describes how to set up a development environment for BEFdata on a Linux machine.

  1. Prerequisites

Install the following on your system.

  • Ruby 2.3.8 (You can use a ruby version manager e.g. rvm)
  • PostgreSql (version <= 9.6.15)
  • ImageMagick
  • Node.js
  1. Download the source code
  • You can e.g. use the git clone command or the download button to retrieve the source of the project

You can find the latest stable version on the branch with the greatest version number (e.g. v1.6.3) After you cloned the project you should checkout that branch (e.g. git checkout v1.6.3).

  1. Install bundler

When you have setup ruby you can install bundler gem. It acts as a package manger for the rails application.

gem install bundler

Open an console in the root folder of the BEF-Data application. Then you can issue the command below which will install all required libraries for BEFdata to run.

bundle install
  1. Configure the software

You can use the template configuration files in the project config/ folder. The template files are all named with the appendix .dist. When you remove the .dist from the name of the file, the application will pick up their contents on start and use it for it for configuration. You need to rename these:

configuration.yml.dist -> configuration.yml
database.yml.dist -> database.yml
application.yml.dist -> appication.yml
mailer.yml.dist -> mailer.yml
secrets.yml.dist -> secrets.yml

Afterwards you need to open them and make your changes to the configuration according to your needs. The files are commented pretty well.

  • Create a secret key for the default local environment (development)
bundle exec rake secret

Copy and paste it into the configuration file in the appropriate section (development) in the secrets file.

config/secrets.yml

  • Set up the database

We are using postgres as database backend. First you need to configure your local postgres installation. You should change the password of the root user in postgres which is named postgres. When you have done this create a new user for your BEFadta instance and a database you want to use in our application. Make the new user the owner of the database.

  • Connect to the database

In the configuration file config/database.yml you can set the host the database the username and password for different environments (production, test, development). As we are running this on a local computer we, for now you use the development settings. Add all information to the section from the your postgres database setup.

  1. Configure ImageMagick

The ImageMagick convert command is used to create thumbnails of the user uploaded avatar images in conjunction with the paperclip gem. Thus convert needs to be accessible by paperclip. You have to set the path to the executable in the configuration file of the environment you are running BEFdata in. For example in production config/environments/development.rb.

# setup paperclip
Paperclip.options[:command_path] = '/usr/bin'
Paperclip.options[:log] = false
  1. Initialize the database

Now you are ready to initialize the database. For this you can use the commands below.

# setup the database
bundle exec rake db:setup

# fill the database with default values
bundle exec rake db:seed
  1. Run a local server

Afterwards you can point your favorite browser to following local address localhost:3000. There you should find your own BEFdata instance running. You can login with the username "admin" and the password "test". You should change that password in the profile of that user (see first login below).

bundle exec rails server

Production

  • ... (will come later)

Main navigation and info bar

On the top of BEFdata you find the main navigation. Depending on your role in the project different entries will be visible to you. On the right hand side next to the main navigation is the information bar. It will inform you about requests to your data and about changes of data from others that you have used in your projects. It also contains a link to you personal profile and a reference to the cart (see create a paper proposal)

Side navigation or action bar

On the right hand side of BEFdata you find the action bar. Depending on your role in the project and the context (e.g. dataset, projects) different entries will be visible here to you. They are separated into different sections e.g. like information, tools or edit.

First login

BEFdata comes with a predefined administrator account. After setup you can use this login and the default password (admin/test). It is advised to change the password of the administrator account right on the first login. You can do this via the user profile page.

Default:

  • username: admin
  • password: test

Change:

  • Click > User Icon (in the top right)

  • Click > Metadata Icon (in the right bar under the Pen)

  • Edit the User (scroll down to login information)

  • Save User

Projects

As an admin you can not only maintain the metadata of existing projects. You can also create new ones. To craete a new project first click on projects in the main navigation.

In the actionbar on the right hand side you have to click on the plus symbol.

Then you get a form where you can fill out all information about the new project. You can also associate people to the project here. Attachments like project proposals can be attached later (see below).

If you view a project page you find an overview of various tabs with different content like the project metadata, associated users, datasets, and project internal proposals and finally attachments. If you click on the paper clip symbol you can add a new attachment. Each attachment has its own actions to allow editing or deleting them.

Click on the paperclip:

A form opens where you can select a file and add some description.

Finally the new attachment is listed in the overview of attachments in the project. You can download, edit and delete each attachment using the buttons in the footer of each listed item.

Users

  • Create a new user

At the moment BEFdata does not offer user self registration. Administrators have to create accounts for other users in the project. The steps below create a new user:

Click on staff:

You get the overview of all staff. On the right hand side you find the action bar. Here you find a plus button which you can click to add a user.

Fill out the form for the user.

Select Roles:

BEFdata has a role system to adjust rights of logged in users. Depending on the role of the user the interface will offer more or less options (e.g. managing datasets of others and voting for new projects). The currently available roles are listed described below:

Admin

The administrator is the super-user account and has the right to do anything from adding and manging users, managing projects and datasets up to project board related tasks like voting for new projects. He can also take actions on behalf of other users.

Data Admin

The data administrator is allowed to manage the data of all users. He can add new datasets, mange the data and the metadata.

Project Board

Members of the project board take a leading role guiding the overall project. When a new proposal is created from a user in the project, they have have the first vote on if this proposal is going to be accepted or not (or under which conditions, see also the paper proposals section).

Alumni

They have no special rights. This role basically allows to keep old project members in for reference purposes.

The roles can be adjusted by an administrator in the profile page for each user.

  • User profile page

You reach you user profile by clicking on the small user icon on the right hand side of the main navigation.

If you view a user page you find an overview of various tabs with different content like projects, datasets, and project internal proposals. You also find your personal credentials here. These are are important as you need them when you want to access your data with the R package (see using rBEFdata).

Add Data

Datasets can be added in several ways. BEFdata accepts data in Excel (2013) and CSV format. For the best results and convenience It provides a predefined spreadsheet which stores metadata and data of a single research project in one place. An template can be downloaded from BEFdata and you can use it for your own data and metadata!

  • Get a template Spreadsheet:

First click on datasets in the main navigation. On the right hand side you find the action bar. In there you see a download section with a button that lets you download a template workbook.

The workbook contains 5 sheets. The fields in the spreadsheet come with help text associated so to guide you through the documentationh process.

The first sheet contains general metadata. Here you can fill in a title People, study design and time related information. Always fill your information into the black boxed fields.

The second contains acknowledgements. Here you can mention people that helped you with part of your data collection. For example if you had help identifying species you can mention the people and reference them to the according part of the data by providing the column header from your raw data.

The third sheet contains a detailed description for the columns in your raw data sheet. You can fill out e.g. units, keywords or data groups (More on this later), etc.

The fourths sheet contains metadata about the categorical columns in your dataset. You can store used abbreviations, thier long forms and category descriptions. This is useful to make your categories meaning transparent.

The last sheet in the workbook is an empty one which you can fill with your raw data. The first row stores the header.

  • Upload

When you have finished filling the workook you can upload and import it into the database. Therefore select youf file for upload. Click on "create" and wait.

Access level Description
Visible for public Dataset is visible for visitors to the site. This is the default setting.
Free for public Dataset is open access, visitors to the site can download.
Free for members Members of the same project a dataset belongs to can download.
Free within projects Members of the collaboration can download, thus, all authenticated users with login access.

Validate Data

After uploading data there often remain invalid values. Each data column is validated against the data group it was assigned to, depending on its Data type.

Type Explanation
text is not validated but saved as is.
year is first checked for Categories and then saved as date. Invalid values have to be validated and/or saved as categories.
date is first checked for Categories and then saved as date. Invalid values have to be validated and/or saved as categories.
number is first checked for Categories and then saved as number. Invalid values have to be validated and/or saved as categories.
category is checked for Categories and then saved as number. Invalid values have to be validated and/or saved as categories.

Each data group stores it's own controlled vocabulary, the Categories. If during validation there are categories detected, that are new to the data group, then these values are flagged as invalid and are presented to the user. You can either correct the values or accept them as correct.

Below there are the steps to follow when approving invalid categories:

  • Single Columns

    • first open the dataset you want to approve
    • click on the data tab
    • look for the column you want to approve
    • click on the approve link in the footer of the column (hand symbol) or on the approve button in the action bar on the right hand side
    • Go through the steps of the wizard and approve the metadata and data
  • Quick approval

    • first open the dataset you want to approve
    • click on approve button in the action bar on the right hand side
    • click on quick approve button in the action bar on the right hand side
    • Quick approval only allows to change datagroup and datatype

Create paper proposal

Each user can set up a paper proposal. You can create a proposal based on exising datasets or without data in case the data is not yet collected in the project.

Step-by-step:

  1. a) With existing data: First select the datasets you want to use in the proposal. You can click on the cart icons on datasets to do this. Afterwards click on the cart icon in the top right corner (it should show the count of dataset currently selected). There you can click on create proposal from cart in the right hand actionbar. Adapt all metadata and save the proposal.

    b) With non existing data: Click on papers. On the index page in the actionbar you find a plus button for adding a proposal. Fill out the metadata and save it. Add information about which data you might want to collect and send the proposal to the project board without data. Adapt all metadata and save the proposal.

Data requests

If you create a paper proposal (see create a paper proposal) you can request access to data from other users on the portal. The data owner receives the request for data access wich can be seen on the top right corner next to the main navigation. The icon for data requests will be highlighted in green color and show a the count of new request for you to check. If you click on it you will get an overview.

In that overview you find all notifications on dataset requests which requset your data. You can vote on them and decide to grant or reject access to your data and comment e.g. to request changes on the proposal. You can also use the opportunity and read into the proposal in more detail, get in contact with the requester(s) to discuss about the conditions of a new collaboration.

Members of the project board also find board related notifications in that overview. They will be asked about every paper proposal first and have the responsibility to decide if a new research proposal will be granted or not.

Further you also find the votes history there. You can click on the small eye symbol on the of the items in your notifications to mark the messages as read or unread. If it is read it will be moved to the history for reference.

Notifications

BEFdata offers a notification system which is triggered on changes of datasets. The dataset owner or administrators can send a message to inform people which have downloaded the data or used it in a proposal before.

To send a notification to other people, go to the page of the dataset that you have changed. Click on the "Changes" tab. There the the system prepares a new message on each registered change on the data. You can click on the small glider symbol on the generated message.

You can edit the message in the opening dialogue to your liking. Be as precise as possible to make clear what you have changed so the recipients know how to act accordingly (e.g. adapt their analyses, scripts etc.). Finally you can decide who will get informed and you can send the message.

If you receive such a message from other users, you find it in the top right corner next to the main navigation. The icon for notifications will be highlighted in green color and show a count of new notifications for you to check. You can click on the icon and get a detailed overview of the messages to read. Finally you can mark them as read.

Manage keywords (admin)

Keywords can be used to tag your datasets. You can add them editing the metadata of your datasets on the portal. Another option is to enter the keywords right in the workbook (see add data section). The keywords can help to search your data in the portal. They are offered as filter in the search interface of the data in the portal.

As everybody is allowed to add keywords a diverse pool of keywords can grow over time. It is a good idea to maintain that pool to increase its utility. Navigate to the keywords tab in your main navigation. You can search for keywords and sort them here. You can select keywords by clicking on them. They will get highlighted in green if selected. Depending on the count of selected keywords, different tools from the action bar on the right hand side become available. You can rename (min 1 selected), merge (min 2 selected) and delete keywords here (min 1 selected).

Manage datagroups and their categories (Admin)

Datagroups are an essential tool in BEFdata as they allow linking similar content across the whole project database. For categorical columns the pool is like a collection of known or valid values to a category (e.g. names of research plots). When you link a column of a new dataset to an existing pool, the categories in your dataset are compared against the known values. Unknown values are presented to you during the validation step (see validate data) so you can accept them they are or correct them if necessary.

Use it with R

You can use the R statistics package rBEFdata to download and analyse your data.

  • Install the package
  • You need your credentials (in your profile)