Loading Data

Data Loaders are used to load data from files into the Scifeon database. The data is often in a .csv file or similar.

For some data structures Scifeon is already capable of loading the data in. However, for structures not recognized by the default data loaders, a custom data loader has to be built.

Custom Data Loaders

To make an entity panel, a decorator is used. The decorator contains information about the name and type of the plugin.

import { PLUGIN_TYPE, scifeonPlugin } from '@scifeon/plugins';
@scifeonPlugin({
    name: "Fermentation-Online-Loader",
    description: "Load fermentation online data from CSV file",
    type: PLUGIN_TYPE.DATA_LOADER,
    match: (context) => 
        context.experiment &&
        context.experiment.type === 'Fermentation' &&
        context.fileInfos.length === 1 &&
        context.fileInfos[0].filename.toLowerCase().endsWith('.csv') &&
        context.fileInfos[0].filename.toLowerCase().startsWith('fermentation')
})

The code contains the decorator for the fermentation online data loader.

There are several different types of plugins. For custom data loaders we use the DATA_LOADER type.

The match method decides when the panel is shown. The context variable contains information about the current page. If the match method returns true, the data loader can be used with the dataset.

This data loader can only be used if the current page in Scifeon is a fermentation experiment, only 1 file is selected for upload, and the file name contains a "fermentation" prefix and a ".csv" suffix.

The data loader class consists of 2 methods: an init converting the data in the file to Scifeon entities, and a getResult method which simply returns the entities.

Creating Scifeon Entities

The init method is called once a data loader is chosen. This method reads the file and translates it into Scifeon entities.

Below is a code snippet from the fermentation data loader:

let content:any = { timeSeries: [] };

for(let iRow = 1; iRow < csv.data.length; iRow++) {
    let col = [csv.data[iRow][0]];
    for(let iCol = 1; iCol < csv.data[iRow].length; iCol++)
        col.push(Number.parseFloat(csv.data[iRow][iCol]));
    content.timeSeries.push(col);
}

Fermentation experiments create huge amounts of data. To save this efficiently in the database, each row of the data matrix is saved in the timeSeries list, which is then converted to a text string.

This is then added to the entity object:

this.entities = [
    {
        eClass: 'ResultFermOnline',
        resultSetID: '#rsID',
        type: 'FermOnline',
        content: content,
        fermentationID: ds.f.id
    }
];

When creating entities, some properties such as eClass and ID are mandatory while properties not present in the data model cannot be created. For a full list of properties, see the data model.

Returning The Entities

The getResult method is used by the view to get the entities meant for upload.

getResult() {
    return { entities: this.entities };
}