Loading FACS Data

Data loaders read data of a certain format and send the Scifeon entity equivalent to the database. The ability to write custom data loaders is a powerful tool for enhancing the Scifeon data management experience.

In this example we will dive into the workings of a specific data loader from the FACS app. This data loader is able to read .csv files produced by cytometer software and convert it into entities fitting the FACS template in Scifeon.

The loader in itself does not read the actual files. This is delegated to the ReadPopulationsDiva classes. instead, this reader focuses on managing and validating file metadata.

The Scifeon view uses a set of predefined decorators and functions to help integrate the data loader into the view:

• @scifeonPlugin: Sets parameters for the data loader
• init: Run on page load
• readFiles: Run on data load
• getResults: Bound to confirmation button
• entitiesView and optionsView: HTML function shown on data load

@scifeonPlugin

Like other data loaders in Scifeon, the data loader class requires a plugin decorator:

@scifeonPlugin({
    name: 'FACS-Loader',
    description: 'Load FACS populations data from CSV',
    type: PLUGIN_TYPE.DATA_LOADER,
    match: (context: FileContext) => {
        if (!context.experiment 
            || context.experiment.type !== FACS.ExperimentType 
            || (context.fileInfos.length && !context.fileInfos.every(fi => !!AppFlowCytDataLoaderDivaCSV.matchFilename.exec(fi.filename))))
            return false;
        return true;
    }
})

private static matchFilename = /^(\d{8})_([^_]+)_([^_]+)_([^_]+)(_[^\.]*)?\.(csv|pdf)$/i;

The decorator marks the following class as a Scifeon plugin of the type described in the type variable, in this case, a data loader. The decorator also contains information on what the class contains such as a name and a description. This is mostly useful for other developers as they only show up in the console.

An important aspect of the decorator is the match variable. This variable should be a function returning a boolean (either true or false). This function decides whether or not the plugin is shown on a given page in Scifeon. The context variable contains information on the shown page. For this data loader, it will only return true if the current page is an entity page for a FACS experiment. Moreover, one or several files should be present for upload with names matching the regex expression in the matchFilename variable.

init

The init function is called when the view is initialized. This makes the function handy for functionality that you want to execute straight away. In this case, the init function is used to validate the file meta data:

The matchFilename variable catches 5 groups separated by underscores. The first group has to be numbers ((\d{8})) while the rest of the groups can consist of any characters except underscores. The (_[^\.]*)? is the fifth group which catches all remaining characters, including underscores. The file should then end in either ".pdf" or ".csv".

The groups contain metadata about the dataset: The first group is the data, second is the experiment name, third is scientist initials, fourth is datatype, and the fifth contains secondary initials.

if (!fileInfo.plateName) {
    this.initErrors.push(fileInfo.filename + ':<br/>Please, make sure that the population filenames match the pattern YYYYMMDD_RunID_Inits_Count/InfU/Txg.csv');
}

The metadata is checked for validity: The experiment name and initials should match the current ones of the current experiment entity, and the date should match the specific YYYYMMDD format and the current date. If these do not fit, the upload is cancelled and an error message is displayed.

readFiles

The readFiles function is called if the init call was succesful. This function requires an attached file and is often used in data loaders. In this case, the function acquires data from the database and links it to the new data:

If the metadata is valid, the entity creation process will begin. First, the script will look for result sets already existing in the experiment, If not present, a new result set entity will be created.

Input samples, existing gating sets, and FACS templates will then be acquired from the database with the following datasetQuery call:

const dataset = await this.server.datasetQuery([
        { eClass: 'Sample', collection: 'samples', filters: [{ field: 'OriginID', value: stepInfoStaining.step.id }], sortings: [{ field: 'id' }] },
        { eClass: 'ResultFACSGatingSet', collection: 'gatingSets', filters: [{ field: 'ResultSetID', values: rsIDs }] },
        { eClass: 'Template', collection: 'templates', filters: [{ field: 'Type', values: FACS.AllTemplateTypes }] }
    ]);

Data from the files can then be extracted: For each file, plates (the physical unit containing the samples, represented by a plate entity in the database) with matching names are extracted from the experiment, and samples (from the dataset acquired previously) are matched.

We can then start extracting data from the files. The extraction happens through separate classes, ReadPopulationsDivaV1 and ReadPopulationsDivaV2. Which one is used depends on the data format. click the links to investigate further.

fi.plates = this.context.plates.filter(p => p.name === fi.plateName);
const samples = this.inputSamples.filter(s => fi.plates.find(p => p.id === s.plateID));

const resultV1 = new ReadPopulationsDivaV1().read(fi, samples);
const resultV2 = new ReadPopulationsDivaV2().read(fi, samples);

The resultV1 and resultV2 has the following structure:

{
    success: boolean,
    gatingSetInfos: [
        {
            exportDate: DateTime,
            counts: [any],
            populations: [any],
            tubes: [any]
        }
    ]
}

The [any] objects contain entities of the classes 'ResultFACSCount', 'ResultFACSPopulation' , 'ResultFACSTube' created from the data contained in the uploaded .csv file.

As a final step we compare the populations (p) from the upload file to the population template entities already in the database:

const name = p.name.replace('pos. cells', '').trim();

let tpt = this.templates.find(t => t.type === FACS.TemplateTypePopulation &&
(t.content.fileLabels.includes(name) || t.content.fileLabels.includes(p.name)));

If no template matches the populations, a new template is created and added to the growing list of entities to be added to the database.

getResults

The getResults function is called when the upload is confirmed. This function bundles all the entities in a list. This list is then sent to the database.

for (const fi of this.reportFileInfos) {
    entities.push({
        subjectID: this.selectedRSInfo.rs.id,
        subjectClass: 'ResultSet',
        eClass: 'File',
        resultSetID: this.selectedRSInfo.rs.id,
        name: fi.filename,
        content: btoa(fi.content),
        size: fi.file.size,
        mediaType: 'application/pdf',
        type: FACS.FileTypeDivaReport
    });-
}

The above shows the conversion of file info objects to a Scifeon file entity. This mapping happens to all the entities after which they are pushed to the entities list.¨ The entities list is then returned in an object:

return {
    entities: entities
}; 

This object is used to add the entities to the database.

entitiesView & optionsView

These functions are used to introduce visual elements directly from the view model. This can be used to allow the user additional information and/or options while using the app.

In the FACS app, this is used to give the user more control of the uploaded samples: specific populations can be deselected, and multiple sets can be merged.

1 is added by the optionsView function and allows the user to specify result-, and gatingSet.

2 is added by the entitiesView function and allows the user to view and de-/select populations before uploading. A code sample from this function can be viewed below:

entitiesView = () => `
<b>Populations:</b>
<table class="full-width" style="font-size:90%;">
    <thead>
        <tr>
            <th>Include?</th>
            <th>Lineage</th>
            <th>Name</th>
        </tr>
    </thead>
    ...