The qpcr.Parsers

This module contains classes designed to work with irregularly structured datafiles. It provides Parsers that are able to extract the replicate identifiers and Ct values as pandas DataFrames from irregular csv and excel files.

In fact, qpcr.Parsers are already implemented in the qpcr.Readers so you will often be able to read irregular datafiles directly with one of the qpcr.Readers and will not have to manually work with the qpcr.Parsers.

Working with “irregular files”

Any datafile that does not only consist of a replicate identifer and Ct column is called “irregular”. In fact, most excel sheets or csv exports from qPCR machines are actually irregular as they often contain some information about the run, and melting curve data, and so forth. Such data is not relevant or of interest to qpcr, however, so we have to extract the data we are intersted in. This is the job of the qpcr.Parsers. They read in an irregular datafile and use a guided-parsing approach to find the relevant sections within the datafile. If your datafiles contain multiple datasets / assays, the qpcr.Parsers will be able to extract all of them and store their extracted data.

There are essentially two ways how they can do this, which are explained below.

“Finding” relevant datasets through `assay_patterns`

The Parsers are quipped with a method called find_assays which locates assays (or more formally “datasets”) within the datafile using regex. Of course, in order to do that they need to know the patterns they are supposed to look for. Some patterns are already pre-specified in the qpcr.Parsers.assay_patterns dictionary and can simply be specified using their key. If your own pattern is not yet pre-defined, post an issue on github and supply some samples of how your assays usually appear in your datafiles alongside with the name of the machine that produces your datafiles.

Of course, you can also manually specify your own regex pattern. The only constraint is that is must have one capturing group for the assay name.

Note

All assay headers must be located either in the same column or the same row to be identified by a Parser!

Once the assays in your datafile are identified, the data columns belonging to them are searched for. The constraint here is that they must start either in the row exactly below the assay headers or have at most one single row in between them. Anything else is no good! The data columns must be labelled (i.e. have a header). By default Name and Ct are assumed as data column labels / headers, but this can be changed.

Working with multi-assay files

While working with datafiles that contain multiple assays you will likely want to use all the assays from the datafile, here’s how the qpcr.Parsers help you do this.

Making indivdual assay files from a multi-assay file

This is the core-business of the qpcr.Parsers. So you can simply set up a Parser, set a saving location using the Parser’s save_to method and then pipe your file through. All done at this point. Of course, you can also work with the dataframes directly using the Parser’s get method. Like this you easily separate the assays which you can then pass to your main analysis as assays and normalisers.

Using a multi-assay file directly for my analysis

So, you want to just feed in your one datafile and expect to get a table with your Delta-Delta-Ct values for all assays against all normalisers?! Sure, no problem! Parsers will be able to do that, but you can more easily read a multi-assay file using the MultiReader which allows you (after setup) to simply pipe through your datafile and returns immediately a list of your assays-of-interest and normaliser-assays. How does it know which is which though? That is where the decorators come into play which you can learn more about down below.

Working with `qpcr.Parsers`

Because they are already implemented in the Readers it likely that you will never actually use the Parsers directly. However, working with the Parsers is virtually the same as working with the Readers.

myfile = "my_big_irregular_file.csv"

# setting up the parser
parser = CsvParser()

# we know that the assays / datasets are all named by a scheme
# we can define a regex pattern to match this
mypattern = "qPCR run: ([A-Za-z0-9-. ]+)"

# now we can pipe our file through the parser to get the dataframes of our assays
data = parser.pipe( myfile, assay_pattern = mypattern )

# at this point we have a dict of dataframes with their extracted patterns

Decorators

A decorator technically is a function that wraps another function when coding. Well, that’s not quite the case for the qpcr decorators but the idea is similar. Instead of wrapping functions the qpcr decorators wrap assays in a multi-assay file. What does “wrap” mean? It means they provide meta-data about the assay in question. What does that mean? There are multiple implemented qpcr decorators. For irregular multi-assay files the two important decorators are: @qpcr:assay and @qpcr:normaliser – is it now clear what they do? They are placed in the cell exactly above the cell where the assay header is located (seriously, anything else won’t do!) and tell the qpcr.Parsers (because all the MultiReader is doing is setting up some Parsers…) if a specific assay is an “assay-of-interest” or a “normaliser-assay”.

So, let us recap this quickly: qpcr.Parsers can identify assays either through de novo finding using regex patterns or through decorators. To tell a Parser to use a specific decorator for finding assays you can specify the decorator argument in pipe or parse (pipe wraps read+parse). To specify a decorator like this you only write qpcr:assay or qpcr:normaliser (the key is bascially the decorator but without the @)

If you work with qpcr.Parsers directly you can choose if you only wish to extract “assays-of-interest” (decorated as @qpcr:assay) or “normalisers”, or whatever other decorators are available. However, this flexibility is not available when calling Parsers indirectly through one of the qpcr.Readers.

Decorator	Code-reference	Filetype	Available for / Used by qpcr.Readers
any except qpcr:column	qpcr:all	Irregular single- or multi-assay files.	SingleReader, MultiReader, MultiSheetReader
@qpcr:assay	qpcr:assay	Irregular single- or multi-assay files.	SingleReader, MultiReader, MultiSheetReader
@qpcr:normaliser	qpcr:normaliser	Irregular single- or multi-assay files.	SingleReader, MultiReader, MultiSheetReader
@qpcr:group	qpcr:group	Horizontal Big Table files	BigTableReader
@qpcr	qpcr:column	Horizontal or vertical Big Table files	BigTableReader

Note

Just like assay headers, all decorators must be located either in the same column or the same row to be identified by a Parser!
If you are using excel you may have to add a single tick ' in front of your decorators.
When specifying a decorator any non-decorated assay will be ignored!

“Custom decorators”

You might be tempted to think that you could also specify your own “decorator pattern” together with your assay patterns. While this is not strictly a built-in feature, there is a way of making this work. The decorators are stored in a simple dictionary within the qpcr.Parsers submodule. Hence, you can add your own entries to this dictionary and then let them be accessed regularly through their keys by the Parsers.

from qpcr import Parsers

# the new decorator should be specific for a certain experiment
mydecorator = "(@qpcr:experiment1-assay|'@qpcr:experiment1-assay)"

# now add the decorator
Parsers.decorators[ "mydecorator" ] = mydecorator

# now the decorator will be available for the Parsers to work with...

parser = ExcelParser()
data = parser.pipe( myfile, decorator = "mydecorator", assay_pattern = mypattern )

class qpcr.Parsers.Parsers.ArrayParser[source]

Bases: qpcr.Parsers.Parsers._CORE_Parser

Handles only parsing of irregular files that contain multiple assays. However, it does not read any specific filetype but requires a numpy.ndarray as input for it’s read method.

pipe(data, **kwargs)[source]

Accepts a numpy array for its data source, and parses for assay datasets.

Parameters

data (np.ndarray) – A numpy array of some data to parse.
**kwargs – Any additional keyword argument that will be passed to any of the wrapped methods.

Returns

assays – A dictionary of all the extracted assays from the datafile storing the data as pandas DataFrames. Individual assays can also be accessed using the get method.

Return type

dict

read(data)[source]

Accepts a numpy array for its data source.

Parameters: data (np.ndarray) – A numpy array of some data to parse.

class qpcr.Parsers.Parsers.CsvParser[source]

Bases: qpcr.Parsers.Parsers._CORE_Parser

Handles reading and parsing irregular csv files that contain multiple assays. It extracts datasets either through regex pattern matching or/and through provided decorators within the datafile.

pipe(filename: str, **kwargs)[source]

A wrapper for read+parse

Note

This is the suggested use of CsvParser. If a directory has been specified into which the datafiles shall be saved, then saving will automatically be done.

Parameters

filename (str) – A filepath to an input csv file.
**kwargs – Any additional keyword argument that will be passed to any of the wrapped methods.

Returns

assays – A dictionary of all the extracted assays from the datafile storing the data as pandas DataFrames. Individual assays can also be accessed using the get method.

Return type

dict

read(filename: str, **kwargs)[source]

Reads an input csv file.

Parameters

filename (str) – A filepath to an input csv file.
**kwargs – Any additional keyword arguments to be passed to pandas’ read_csv function.

class qpcr.Parsers.Parsers.ExcelParser[source]

Bases: qpcr.Parsers.Parsers._CORE_Parser

Handles reading and parsing excel files that may contain multiple assays. It extracts datasets either through regex pattern matching or/and through provided decorators within the datafile.

pipe(filename: str, **kwargs)[source]

A wrapper for read+parse

Note

This is the suggested use of ExcelParser. If a directory has been specified into which the datafiles shall be saved, then saving will automatically be done.

Parameters

filename (str) – A filepath to an input excel file.
**kwargs – Any additional keyword argument that will be passed to any of the wrapped methods.

Returns

assays – A dictionary of all the extracted assays from the datafile storing the data as pandas DataFrames. Individual assays can also be accessed using the get method.

Return type

dict

read(filename: str, sheet_name: str = 0, **kwargs)[source]

Reads an input excel file.

Parameters

filename (str) – A filepath to an input excel file.
sheet_name (int or str) – The name of a specific spreadsheet of the file to read. If none is provided by default the first sheet will be read. Only one single sheet can be read at a time. If an integer is provided the sheets will be accessed by their order, otherwise by their name (if a string is provided).
**kwargs – Any additional keyword arguments to be passed to pandas read_excel function.

The qpcr.Parsers

Working with “irregular files”

“Finding” relevant datasets through assay_patterns

Working with multi-assay files

Working with qpcr.Parsers

Decorators

“Finding” relevant datasets through `assay_patterns`

Working with `qpcr.Parsers`