This User Guide details piccolo3 *.pico (json files) data processing. It has been adapted from the User Guide for piccolo2 *.picco files written by Lena Scheiner and Marika Honkanen at FMI Research
Station Tähteläntie, Sodankylä, Finland 20190509

The *.pico  data format is ‘human’ readable. Piccolo3 data are stored in json file format with a .pico extension and can be read by most numerical computing programming languages (Matlab, R, python etc). TeamPiccolo have now moved all data processing to python3 and use NetCDF processed data storage format. TeamPiccolo make the functions they use available through piccolo3-utils. Please cite the DOI reference on GitHub as appropriate and acknowledge the authorship if this Used Guide to Scheiner and Honkanen (2019). Users are free to use these routines at their own risk or write their own parsers as required. TeamPiccolo can only offer support for python3 routine contained within piccolo3-utils.

1: Installation of piccolo3 processing utilities in Ubuntu 20.04
1.1. Install aptitude using a terminal. Aptitude is a user interface where  packages  can be search for and installed

sudo apt install aptitude

Then

sudo aptitude

to open the GUI. You can now search (/) for, pick (+) the required packages and install (g and g). Use aptitude to install the following packages (alternatively use sudo apt install and list the packages to install)

1.2. The data processing requires python3 and some  packages. Install: python3-xarray; python3-numpy; python3-matplotlib; python3-bitarray; python3-scipy;  python3-pyqt5, and also install netcdf-bin and git.

A virtual environment should be  used. Install the packages python3-virtualenv.

1.3. Next download the piccolo3-common and piccolo3–utilities packages from the TeamPiccolo GitHub repository. Open your browser  and go to https://github.com/TeamPiccolo.

Open the the piccolo3 -common repository  and press the button ‘code’ on the top right and copy the http://* -link.

1.4 Next create the piccolo3 directory structure and install the program in the virtual environment.

Open a new terminal and make a piccolo3 directory.

sudo  mkdir piccolo3

Go to that folder with ‘cd’ and make a directory for the source code

sudo mkdir src

Copy the python3 packages into your virtual environment named “piccolo”

virtualenv –system-site-packages -p python3 piccolo

Activate the virtual environment ‘

     . ~/piccolo3/piccolo/bin/activate

N.B. do not omit the preceding ‘ . ‘

There should now be (piccolo) at the beginning of the command line. You are in the virtual environment.

1.5. Go to the source folder ‘cd src‘ and in the terminal  type git clone,  paste the piccolo3-common link http://* from GitHub and run.

Then  go to the piccolo3-common directory that has just been created

   cd piccolo3-common

now install piccolo3-common in your virtual environment by

   python setup.py  install

1.6. Go back to the directory src, e.g. with ‘cd ..’ and repeat step 1.5 above but for piccolo3-utils

2: copying data
2.1. Copy data from to your local computer with the ‘rsync’ command that can copy and synchronize data also from remote systems.
From remote (Pi): ‘rsync -avv pi@ IP:/mnt/piccolo2_data/ localComputer/data-director’
2.2. If you mount usb to your local copmuter: ‘rsync -avv localComputer/usb-directory localComputer/data-directory’

3: Data processing using python3-utils

3.1 To activate the virtual environment use command ‘. ~/”foldername”/piccolo/bin/activate’.

 3.2 To convert your data from *.pico to netcdf files, use the command

piccolo3-read ~/”data storage folder”/*.pico -c “Calibration files”*.nc -p “prefix output data

The outputs are 4 netcdf files, one for each fibre. The “-c” in the command requires the datafiles with the information about radiometric configuration. That converts your data into physical units. The “-p” sets the prefix for the outputfilename. The structure of the netcdf files can be examined with e.g. ‘ncdump -h “filename.nc“‘

You can now post process the daat by using ‘ipython3’ for example.

open ipython3 in a terminal

Import the packages xarray, numpy and pyplot from matplotlib using ‘import “name of package”‘.

Use e.g. ‘data = xarray.open_dataset(“outputfilename.nc“)’ to load your data.

To show all your spectra from one outputfile “outpufilename.nc“, use ‘pyplot.plot(data.wavelengths, data.spectra.T)’ and after it ‘pyplot.show()’.

Using the attributes of data, a specific selection of data is possible:

To show only one spectrum of one sequence use e.g. ‘onedata = data.where(numpy.logical_and(data.batches == 1, data.sequences==1))’ and ‘pyplot.show()’. There you can identify batch and sequence numbers.

To show spectra of one run use e.g. ‘onerundata = data.where(numpy.logical_and(data.runs == “runsname”, data.bathces==1))’ and ‘pyplot.show()’. You can check your run names using ‘data.runs’.

To show the mean spectrum of the selected data, use the mean attribute of spectra e.g. with the command ‘pyplot.plot(data.wavelengths, data.spectra.mean(dim=’measurement’)) and ‘pyplot.show()’.