Piccolo3 System User Guide. Section 04. Reading, Processing and applying calibration coefficients to *.pico files
This User Guide details piccolo3 *.pico (json files) data processing. It has been adapted from the User Guide for piccolo2 *.picco files written by Lena Scheiner and Marika Honkanen at FMI Research
Station Tähteläntie, Sodankylä, Finland 20190509
The *.pico data format is ‘human’ readable. Piccolo3 data are stored in json file format with a .pico extension and can be read by most numerical computing programming languages (Matlab, R, python etc). TeamPiccolo have now moved all data processing to python3 and use NetCDF processed data storage format. TeamPiccolo make the functions they use available through piccolo3-utils. Please cite the DOI reference on GitHub as appropriate and acknowledge the authorship if this Used Guide to Scheiner and Honkanen (2019). Users are free to use these routines at their own risk or write their own parsers as required. TeamPiccolo can only offer support for python3 routine contained within piccolo3-utils.
1: Installation of piccolo3 processing utilities in Ubuntu 20.04
1.1. Install aptitude using a terminal. Aptitude is a user interface where packages can be search for and installed
sudo apt install aptitude
Then
sudo aptitude
to open the GUI. You can now search (/) for, pick (+) the required packages and install (g and g). Use aptitude to install the following packages (alternatively use sudo apt install and list the packages to install)
1.2. The data processing requires python3 and some packages. Install: python3-xarray; python3-numpy; python3-matplotlib; python3-bitarray; python3-scipy; python3-pyqt5, and also install netcdf-bin and git.
A virtual environment should be used. Install the packages python3-virtualenv.
1.3. Next download the piccolo3-common and piccolo3–utilities packages from the TeamPiccolo GitHub repository. Open your browser and go to https://github.com/TeamPiccolo.
Open the the piccolo3 -common repository and press the button ‘code’ on the top right and copy the http://* -link.
1.4 Next create the piccolo3 directory structure and install the program in the virtual environment.
Open a new terminal and make a piccolo3 directory.
sudo mkdir piccolo3
Go to that folder with ‘cd’ and make a directory for the source code
sudo mkdir src
Copy the python3 packages into your virtual environment named “piccolo”
virtualenv –system-site-packages -p python3 piccolo
Activate the virtual environment ‘
. ~/piccolo3/piccolo/bin/activate
N.B. do not omit the preceding ‘ . ‘
There should now be (piccolo) at the beginning of the command line. You are in the virtual environment.
1.5. Go to the source folder ‘cd src‘ and in the terminal type git clone, paste the piccolo3-common link http://* from GitHub and run.
Then go to the piccolo3-common directory that has just been created
cd piccolo3-common
now install piccolo3-common in your virtual environment by
python setup.py install
1.6. Go back to the directory src, e.g. with ‘cd ..’ and repeat step 1.5 above but for piccolo3-utils
2: copying data
2.1. Copy data from to your local computer with the ‘rsync’ command that can copy and synchronize data also from remote systems.
From remote (Pi): ‘rsync -avv pi@ IP:/mnt/piccolo2_data/ localComputer/data-director’
2.2. If you mount usb to your local copmuter: ‘rsync -avv localComputer/usb-directory localComputer/data-directory’
3: Data processing using python3-utils
3.1 To activate the virtual environment use command ‘. ~/”foldername”/piccolo/bin/activate’.
3.2 To convert your data from *.pico to netcdf files, use the command
piccolo3-read ~/”data storage folder”/*.pico -c “Calibration files”*.nc -p “prefix output data
The outputs are 4 netcdf files, one for each fibre. The “-c” in the command requires the datafiles with the information about radiometric configuration. That converts your data into physical units. The “-p” sets the prefix for the outputfilename. The structure of the netcdf files can be examined with e.g. ‘ncdump -h “filename.nc“‘
You can now post process the daat by using ‘ipython3’ for example.
open ipython3 in a terminal
Import the packages xarray, numpy and pyplot from matplotlib using ‘import “name of package”‘.
Use e.g. ‘data = xarray.open_dataset(“outputfilename.nc“)’ to load your data.
To show all your spectra from one outputfile “outpufilename.nc“, use ‘pyplot.plot(data.wavelengths, data.spectra.T)’ and after it ‘pyplot.show()’.
Using the attributes of data, a specific selection of data is possible:
To show only one spectrum of one sequence use e.g. ‘onedata = data.where(numpy.logical_and(data.batches == 1, data.sequences==1))’ and ‘pyplot.show()’. There you can identify batch and sequence numbers.
To show spectra of one run use e.g. ‘onerundata = data.where(numpy.logical_and(data.runs == “runsname”, data.bathces==1))’ and ‘pyplot.show()’. You can check your run names using ‘data.runs’.
To show the mean spectrum of the selected data, use the mean attribute of spectra e.g. with the command ‘pyplot.plot(data.wavelengths, data.spectra.mean(dim=’measurement’)) and ‘pyplot.show()’.
Comments are closed
Comments to this thread have been closed by the post author or by an administrator.