Any views expressed within media held on this service are those of the contributors, should not be taken as approved or endorsed by the University, and do not necessarily reflect the views of the University in respect of any particular issue.

Magnus Hagdorn

Magnus Hagdorn

Research Software Engineer

Setting up a Globus Online Service

The School storage is continuously increasing and datasets of many terrabytes are no longer exceptional. One of the big problems with large datasets is how to transfer them from one system to another. One of our research groups collects micro tomography datasets of the order of tens of terrabytes. These need to be transferred from the facility where the data were collected to our storage for further processing. Traditionally services like ftp or rsync were used. These services work well for small datasets but not for these large datasets. This is where Globus Online steps in. Globus Online allows you to use your local identity to schedule moving large datasets from one system to another. As a user you log into the Globus Online service and authorise access to your data collections. Once authorised you can schedule data movement. Globus also allows guest access in which case it works a little like dropbox. This post is about setting up a managed Globus Endpoint in the School of GeoSciences using a base LCFG system.

A Globus Endpoint had been on our TODO list for years. The main stumbling block was that the identity provider (shibboleth) as implemented by the University of Edinburgh did not export all the metadata required by the Globus Service. Getting this changed required considering data protection implications and a change that had to be made at university level. Once the shibboleth service was reconfigured to provide the required information to Globus Online the implementation of the service became straight-forward.

We use LCFG to manage our Linux systems. All school storage is centralised using ceph. Individual directories are mounted using the automounter maps defined in LDAP. For our Globus Endpoint we wanted to use the same technology stack. Globus is managed using command line tools that setup a webserver and let’s encrypt certificates. There are LCFG components that will manage these. In the end I decided to let the globus tool do their configuration and use LCFG just to set up the base system and manage the firewall. We also restrict our Globus Endpoint to only expose users’ scratch directory and, on request, group directories. The globus package gets installed from our package service which mirrors the official Globus Online package repository.

Once the base system is installed and we got suitable firewall exceptions for the Globus Service in place we configured the system using the instructions in the official Globus Online Documentation. Given the long wait to get the IDP sorted out, the installation was very straight-forward.

In the School of GeoSciences we use the universities IDP for user authentication. However, we only allow a subset of university users onto our system (mostly just the members of the school). So, it was interesting to see what would happen when a university user but not a school member tried to access our service. As hoped, Globus Online complained that although the user was an authenticated member of the university the user named had no valid mapping for our system.

Leave a reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.


Report this page

To report inappropriate content on this page, please use the form below. Upon receiving your report, we will be in touch as per the Take Down Policy of the service.

Please note that personal data collected through this form is used and stored for the purposes of processing this report and communication with you.

If you are unable to report a concern about content via this form please contact the Service Owner.

Please enter an email address you wish to be contacted on. Please describe the unacceptable content in sufficient detail to allow us to locate it, and why you consider it to be unacceptable.
By submitting this report, you accept that it is accurate and that fraudulent or nuisance complaints may result in action by the University.