1

Service Feedback, how to ‘git gud’

|

Running a service like Noteable comes with its own particular set of problems. Academic colleagues who have used the service in their teaching always say that the service is great, which itself is very good, the problem comes when trying to solidify this into something that you can take to others.

I’ve realised that it sounds like I’m complaining that people like our service, let me clarify, the problem is getting other people to believe me and get on board as well. Within academia it can be difficult to find time to experiment with something new, one of the best parts of being a Learning Technologist is to make it easier for colleagues to find tools and services that suit their needs and help them to deliver teaching.

This is how I meander my way to talking about feedback, the process of documenting how people found using your service; turning this into things that you can work on and then using it to (identify and then) shout about what your service is good at.

So, you’ve decided to do some service feedback. Here’s what you should know…

I spent a lot of time trying to find some sort of template for what questions to ask but I ran up a blank, I also found a collection of incredibly strange/weird questions which I had to fight the urge to include purely to indulge myself: “If our service were a car, what brand of car would it be?”

Since I couldn’t find anything I decided to try and make my own and then share the template out for others to improve on. I’ll break this down into two components, questionnaire feedback and User Stories/Use Cases (call them what you want but in the end it’s a first-hand account of what your users do with the service and how they found it). Today I’m going to focus on the first.

Questionnaire Feedback:

Questionnaires are usually a bit of a go to, they don’t take as long and you can normally get so numbers which look good on a page at the end of it. They (can be) a quick way of finding things out and the information can be useful, if you have the right questions that is…

In thinking about what the right questions are you really need to think about; what information do you want to know and do these questions allow your users to give you that information. For the vast majority of you reading this that will be very obvious but it still bears thinking about. Below, i’ll list all of the questions that I eventually created alongside the rationale.

How have you used the Noteable service in your teaching? (Free text answer)

A basic opener to get a bit more information about how your users have been engaging with your service. This sounds like a fairly bland opener but it does a number of things; gives you context information and also gives you a handy list of “this service can be used for…”

Do you intend to use the Noteable service again? (Free text answer)

Very straight to the point, this service was offered in a trial phase so it’s important to know how many people thought it was good enough to want to use again. This gives you direct retention numbers and helps you to justify that you are fulfilling a continuing need.

How likely are you to recommend the Noteable service to a colleague who has similar teaching needs? (Free text answer)

For a service that wants to grow this is key. People might like your service but if they really like it then they will suggest it to their colleagues. It’s a good metric as it’s a great thing to brag about, essentially it’s not just you that is saying that the service is great but also the users. Users will always be more open to hearing how good a service is from their colleagues so there is also the implication that the service will continue to grow naturally.

How would you rate the ease of use of the Noteable service? (Scale Question 1Low-5High)

Ease of use means how easily is it for you to use each time you use it in your teaching. You want to know if your users constantly feel like they are straining to use your service. Making the question more open like this means you aren’t highlighting what you think are barriers but looking out for anything that could be. A caveat with these questions is that any score under 3 will open a follow up question asking about specifics.

How would you rate the ease of adoption? (Scale Question 1Low-5High)

This is in relation to how much effort it took to introduce your service into their teaching. You obviously want to pay attention to this as it is going to impact the amount of new users and the early attrition rate. Again, this has a follow up if it gets a low mark to get more useful information on what you can improve on. Removing or lowering as many barriers to adoption is going to be the biggest thing to help you grow a service.

What would you do without the Noteable service? (Free text answer)

This might be one that is more suited to our position with the Noteable service. We are offering a centralised service so it’s good to know if this is reducing a burden or making it easier for people to use this specific tool – Jupyter notebooks. As a pilot service we need to be able to say there is a justifiable need for providing this service, that without this service users would be inconvenienced.

What is the most important feature that you would like to see? (Free text answer)

Nothing like crowd-sourcing your new features….

To what extent do you agree with the following statement: “Using Noteable helped me to deliver my teaching/training” (Scale Question 1Low-5High)

Directly asking if introducing this tool has any discernible affect on the ability of our colleagues to actually deliver their teaching. It’s often very difficult to quantify where something has a pedagogic impact but by asking this directly we are getting the answer from the people who are most likely to know.

To what extent do you agree with the following statement: “Using Noteable helped my students to interact with the material” (Scale Question 1Low-5High)

Again, this is something that is difficult to quantify but can have a massive impact. If your students can more easily engage with the material it means you can get further into more complex subject, spend more time in theory and less time getting over barriers. This will be followed up with student questionnaires at a later date.

It’s important to not lose sight of how valuable this information can be, whilst you will want to shout about good feedback make sure you spend the time fixing what users have reported and working on what they have said they want in the future. Few things can improve your relationship with your users more than “you said, we did”.

This questionnaire is currently live, I will share the feedback when it is finally complete. Please let me know if you have any improvements or insights into this, I’m always trying to make this better.

p.s. if you are going to steal these questions, at least replace Noteable with [your service name]. (I have actually seen this before and it is amazing/terrible)

 



More about Noteable

|

I’ve recently tried to become a little more involved in the wider Jupyter community and I have to say that it’s the most amazing community to be part of and work with. We at the University of Edinburgh have already managed to get funding for our nbgrader workshop which has been a great excuse to introduce the University, myself and the Noteable service.

If you want to get involved, or even just be in on the discussions then first head over to Jupyter Discourse, secondly make sure to get in on the newly revived Jupyter Community meetings. The community is always full of interesting nuggets or information and with such a rapidly growing community there is always something new coming up. If you worry about not being able to keep up then I can highly advise the Tracking Jupyter Newsletter by Tony Hirst

Being involved in the community has already started to pay off with the following discovery, originally found on Discourse – https://discourse.jupyter.org/t/in-depth-comparison-of-cloud-based-services-that-run-jupyter-notebook/460/14

This is a great intro to what’s out there is the Jupyter ecosystem, a breakdown comparison of 6 services that allow you to run/share/preview Jupyter notebooks……but Oh Dear, Noteable isn’t on the list! Time to get our name up there with the big companies like Microsoft and Google. So here is my contribution to how Noteable compares, including some bonus points! Noteable is a service designed for use in educational institutions, currently looking for partners in the UK.

Supported languages: Python 2.7, Python 3.5, Julia, R, Sage, Matlab, Calysto, Octave, Haskell (also able to install other languages by request!)
Ability to install packages: Yes, for the duration of your session. Also we can install packages by request to avoid having to install them yourself (also we have a lot already – https://noteable.edina.ac.uk/code-libraries)
Interface similarity: Uses Native Jupyter notebook interface
Keyboard shortcuts: Uses all of the same keyboard shortcuts as Jupyter
Missing features: None
Added features: nbgrader (currently in Beta)built using Docker containers to allow you to switch between environments quickly by launching a new container.
Ease of working with datasets: You can upload a dataset to your project from your local computer, and it can be accessed by any notebook.
Internet access: Yes
Ability to work privately: Yes
Ability to share publicly: Not directly but you can use Git to share your repos or send notebooks manually.
Ability to collaborate: Not currently
Ability to upgrade for better performance: As a service designed for education there is no ‘free’ version but we do offer trial access to Noteable to institutions that are interested.
Documentation and technical support: General documentation for integrating with your LMS via LTI is available. General introduction documentation to using Jupyter is currently being created.
Conclusion: Noteable is the only service developed and designed for use in education, primarily further and higher education. We have already developed a working LTI connection to fit in with your LMS and work closely with our users to add features that are of the most benefit.

If what you see above sounds like something you would be interested in then feel free to get in touch and ask for a trial for your institution.

Extra

After following the original thread on Discourse and the Google Group there have been some extra questions raised:

– Jupyter Authenticator Support: Can it work with my organizations’ existing single sign on app? e g. 2FA, TOTP, the new Webauthn, GitHub, GitLab

Noteable can work with your institutions login, with a working LTI we can also directly connect with your LMS meaning that your users can be seamlessly logged in and also directed to specific course (which becomes important with our new Assignments)

– Mobile interface: Can any of the existing iOS and Android front ends connect to the system?

Directly from Noteable we do not offer our own App but I can say that we work on Juno, this is a paid App but does allow you to try some basic notebooks for free.  They have also recently announced that they will be donating back a portion of their profits to Jupyter so a win all around.

– Backups: How does the system backup work? How frequently, to where, and are there backups tested to make sure that they work?

Noteable auto-saves your work at regular intervals (every few minutes if you are making changes). You can also choose to save a Checkpoint if you think you might need to revert.



And we’re off!

|

The Noteable service is into its next big pilot phase this Semester. Over 500 students will be using the service this week across 6 different courses in 6 different School within the University of Edinburgh.

But what is the Noteable service and what is the scope of this pilot? I’m glad you asked (You did ask, I heard you)

Here’s a quick summary of what the pilot involves, what we will be hoping to achieve and how we will measure it. Comments are more than welcome. (Nice ones)

 

What will be delivered

The Noteable service is a cloud-based application providing access to Jupyter notebooks online. Noteable provides a central space to store and run Jupyter notebooks in a variety of languages.

The purpose of Noteable is to allow students and staff to access Jupyter notebooks at any time without the need for pre-installation which can be cumbersome and difficult for programming novices. Noteable is integrated with the institutional VLE to allow for a central launch point into a pre-set environment without the need for a separate login.

Duration

The initial pilot will run for the duration of Semester 1 of the 2018-19 Academic Year concluding in December 2018. Feedback information from staff and students as well as usage data and indications of further demand will be collected and presented before the end of December 2018.

Users – Students, Courses, Schools

For the pilot of this service, we have worked to ensure pilot users from across the University. The Semester 1 pilot will involve 580 students in 6 courses from 6 different School within the University.

We are also aware that the Noteable service will be used as part of the Digital Skills Programme and we will report of the number of students involved in these sessions at the end of the semester.

Benefit

There are multiple benefits to be derived from the Noteable service pilot. Firstly, staff who already incorporate Jupyter notebooks into their teaching will benefit from removing the need to have students install Jupyter beforehand which is time-consuming and can cause issues, especially when using multiple additional packages. We will work alongside current users in the creation of Jupyter specific OER materials which will help new users be more easily able to adopt the service.

 

Success Criteria

The key success criteria for the Noteable service pilot will be based around 3 components:

  1. Demonstrate Need
  2. Service Fit
  3. Service Cost
 1. Demonstrate Need

The first goal of the Noteable pilot will be to determine that there is a need for a centrally supported notebook service for teaching purposes. This can be quantified both with the number of current users as well as secured future users. This will be broken down into two measurable indicators: number of courses using Noteable and number of students within these courses as a total. There will also be an additional measure citing the number of different Schools associated with the pilot to help ensure that the service is widely accepted across the University.

Goal: 6 Courses with up to 500 students in Semester 1

 2. Service Fit

Define whether the Noteable service fits the needs of the user community. This includes comparing the Noteable service with other comparable services. This criterion cannot be easily quantified and will largely be based on the feedback from the current and prospective user community. As part of this evaluation, there will be a suggestion as to whether to use Noteable or use a comparable service.

Goal: Create a comparison document with service recommendation

 3. Service Cost

Determine the total cost of running the Noteable service including the cost of staff (both development and support) and operational/infrastructure cost. This can be determined as a yearly cost and also broken down to a per-user cost. This defined cost can then be used as part of the comparison with other comparable services as per criterion 2.

Goals: Cost per year and Cost per user.

 

And to makes all this seem a little more real, here’s a graph of the use for Tuesday

Jupyter notebook Usage Graph

First ‘big’ usage day, peaking at over 100 concurrent Jupyter notebooks. The best bit is; this isn’t even our busiest day.



ProgTeach Play by Play

|

The first ProgTeach Symposium took place on the 24th of August, this has been something that colleagues and I have been working towards for some time so it was great to finally see this come together. The premise is fairly simple, across the University of Edinburgh lots of people teach computation in some form so let’s make time for getting as many of these people in the same room together and discuss what they do (and how they do it).

As my first main event that I’ve run at the University of Edinburgh, my main concern was that I had put something together that no one would want to attend…..we filled 80% in 24 hours. As you can see below the event was pretty well attended, what you can’t see is how diverse the audience was, colleagues from 13 different schools turned up with a smattering of units within those schools.

So to kick us off we had an introduction from Anne-Marie Scott, Deputy Director of Learning, Teaching and Web Services. Anne-Marie has been very keen on the idea of running these types of events to get people talking together about their teaching practice and is also my boss’s boss so no pressure.

Up next our first speaker was Dr Areti Mantaki from the Centre for Medical Informatics to talk about her experience of delivering large (huge) scale distance learning introduction to programming courses.

Areti shared some great incites about teaching coding generally and also some of the challenges (and surprises) of teaching entirely online. Something that a lot of attendees agreed with was how important it was to include mistakes in your examples but also spend time walking through the solution.

Next up we had Professor Chris Sangwin and Dr Stuart King, both from the School of Mathematics who talked about their introductory programming course. This including speaking about introducing Jupyter notebooks into their teaching as well as using CodeRunner as an assessment tool. Important to note that the students really enjoyed the ‘game’ of pursuing getting full marks in practice question sets.

Then it was my turn (James Slack) to talk about the University’s Noteable service currently in a pilot phase. For those not familiar with Noteable this is the University’s own JupyterHub environment, giving access to Jupyter notebooks for teaching purposes. Although this is still in a pilot phase the demand has been high and we will be delivering the service to over 500 students this coming semester.

The next speaker was Andrew Kirk from the Digital Skills and Training Team who are using the Noteable service as part of their Digital Skills Programme to deliver beginner Python courses to students from across the University. The courses were overwhelmingly popular, the first session had 90 bookings for only 20 places and will run again in the coming semester.

After the lunch break, we had a quick talk from Sean McGeever about the carpentries workshops in Edinburgh and the creation of a supportive RSE community. The theme of this talk about building a community to support colleagues from across the University was very closely aligned with the whole theme of the day. Sean also got the funding for cake so bonus points for Sean.

For the penultimate session, we wanted to focus on the on the discussion between colleagues about how they approach certain challenges of teaching computational content. Ironically no-one had the confidence the attempts to answer “How do you build confidence in learners”.

Heres a full breakdown of the discussion points – Discussion (Word Document)

And then to wrap up the day we had a quick talk about creating Jupyter based OERs to support new users, there is obviously a lot of material out there for ‘Intro to Python’ already but what can be really useful is a collection of these in notebook form that we know will work straight away in Noteable and also some ancillary materials about how to use notebooks to make it easier for people to introduce them into their teaching.

And that was that! A very interesting line up of talks and very useful discussion between disciplines which is something I’m always interested in promoting. I’ll leave it to Anne-Marie to wrap up the day:

And yes, there will be another one!

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 



Service Comparisons

|

A large part of my current project involves looking at other companies or services that offer Jupyter notebooks. The University has decided that it wants to provide notebooks but we also have to justify creating our own service. If there is something else that does what we need is it worth making our own version of it?

With this in mind, I have been creating a shortlist of services that are comparable and also drawing up a list of criteria to compare.

Here is the list of services:

Microsoft Azure Notebooks

Google Colaboratory

CoCalc

Anaconda Enterprise

And here’s a list of the criteria against which they will be compared:

Pricing  (how much do they charge and how do they charge it, what do you get access to when you pay, is there a free tier etc.)

Technical Specifications (e.g. RAM, CPU cores, Storage)

Extensions Available (e.g. nbgrader, RISE)

Supported Languages (There are many varied use cases throughout the university so how do we make sure all are supported, how quickly does the service add requested kernals.)

Focus/Audience (e.g. Azure is focused on Machine Learning, Goolge Colab is focused on collaborative programming)

Business Model (e.g. Azure is currently in a preview stage and may not be continued)

Community (This includes using the community as a measure of sector involvement and also how quickly the service administrators respond to feature requests or issues)

This comparison will take place over the summer and then I will publish our results. This is all about making sure that we are providing the best service that suits the needs of our users.

If you have anything you would like to add or think we have left any service or criteria out then let me know.



Noteable’s place it the City Region Deal

|

Heard of the City Region Deal for Edinburgh? No?! Well go check it out now, I’ll wait here for you.

City Region Deal Homepage

Interesting right? Well, I managed to gather some more information recently from one of the launch events currently making their way around the University. After you’ve gotten over the astounding £1.1bn investment there are some even more intriguing numbers on display. The one that I am mostly drawn to is the target to produce 100,000 students with a formal certification in Data Science related skills. Thankfully this is spread out over 15 years and is also shared with Heriot-Watt University but this is still a very significant commitment.

This doesn’t mean that we will become a factory churning out Data Scientists but it does mean that many of our students will leave with some amount of training and understanding of data science and the associated digital skills which Edinburgh and the wider world demands. Integrating key data science skills into a wide variety of areas of study means that more students will be equipt to compete in a digital world and we can also help to create new data initiatives based on the varied insights of our students.

There is also a social side to this deal which means that the benefits are not confined to the University but to the wider region both in terms of the economic benefit of new or improved businesses but also providing digital skills training to the public to further help drive the digital economy of the area.

So where does the Noteable service come into all this? The Noteable service provides a platform for learning how to programme and manipulate data. This platform is especially helpful for introducing students and learners to the basic concepts of a language quickly as it doesn’t need any pre-install. It may be a small cog in a very big machine but it shows that the University of Edinburgh is committed to helping to develop digital skills by providing such a platform. As well as supporting teaching ‘in-house’ the service is also well geared towards introductory courses for the public as it requires little setup or programming knowledge beforehand.

All I need to do now is find some willing instructors…….

If you have any ideas on how you could become involved in the City Region Deal then the official advice is to contact your Head of School however if you are planning on using Noteable then let me know – james.slack@ed.ac.uk



Jupyter Literature

|

Lately, I’ve been looking into the uses of Jupyter notebooks within a teaching context and I’ve noticed something of a lack of literature. There are articles looking into the other uses of notebooks (collaborative and reproducible research) but little in the way of articles about the introduction or use of them within teaching. There has been a growing movement to introduce Jupyter notebooks within teaching especially in higher education institutions but, at the moment, the ‘noise’ about them seems mainly confined to those circles. I’m not saying that this is bad as the community built around Jupyter notebooks is amazing but the lack in literature makes it difficult to get outsiders on board, specifically IT departments who could be helping to build the infrastructure to support notebook use in their institutions.

If we can address the gap then it will be easier to get new people involved and also get buy in from institutions as a whole and help embed notebooks into teaching (providing we can prove that there is a benefit) Hhmmmm…. seems like the scope of this project may have just expanded.

I’ll keep this blog updated with any findings as we go along, feel free to chip in if you have any experiences or knowledge of using Jupyter notebooks for teaching.



How to Succeed at Failing

|

Aside from being a learning technologist a big interest of mine is American football (Philadelphia Eagles fan). As you might imagine its very rare that these two aspects of my life coincide but something about the end of the Super Bowl stood out to me. In a speech just after the win Nick Foles, the Eagles quarterback, mentioned something that I believe to be very important both in life and in learning.

“I think the big thing is don’t be afraid to fail,” Foles said. “When you look at it, you have a bad day, you think your life isn’t as good, you’re failing. Failure is a part of life. It’s a part of building character and growing. Without failure, who would you be? I wouldn’t be up here if I hadn’t fallen thousands of times, made mistakes. We all are human. We all have weaknesses.”

Failure has traditionally had something of a bad reputation, something to be avoided at all costs, but there has been a growing idea that failure is a key part of the learning process (see list below). If you don’t fail you won’t grow, you won’t learn what your current limits are and you aren’t going to develop your skillset. Failing also builds resilience, which is something that employers are increasingly interested in, having employees that can deal with setbacks and carry on working creatively to solve a problem is a great asset.

Today, however, I want to talk specifically about how we can use Jupyter notebooks as a place to fail in a positive way. For those who haven’t heard of these, I’ve done a brief introduction here. One of the most useful parts of notebooks is being able to immediately see the output from your code cells. This allows you to ‘fail fast’, quickly trying something and then adjusting based on the outcomes. By repeatedly trying something and adjusting every time you build resilience and start to learn how to adapt your skills to a situation.

The next part we need to talk about is how we document failure. I don’t mean making students wear dunce caps but I do think it’s important to detail the steps you went through even if you ended up failing. Early on in school I was always told ‘show your working’ which I definitely didn’t do but now I understand the importance of the concept. Getting a question wrong on a test doesn’t necessarily show me what you don’t know but understanding how or why you got it wrong certainly does. Showing that you have failed, tried again and then succeeded shows me even more.

I’m interested to hear your opinions on this, as an academic do you think it is helpful to have students include their failed attempts or would it make marking more difficult? Add your comments below.

List of Failures

Failure Is Essential to Learning – Edutopia 2015

Stanford Resilience Project

The Importance of Failure in Learning – ironically a failed attempt to introduce failure

Is Failure Useful – a bit off piste about failure in museums but worth a read

Innovation: Learning from Failure – An interesting insight into the tech industry’s embrace of failure



Hello World

|

It’s alive! ALIIIVEE!!!! [Cackles manically as thunderclaps in the background] The Noteable service is alive, up and running, ready for action. Okay so actually it was technically alive last year as people just couldn’t wait that long but now it’s official. In brief, the Noteable service is a cloud-based service providing Jupyter notebooks. If you haven’t come across computational notebooks before then I’ve written a brief explainer Here. We are now into the pilot phase of this project, looking into the benefits (and cost) of providing a centrally supported cloud-based notebook service. Whilst Notebooks have implications for many different areas, we are specifically interested in the benefits within learning and teaching. I’ve copied in the scope section of the project draft so that you can get more of an idea of the main aims of this pilot.

Scope
  • To assess the pedagogic impact of using notebooks to introduce programming concepts.
  • To assess the desire and need for a centrally support notebooks service within the University of Edinburgh
  • To assess the use of Jupyter notebooks for creating OER materials
  • To assess the running cost of supplying this service within the University of Edinburgh
  • Work with EDiNA to develop a development plan for the Noteable service to incorporate service improvements and feature requests

Sounds easy enough right? Generally, as a learning technologist, you have your hand snapped off when you offer up a new service that people are interested in but the important work will be quantifying the impact offering this has and comparing it with what it costs the University to provide. There’s a small bonus part in here that I am particularly interested in, OERs, I think that notebooks make a great basis for OERs. They can be easily shared via Binder or nbviewer and they can be easily aligned with the 5 Rs. I’ll be trying to blog about this rather regularly, either working with Edinburgh academics to publish OERs or pointing out interesting existing resources.

Computational notebooks have been growing in use for a variety of uses in recent years, especially within HE institutions but this has created something of a quandary (or at least an opportunity). A lot of Jupyter notebooks use currently seems to be based around individual academics or small teams working to deliver their own material in their own way, there is little in the way of central support. This bit is where Noteable comes in, developed by EDINA, this is a centrally support service at the University of Edinburgh to provide notebooks. The idea is first to provide the service so that users have access to notebooks online, my part is to then make sure that users have the support they need to integrate these into their teaching. Alongside this, we want to create and promote a community of notebook users and publicise their uses to get more people involved. Some people will have come across notebooks before and are already using them in interesting ways but there are a lot more people who could start to use them if supported in the right way and that’s what I’m here for.

If you are a member of staff and are interested in using the service then get in touch. If you have stumbled in here from the wilds of the general internet then watch this space, I’ll keep you posted.