Designing Surveys can be a gargantuan task

Online surveys in Higher Education research is being increasingly used to gauge public opinion, project evaluation, measure consumer preferences, assess employee satisfaction, and collect feedback on various topics efficiently and cost-effectively. However, designing a good survey, keeping the participant’s experience in mind, and making sure it is effectively asking the right questions, while being time sensitive, can be a tricky task. In our case, the Playful Engagement Strategy was being updated after nearly four years and we wanted to understand how the wider management staff at the ISG observe changes within the creative engagement patterns and behavioural adaptations of the section, service or team members working under them. As the strategy promotes an alternative methodology of working and collaborating, the target was to ensure that the survey did not feel like an administrative burden on the management staff.

As explained by Schwarz (1999), the wording of the question, its context of use and format, juxtaposed with the instrument of choice can bring out different answers from the participants. Therefore, a series of trial and error was accompanied and anticipated during the initial testing phase of the survey design. In co-ordination with Charlie Farley, the Open Education Resources Advisor at ISG, and my manager in line, we created and tested three sequential surveys. Each of these surveys, though interconnected, was an attempt to gauge the experiential evidence of opportunities and barriers towards creative and playful engagements at the ISG, through varying qualitative and quantitative lenses.

The first version of the survey was partly qualitative and partly quantitative with impressions of particular events, anecdotes and incidents gathered from previous experiences. This version was trying to analyse how the selected events and opportunities have been perceived by the ISG staff and the barriers they foresee in executing the same post-pandemic and the shift in working habits. The drawbacks of this version, understood through peer analysis, was that it probed answers and therefore presented biases in the questions themselves.

Following this, a new survey was developed which was divided into three segments. The first segment had single selection options for the directorate they belonged to and section/service/team they managed. We also included the question “Did you know about the Playful Engagement Strategy at ISG?”, with a yes or no answer, to understand how many respondents had previously engaged with the strategy. Following this, the survey had two completely open-ended segments mapping the opportunities and barriers of the strategy within their unit. The idea was to not probe answers through selection lists and rather allow the respondents to devise their own answers. The survey had attached links to the previous works done around playful engagement within the ISG along with definitions of what playfulness in adulthood means. We circulated this survey within the EDE managers as a pilot test, to gain firsthand reviews of the survey. The responses we received were mixed, with some finding the survey extremely lengthy considering the open-ended nature of the questions. Some respondents also found the questions unclear, considering they had no prior knowledge of the strategy.

To rectify these problems, we re-structured the survey, maintaining the same format, but changing each question into an optimal variable, quantitative survey. It rather than understanding each respondents detailed qualitative experience around playfulness at ISG, tried gathering the degree of knowledge around the opportunities and barriers existing within their unit. The questions targeted on the feelings and perceptions of the managers while maintaining as single selection option for either a- yes, no, or not sure.

A few questions were as such-

Do you feel there are playful engagement opportunities in the way that your team interacts with each other? (e.g. physical spaces, hybrid and online work environments, innovation, collaboration, goal and achievement setting, and more.)

Or,

Do you feel that you and your team have opportunities to engage with wider ISG Community activities? (e.g. Healthy Working Lives, seasonal celebrations, crafting, networking events, volunteer days, ISG Bakers, reading groups, activity groups)

The questions for barriers to playful engagement, were designed differently. The spectrum of options changed to- time, personnel, self and others perception, resources, materials, knowledge, awareness, training and support. Each respondent could mark either a- yes, no, or not sure, against the options provided, signifying which areas they perceive as barriers. Parallelly, each of the respondents could also answer an optional open-ended question on the opportunities and barriers they have observed, especially in relation to their answers in the previous questions. This additional qualitative response gave deeper insight into what should be our focus for the later stages in the research process.

This version of the survey was approved for further circulation, across ISG, and we received a positive number of responses within a short period of time. The idea was always to have an overview of the situations and circumstances which have emerged since the strategy was first formulated in 2019. Looking at the responses, we found valuable critiques, interesting anecdotes and more areas to be worked upon in terms of survey design. The entire process re-instated how necessary it is to be participative while designing surveys, to address any biases which the designer might be consciously unaware of. Also, research is an iterative process, especially when we try and put emphasis on democratic feedback for participants and respondents. It is equally valuable to pilot test surveys to understand the elasticity of its reception and therefore the generalisability of the responses. This entire experience helped me build better reflexivity around the practicality of research, which inevitably I wish to apply in the later stages of the research process.

References

Schwarz, N. (1999). Selfreports: How the questions shape the answers. American Psychologist, 54(2), pp.93–105. doi:https://doi.org/10.1037/0003066X.54.2.93.

(https://www.pinterest.co.uk/pin/857161741606785791/)