Three Tips to Creating an Engaging Survey - that you might not have heard before
A great survey does more than collect valuable insights. But how do we keep people engaged?
Extension teams are made up of awesome people trying to do amazing things, but how do we measure the impact of the work these teams are doing?
Superpower: Throwing a frisbee
Vaughan leads our impact modelling work area. His work also spans workforce research, and extension programme design. He’s frequently involved in projects involving both quantitative and qualitative analyses.
His experience involves working with clients across a range of primary industries including horticulture, dairy, and sheep and beef, where he enjoys developing tools and resources that are interactive and intuitive for a range of uses.
Vaughan holds a Master of Agribusiness (with Distinction) and a Bachelor of AgriCommerce in Agricultural Economics, both from Massey University. During his Masters, he worked with Zespri to conduct a situational analysis of the economic impact of sooty mould in New Zealand’s kiwifruit industry.
Extension teams are made up of awesome people trying to do amazing things, but it can be difficult to measure the impact of the work these teams are doing on the intended financial, environmental, and/or social outcomes.
This matters when extension teams are discussing the return on investment (ROI) with investors at the start or end of the project. The lack of good and accessible information to estimate the ROI could mean that projects which should have been funded were not, or projects that should not have been funded were.
There are three challenges associated with collecting the information useful for these discussions:
Modelling can be seen as complement measurement in an evaluation of extension activities to address these challenges.
The ‘QUICK’ – Quantifying and Understanding the Impact of Capability and Knowledge – modelling framework has been developed to quantitatively characterise the causal chain depicted in intervention logic models (see image above); training leads to changes in capability and motivation, which leads to practice changes, which leads to better outcomes.
The key logic for underpinning the QUICK framework include:
This is an immensely complex system; however, a lot of the complexity disappears as soon as the modelling scope is reduced from the whole system, down to an individual level.
The QUICK framework has been constructed using an agent-based approach where the above logic of participation is repeated for each individual in a virtual population where participation is randomly initiated depending on several factors. Effectively, this approach creates a virtual database of all the practices and the changes that are occurring for different segments of the virtual population at different points in time.
As is the case with most forecasting activities, there is a certain level of uncertainty which must be addressed by the QUICK framework.
The approach to handling uncertainty in the QUICK framework suggests that the approach would be better described as semi-quantitative, where ‘semi’ refers to the preference to have inputs and outputs as ranges rather than a single number.
The framework is more interested in being robust in the approach and confident that the outcomes fall within a range rather reporting outcomes as a precise number.
The QUICK framework has been applied to several different contexts for Scarlatti’s clients in New Zealand’s primary industries (e.g., Beef + Lamb NZ, Extension E350, Ministry for Primary Industries) including:
Despite the historic application in New Zealand’s primary industries, the QUICK framework is robust enough to be applied in any extension context which targets a change in behaviour.
Scarlatti’s Director, Adam Barker, presented the QUICK framework at the 2022 APEN (Australia-Pacific Extension Network) virtual conference.
Check out Adam's presentation, which covers the QUICK framework in more detail and discusses its application, to the Ministry for Primary Industries’ Extension Services programme.
Contact the impact modelling team