Overview of the Delphi survey results

1. The Delphi survey

The Delphi method allows for a reliable and creative exploration of ideas or the production of suitable information for decision-making. It is based on a structured process for collecting and distilling knowledge from a group of experts through a series of questionnaires combined with controlled opinion feedback (Adler and Ziglio, 1996). Between October 2014 and February 2015, three questionnaires were issued to 334 panellist identified as experts in the fields of survey methodology, children and youth, well-being, and policy. Response rate were very good, ranging from 75% in the first questionnaire, to 58.9% in the second and 61.7% in the third one.

Delphi respondents worked in research / academia (n=98, 46%), policy (n=50, 23.5%), NGOs (n=50, 23.4%), and other types of organisations (n=16, 7.5%). In the first questionnaire, respondents mostly identified as researcher or expert (n=84, 33.6%). Other respondents worked in directorate (n=24, 9.6%), senior management (n=46, 18.4%), lower management (n=45, 18.0%), employee or volunteer (n=27, 10.8%), or other types of positions (n=24, 9.6%).

Delphi respondents worked in research / academia (n=98, 46%), policy (n=50, 23.5%), NGOs (n=50, 23.4%), and other types of organisations (n=16, 7.5%). In the first questionnaire, respondents mostly identified as researcher or expert (n=84, 33.6%). Other respondents worked in directorate (n=24, 9.6%), senior management (n=46, 18.4%), lower management (n=45, 18.0%), employee or volunteer (n=27, 10.8%), or other types of positions (n=24, 9.6%).

Experts were mostly identified in partner countries. Further European experts were also engaged in the project. In average, the Delphi survey was completed by experts from the following countries: Greece (n=29), Croatia (n=29), Spain (n=25), Latvia (n=22), Georgia (n=20), Portugal (n=25), Hungary (n=14), United Kingdom (n=13), Estonia (n=12), Germany (n=9), Slovakia (n=9), Cyprus (n=3), Ireland (n=3), European Institutions (n=3), Belgium (n=3), Czech Republic (n=1), Italy (n=1), Luxembourg (n=1), Austria (n=1), France (n=1), Romania (n=1).

2. The concept of well-being

The first Delphi questionnaire was used to clarify the concept of well-being as it applies to children and young people (CYP). Well-being was strongly associated by the participants with notions of health (cited 184 times), financial security and employment (cited 104 times), relationships with friends and family (cited 100 times), material conditions (cited 81 times) and happiness (cited 50 times). Overall, respondents largely agreed that 'material conditions' (75.6%), quality of life (94.0%), and sustainability of quality of life (90.0%) are important pillars of well-being. Qualitative comments indicated that material conditions alone are not enough to determine well-being. Respondents largely agreed (98.4%) that a longitudinal survey should include both objective and subjective measures. Half of the respondents (49.0%) considered they should have equal weight.

The concept of well-being used throughout the Delphi survey comprised the following domains: personal well-being (personal satisfaction with life / happiness), relationships with peers, family and home, health, time use, community and neighbourhood, money and possession, personal appearance, education and skills, competence (feeling efficient, effective, and even masterful in one's behaviour), autonomy (feeling of being in control), purpose in life (the belief that one's life is meaningful), and amount of choice (self-fulfilment).

3. MYWeB and the policy need

Experts involved in the Delphi agreed that the role of evidence is important in social policy, particularly in order to have a better understanding of policy impact (65.6%), review the design of social policies (63.2%), monitor progress (60%) and measure the distribution of policy outcomes across different target groups (56.4%). There was a strong consensus amongst panel members that the evaluation of children and youth policies supports policy makers in improving policies (84.7%).

The majority of respondents (63.0%) considered that member state level policy is more important than regional or local level policies in the context of a longitudinal cross-European survey.

Results from the Delphi survey clearly pointed out that well-being domains had high policy relevance. On a scale of one to five where five is high policy relevance, the following domains had the highest scores: education and skills (4.59), health (4.62), family and home (4.30), and personal well-being (4.07). Respondents also indicated that those domains have relatively good data coverage, except for personal well-being. The dimensions pertaining to the eudaimonic approach to well-being (i.e. competence, autonomy, purpose in life and amount of choice) display an important gap between their data coverage and their policy relevance. Qualitative comments pointed out that there are differences between European and national coverage of those domains.

Experts were asked to identify three major well-being policy challenges for children and young people. Respondents identified education (cited 89 times), poverty (cited 58 times), and health (cited 40 times) for children and education (cited 99 times), employment (cited 97 times), and participation (cited 30 times) for young people.

All the responses provided were then combined into broad categories of policy challenges organised under three themes: community/economic factors, family factors, child/young person specific factors. Experts were asked to indicate the extent to which the data collected through a longitudinal survey could address existing gaps in data coverage and therefore inform decision-making. Respondents considered that a longitudinal survey could address existing gaps in data coverage and therefore inform decision making for all of the policy challenges suggested:

From my experience, the data collected through a longitudinal survey can play a key role in addressing existing gaps in data coverage and therefore can be valuable resource at the policy and decision making level."

- Policy expert, Delphi 2

With regard to community/economic factors, a longitudinal survey could be particularly relevant to address data coverage gaps in relation to equality, multi-culturalism, youth migration, and joined up policies/better coordination of services.

Overall, family factors had higher scores than community factors, indicating that longitudinal surveys could provide information that is more useful at this level. Results indicate that a longitudinal survey could provide information filling gaps in relation to education (including early years), family support (including fathers), and poverty. A few policy challenges appear to have been scored lower by respondents working in research than those working in policy or NGOs: food and nutrition, flexible working for parents, and child protection.

At a child/young person specific level, a longitudinal survey could improve the data coverage of the transition from school to work, mental health, and participation /empowerment. A number of policy challenges were scored higher by respondents working for NGOs: disabilities, NEET young people, addictions, resilience, and media/new technologies. Results also indicate that data collected through longitudinal surveys is less relevant to leisure and sports than the other challenges identified.

4. Data coverage

The majority of respondents (59%) agreed that the survey should focus on the entire life-course of a young person from birth to the age of 25. Some respondents indicated that the age limit of a young person should be extended to 30 years old to reflect societal changes (e.g. young people still living at home because of studies or unemployment).

Respondents generally agreed that prenatal information can improve child well-being social policy. A greater percentage of respondents working in policy (93.9%) agreed with this statement than those working in research (82.7%). Qualitative comments indicated that a better understanding of the prenatal period could improve preventative policy.

Over three quarters of the respondents acknowledged a need to improve the lack of longitudinal perspectives (81.6%), improve coverage of particular population groups (78.8%), and improve subjective measures of well-being (75.2%). Data coverage of particular population groups varies strongly. For instance young carers are a group for which data was considered not to be covered at all by 57.6% of the respondents. Over a third of the respondents also indicated that data is not covered at all for absent parenting (43.2%), and children and young people who have suffered significant harm (38%).

Data coverage is better for children and young people from single households, children and young people from low-income families, children and young people from a minority background, and children and young people with physical disabilities.

Furthermore, difference across countries can be very important (e.g. Lack of data about number of young people living with a physical disability in Georgia or lack of data collected about ethnic minorities in France). However, there is a strong consensus amongst experts than a cross European longitudinal survey should gather data from all children and young people rather than focus on specific groups.

5. Questionnaire content

The results from the first questionnaire showed that the majority of respondents (77.2%) agreed that standardised European measures should be complemented by country specific modules or questions. This was further explored in the second questionnaire during which experts were asked to indicate whether well-being domains can be captured through standardised European measures or if they can only be captured using nationally specific measures. The majority of respondents indicated that standardised European measures are applicable to the different domains of well-being: competence (84.7%), health (84.5%), education and skills (81.9%), time use (70.2%), and personal well-being (70.0%), amount of choice (67%), personal appearance (63.7%), money and possession (62.9%), relationship with peers (62.3%), purpose in life (58.9%) and family and home (55.4%). However, the majority of respondents (58.9%) considered community and neighbourhood to be a domain that could only be captured using nationally specific measures. This issue was clarified in the third questionnaire, where 81.0% of the respondents answered that it is feasible to develop a classification of neighbourhoods on the basis of an index of relative deprivation that is comparable across Europe.

Yet, concerns were expressed in relation to the complexity of the task, its cost, different meanings attached in different countries to categories such as homeownership, and how one’s subjective feeling about their neighbourhood impacts their well-being more than objective indicators.

Experts suggested that a cross European well-being survey could be made more sensitive to local variations concerning community and neighbourhood by using additional measurement for each country and/or qualitative measures. It was also suggested that a local expert panel should be appointed in each country to analyse and clean the data.

Household contextual factors also appear important for CYP well-being. On a scale of one to five where five is very important, all contextual factors have an average score above 3.43. The majority of respondents considered the following measures suitable to measure the family and home situation across Europe: educational attainment of members of the household (94.4%), employment status of members of the household (93.0%), household composition (92.0%), occupational status of members of the household (89.2%), size of the household (86.2%), relative household income (85.5%), accommodation type (82.5%), marital status (76.6%).

6. Children and young people's participation

Respondents largely agreed that children's (85% of respondents) and young people's (94% of respondents) views are an important input into designing the research instrument. This was particularly the case for young people (68% strongly agree) compared to children (45% strongly agree). Respondents working for NGOs had a tendency to consider children's views as more important than those working in policy or research. Qualitative comments indicated that CYP views are important because they provide a unique insight on children and young people's well-being, which can be quite different from those expressed by adults.

A number of respondents also mentioned the UN Convention on the Rights of the Child, which stipulates that children have to right to participate in decisions that concern them.

The majority (56.8%) of respondents consider that evidence measuring children's and young people's well-being should always include views from parents and primary carers. However, a third of the respondents (36.8%) considered that information from parents and primary carers is necessary only for some age groups. Some respondents suggested that evidence should also include a wider range of views (teachers, health professionals etc.).

Research undertaken with CYP on their understanding of well-being for this project indicated that that some young people are interested in participating in such a study, because they perceive interviews as a chance to express themselves in front of someone who is willing to listen to them. Children and young people also indicated that incentives would enhance their willingness to participate. Delphi participants considered this information and there was a very strong consensus among participants (93.1%) that complementary qualitative interviews should be introduced in a longitudinal survey in order to enhance participation. Respondents also largely agree (66.7%) that incentives should be introduced to enhance participation.

Further analysis of the qualitative work undertaken with children and young people revealed that their will to participate in research often draws on the fact that it is an opportunity to express their views and that it could help other children and young people. They are interested in the impact of the research. This can be an important factor in empowering children and young people and keeping them engaged throughout the course of a longitudinal survey.

Experts suggested the following methods to disseminate findings:

  • social and multimedia such as videos, podcasts, and apps (cited 74 times),
  • events such as seminars and workshops or participation camps (cited 22 times),
  • presentations in schools (cited 14 times), and making sure the content is targeted to different age groups (cited 12 times).

7. Survey methodology

The first questionnaire did not bring consensus amongst experts about the survey design. Overall 56.6% of the respondents indicated that the most suitable design would be a cohort design whereas 43.4% indicated it would be a household panel design. When asked about further options in the second round, respondents indicated a preference for an accelerated cohort design where data collection would start simultaneously with different age cohorts covering a specific life span. This contrasts with both a narrow age based cohort design which traces a single age cohort as they grow up and a wide age sample which does not differentiate specific age cohorts. The accelerated cohort design was particularly popular for respondents working in policy as illustrated in Figure 1.

Figure 1. Survey design preference by respondents
Figure 1. Survey design preference by respondents' background.

A third of the experts (31%) indicated that if possible, a cross-European longitudinal survey should have an unlimited length. Another third (30%) indicated it should last between 10 and 15 years. Responses indicate that an interval of either two (20.7%) or three years (39.0%) between survey waves was preferable.

8. Scoring criteria

Political desirability

A large majority (73.2%) considered that a cross-European longitudinal survey was desirable, with 55.6% considering it to be very desirable. When considering the political feasibility of a longitudinal child and youth well-being survey, respondents largely agreed with the following statements:

  • It is important that Member State governments are supportive of the survey if it is to be implemented (99% agree, of which 59.4% strongly agree);
  • Demonstrating that the survey will have continuing policy relevance for many years to come will be important if the survey is to be implemented (99% agree, of which 56.8% strongly agree);
  • It is important that significant research groups in each Member State are supportive if the survey is to be implemented (96.3% agree, of which 45.5% strongly agree);
  • It is important that significant NGOs in each Member State are supportive of the survey if it is to be implemented (89.9% agree, of which 31.9% strongly agree).

Technical feasibility

About half of the respondents (59%) consider the cross-European longitudinal survey to be technically feasible. There was a general agreement that it is feasible to build a longitudinal survey from a pre-existing Europe wide cross sectional survey. A higher percentage of respondents working in research/ academia (38.7%) indicated that this was not feasible than respondents working in policy (28.1%) or NGOs (31.6%). There was no strong agreement about the feasibility of linking a longitudinal survey to pre-existing national administrative data. Whilst respondents working in policy or NGOs considered this feasible (63.9% and 61% respectively), those working in research indicated (53.6%) that it is not feasible.

Obstacles to developing and implementing cross-European longitudinal research instruments identified in the first questionnaire include resources (cited 95 times), European Union and national governments' support or interest (cited 50 times), challenges pertaining to methodology (cited 132 times), differences between countries (cited 88 times), implementation (cited 57 times), data analysis (41 times), access to particular groups (cited 9 times), and utilisation (cited 6 times).

In the second questionnaire, respondents indicated that the following criteria were feasible: on a scale of one to five where five is very feasible: building child friendly questions that allow adequate comprehension and suitable response modes (4.28) translating the survey into European languages that will result in robustly comparable international data (4.20), to achieve a representative sample survey in each country (4.12), managing a large and complex data set (4.09), and to obtain children and young people's informed consent (3.92). Some items scored slightly lower such as: implementing consistent fieldwork practice (3.79), to keep the sample members in future data collection phases (3.66), and getting parental consent (3.62).

Financial sustainability

Over a third of respondents (41%) considered the survey to be financially sustainable. Yet, the majority of respondents indicated that improving data coverage is somewhat more important than constraining the cost. This is especially the case for respondents working in research (61.1%) compared to those working in policy (52.4%) and NGOs (50.0%). About a third of the respondents consider that improving coverage is top priority. Furthermore, experts strongly agreed with the following points:

  • the economic benefits of improved well-being far exceed the cost of a longitudinal survey (93.8%, of which 50.6% strongly agree);
  • a longitudinal survey will support policy makers to improve policy design and impact and make policies more efficient (97.9%, of which 58.6% strongly agree);
  • evidence gathered by such a survey will contribute to improved well-being (95.6%, of which 45.3% strongly agree).

About half of the respondents agreed with the following sentence: "The economic value of improved wellbeing far exceeds the cost of implementing the survey. However, these economic benefits are not easy to demonstrate (i.e. they are not 'cashable') and therefore cost would be a barrier to implementation." This was especially the case for policy makers (57.5%) compared to researchers (48.9%) and those working in NGOs (47.6%).

The various strategies identified by the experts to demonstrate policy relevance, draw key decision makers in the process and demonstrate the economic benefits of a well-being longitudinal survey to potential funders are quite similar. They include the following recommendations: clearly identifying specific policy question the survey will address, demonstrating the need for evidence based decision making process, involving the media and increasing public awareness, undertaking cost analysis (SROI, cost of reinsertion vs early intervention, cost of no survey), and providing a sustainability plan.

Language Option