Attitudes of providers to the ‘DEF’
XXXXX XXth 2007
Mr Cole Davis, Psychologist
Contents
1. Introduction
1.1 The purposes and context of the report
1.2 Methodology
1.2.1 The online survey
1.2.2 The interviews
1.3 Reporting
2. The questionnaire, with commentary from the interviews
2.1 Questionnaire Section A: the organisations
2.2 Questionnaire Section B: respondents’ use of the DEF
2.3 Questionnaire Section C: How the DEF is used in the
respondent’s organisation
2.4 Questionnaire Section D: Development Resources
2.5 Questionnaire Section E: Communication of the DEF
2.6 Questionnaire Section F: the inclusion of Fails
3.1 Clear positives
4. Stakeholders and the DEF
4.1 Different sector opinions
4.2 Fails
4.3 Chances Charts
4.4 Recommendations
5. Overall Conclusion
5.1 Surface issues
5.2 Clarity of function
5.3 Stakeholders and the DEF
Introduction
The purposes and context of the report
This report reports on a project designed to identify the strengths and any possible shortcomings of the DEF, including making recommendations stemming from the findings. The project was conducted in two stages: an online survey of users’ views examined levels of providers’ awareness, understanding, participation and implementation of new measures, with particular regard to the DEF; subsequent interviews with DEF users examined in more depth the usability and accessibility of the DEF, its likely impact, the Quality Improvement Pack and areas of potential improvement. The objectives of these investigations form part of the overall aim of assessing the effectiveness of the software and the impact of the New Measures.
Methodology
q The online survey
The survey responses derived from both an earlier, lengthier edition of the online survey on the DEF and a considerably shortened subsequent version. The last contribution by a provider was made on XXXX XXth 2006.
An online survey was completed by providers of further education and work-based training. Most questions were responded to as ratings, although text boxes were also made available for comments on pertinent issues. Respondents were asked if they were willing to participate further: many of the subsequent interviews were based upon positive responses to this question.
There were 294 responses. These were constituted of 261 fully completed responses to the new version of the online survey, 29 fully completed responses to the older version and 4 partially completed older responses. The rationale here was that of the 35 questions to be found on the new survey, only 3 were incompatible with the lengthier previous edition. Given the length of the earlier survey, it also seemed reasonable to include the longer partially completed responses from that edition. Partially completed responses were not included, however, from the DEFer, shorter, version of the survey.
All responses were made online, with the exception of a word-processed copy of the questionnaire, completed by a respondent with visual impairment. This response was transcribed onto the relevant software.
q The interviews
81 interviews were conducted with DEF users during December 2006. All except one were arranged as telephone discussions over 30 minutes, using open-ended questions, but with a series of prompts for the researchers to ensure that a variety of topics were covered, including possible solutions to issues which had been raised (see the appendix). The exception was a sixth form college official who made a special request for a visit: the practical exposition and subsequent discussion proved to be of great value.
Where possible, interviews were arranged with individuals who had completed the online survey and had expressed an interest in participating further, but interviews were later extended to DEF users who had not completed the online survey.
Reporting
The first section of the report is structured according to the (second edition) online survey questions, based upon responses to the survey, with interview responses providing some interpretative commentary. The second section of the report, Planning for Action, as well as containing unused interview material, focuses much more on the issues considered in depth by interviewees as well as a providing a general discussion of the likely way forward given the nature of the responses. Overall conclusions and recommendations will follow.
The online survey data results are largely treated in a quantitative manner. For increased clarity, cross-sector analyses from the online survey generally refer to subsamples of 30 or more providers; the reasons for this will be made clear within Question 2.
Questionnaire Section A: the organisations
1. Geographical spread
There were 294 online survey responses. On the whole, the spread of providers was representative of the general population; the sparseness of the North-eastern representation, for example, was reflected in the census proportions. Only the London area appeared to be considerably under-represented in this survey.
The
spread of the subsequent interviews constituted over-representation in the East
and under-representation in London. In general, however, the geographical
proportions appear similar to those in the survey, the apparent
under-representation of the north-east being an obvious instance.
2. Organisation type
The three major sectors, Work-based Learning, General Further Education Colleges and School Sixth Forms are evenly represented in the online survey, with over 70 providers each, comprising 78% of the survey sample. As Sixth Form Colleges provide over 30 responses, it is reasonable to use this category as an analytical sub-sample.
Specialist colleges seem reasonably represented, especially bearing in mind that some colleges (e.g. those working with students with learning disabilities) are unlikely to use the DEF. Lack of use is almost certainly the reason for the scarcity of Adult and Community Learning Providers (see question 4). The ‘Other’ category consists of local government representatives (10), higher education providers (2), consultants (2) and planning bodies (2).
Subsequent to the survey, 81 interviews were conducted, with representatives of 28 general FE colleges, 14 school sixth forms, 13 sixth form colleges, 23 WBL organisations and 3 specialist colleges. As will be noted from the chart, sixth form colleges are somewhat better represented in this stage.
3. Is the respondent a user approver for the Provider
Gateway?
Respondents in all main provider types were predominantly User Approvers; this predominance was a statistically significant effect (chi square, p < .01).
Questionnaire Section B: respondents’ use of the DEF
4. Have respondents used the DEF?
A sizeable minority of respondents have not
used the DEF. As will be noted on the
right-hand chart, this minority is largely situated within schools and
work-based learning providers. All interviewees were DEF users.
5. If yes, when did providers first use the DEF?
It should be noted that many responses were rather vague, e.g. ‘when it first started’, ‘in the piloting’, ‘earlier in the year’. The breakdown used above represents early users, those who used the DEF in the summer of 2006 and more recent users; the last category will have used the newer version of the DEF.
6. When did you last use the DEF software?
As will be seen, the majority of users have either used the DEF recently as a function of their work or, in the case of the most recent category, have looked at the DEF when stimuDEFed to do so by the survey.
Interviewers found that interviewees who tended to use the DEF less often were less clear about the criteria used for the exclusion of types of learner data. This DEFter problem applied across all types of provider.
7. The usefulness of reports produced by the DEF
There is a generally positive overall reaction to the Summary Charts and the National Comparison Graphs: those selecting one of ‘Very Useful’, ‘Useful’ and ‘Some Use’ comprised 79% and 72% of the respondents for the two resources respectively.
Opinions are rather more divided over the value of the Chances Charts and ‘Other’ charts, 67% and 60% of the respondents selecting the three more positive options.
It should be noted that Work-based Learning respondents were much more likely to respond negatively to the DEF reports. 37% of WBL respondents chose ‘Not much use’ or ‘No use at all’ pertaining to the Summary Charts; 35% for the National Comparison Graphs; 61% for the Chances Charts; and 58% for the ‘Other’ charts.
The interviews uncovered a similar pattern. The DEF reports were generally well-received. Chances Charts were the most talked-about of the report types. Sectional differences were noted, however: WBL interviewees in particular found charts less than readily comprehensible.
8. Respondents have looked at:
A cross-tabulation of this data with sectors confirms the initial impression from the chart: most respondents were more likely to look at both Value Added and Distance Travelled Reports. However, a sizeable minority of School Sixth Forms (42%) and Sixth Form Colleges (35%) would only look at Value Added Reports. Similar, only 41% of Work-based Learning Providers were likely to look at both sets of reports. Essentially, providers with a strict emphasis on academic or work-based qualifications were more likely to consult only such data as is perceived to be relevant to them.
Data files were likely to be looked at by a minority of respondents in all sectors. These were particularly small minorities in the cases of WBL looking at VA data files, and schools and SFCs looking at DT data files. As before, relevance is very likely to be the explanatory variable.
Interviews tended to accentuate the trend: at present, the value added data seems to be having much more impact in colleges than distance travelled. One respondent cited a recent inspection (June 2006) in which inspectors appeared reluctant to use DT data. One provider reported difficulties in gaining access to DT data. Missing DT data limited the value of the system to providers in cases where the majority of learners pursued ungraded qualifications; some respondents said they had encountered difficulties in resolving this issue with their local centres.
9. The usefulness
and user-friendliness of the DEF reports
The majority of respondents (51%) were positive about the reports (Very useful or Quite useful), although almost 20% were negative (‘Not much use’ and ‘No use’).
Again, Work-based Learning respondents were most likely to react negatively (37%).
Interviewees were similarly positive about the overall functionality of the DEF, although this did rather vary across the providers sectors. Usability, however, needed to be substantially improved. The details of both of these issues will become evident.
10. The usefulness of the DEF tutorial
39% of respondents found the tutorial useful (‘Very useful’ or ‘Quite useful’). 44% were neutral, with 17% taking a negative stance (‘Not much’, ‘No use’). One respondent commented that the tutorial was unnecessary for the purpose of running reDEFively uncomplicated software. This may explain the tendency towards the middle of the scale.
11. The number of
people in the respondent’s organisation
using/looking at the DEF output reports?
In general, between 2 and 5 users are likely to use the output reports. General Further Education Colleges are most likely to have shared the reports with ten or more people (45%). Work-based learning organisations are most likely to fit into the sole user category (27%).
Interviews tended to follow this trend of either sole or few users. There was a tendency for distribution of information to be a centralised affair.
12. The ease of finding necessary information from the DEF
Respondents appear to be evenly divided about the ease of finding information, with most opting for the middling ratings. There are negligible differences between the sectors. All the major sectors are evenly divided between the middle ratings, with most veering slightly towards ‘Fairly Easy’; WBL was slightly inclined towards ‘Sometimes Difficult’. This later finding was accentuated during the interviews, although the sector differential may have been based on broader issues.
13. Suggestions for improving the usability of the DEF
There were 289 responses, 98% of the online survey sample.
Usability
Printability: Printing problems provided the most comments (34; 12% of the 289 respondents commenting on this question). The ability to print to a single page of A4 size paper was considered important as a potential improvement. One interviewee suggested a ‘Printer Friendly’ button to reset the display for printer formatting.
Graphs, and more specifically, chances charts (to be discussed under usefulness) were seen as particularly in need of improvement. The interviews indicated a tendency for only one person in an organisation to use the DEF, distributing reportage to heads of departments and others; this clearly needs bearing in mind when improving usability.
On screen display: Similar to the above A4 problem was the request from online users to have all of a given page displayed on screen (7; 2.5%); interviewees also suggested that a failure to centre on the screen and a need to manoeuvre the display was of concern. Interviewees also indicated that greater use of colour would improve visual appeal (this does apparently lead to a difference in perception, currently in the favour of the DEF’s commercial rivals).
Several interviewees reported it as being less user-friendly than most software used in their jobs: “clunky”.
Downloading problems and the exporting of data: Some respondents reported downloading problems; the ability to download with ease to different media (e.g. laptops) was also requested. While some recognised the use of being able to use different formats, others felt that this was not an easy process and that the image quality in the exports could be improved. An interviewee suggested that it is not currently clear that files can be downloaded as Excel spreadsheets: this method
should be a little more prominent.
Cutting and pasting: The ability to cut and paste reports (for example, to use in Self-Assessment reports) was also suggested by a few. Interviewees suggested improvements to the use of PDF files: security properties may be set to allow the easier copying of images while preventing the editing of original documents. To allow easier demarcation, bookmarks could separate out different qualifications within documents. In terms of improving choice of output, one useful change would be to simply add an <ALL> selection to the subject drop down list when exporting to PDF. Another suggestion is for a multiplicity of PDF files so that selection can be made.
Accessibility: This appeared to be an issue in the online survey. In some cases, this pertained to passwords. In others, this was not explained. Links did not always work. Interviewees, however, suggested that considerable improvements have already been made in this area, although links still appear to need checking from time to time.
Readability
Larger font versions: One respondent suggested larger font versions for visually impaired respondents. This is important to ensure compliance with current legislation.
Over-complex data: A large number of online respondents (25, 9%) felt that the data was ‘too complicated’ or that there was too much of it. Similarly, while interviewees from a technical background were positive about the graphical outputs, others found some displays to be “baffling” or “incomprehensible.” Suggestions arising from their comments included simplifying and condensing information, including that which is both complex and not particularly useful. Two respondents cited confidence intervals in this context; these were also discussed in some of the subsequent interviews. One respondent referred to written explanations as well as graphical representation, two suggested referencing by colour, with yet another referring to rollover/popup description of links. Training, CPD and dissemination events were both recommended in view of problems interpreting data and because of frequent albeit welcome changes to the system.
Usefulness
Accuracy of data: This was frequently referred to (27; 9 %). Half of these responses were not specific. The majority of those who did comment further referred to missing learners’ data – often as a result of systematic exclusion such as the criteria for Value Added – or poorly matching cohorts. In the latter case, actual errors were implied (“ensure only our data is on the system!”). An occasional point was that mistrust of the underlying data undermined what would be very positive attributes.
Many interviews were satisfied by the robustness of the data, but a substantial minority expressed concern that learner numbers recorded on the DEF were different from their own records. As was noted, it was found that the latter tended to have used the DEF less often or were less clear about the criteria used to filter learners out of the DEF, but it should also be noted that this problem applied across all types of provider.
Interestingly one interviewee commented that even if a few learners were missing the key messages provided by the DEF could not be gained elsewhere. Some interviewees commented positively on the fact that the data were nationally comprehensive and consistent, and would be used by all the main players, including Ofsted. The capacity of the system to enable the LSC and inspectors to examine the same data in real-time was seen to be a major advantage of the system. Some interviewees, however, cited inspectors visiting institutions and concurring with the belief that there are substantial needs for improvement in the data.
Timeliness of data: This was also a common concern (18; 6%), and a principal concern amongst interviewees. Most responses specifying a preferred deadline recommended September. October was mentioned, especially by interviewees, but this was very much a minority view from the perspective of the survey.
The inclusion of Fails: Inclusion was recommended by some respondents (9; 3%).
An error noted on chances charts: One respondent reported an overlap in prior attainment ranges (34-40; 40-46). Given the possibility of individual scores of 40, this was a source of potential confusion if not inaccuracy. It was suggested that the use of ‘greater than’ and ‘less than’ may make it easier to delineate categories (although in fact the same system could be used if overlaps are looked for).
Greater ability to use the information that is available: Some respondents felt that the functions could be improved in this respect. Chances charts could examine chances for all bands rather than just one at a time and could show multiple learners in one report rather than producing hundreds of reports. Learners’ names could appear at the bottom of pages. Again, interviewees put these areas for improvement into the provider context: adaptations need to bear in mind centrally distributed information. Tutors seem to want to either use the same chart as a reference for discussion with a range of learners or actually show charts to whole groups of learners. In either case, individual graphs are unwieldy.
Broadening the scope of the data: Generally to improve comparability, this suggestion emerged in different suggestions. Organisations such as Xxxxxx, local authorities and consultants were inclined to want information compatible with that at county level, and wanted to be able to consider the performance within localities, local authorities and at consortium level. Consortia were inclined to be critical of some providers’ data being available to them and others not. Two respondents were critical of the exclusion of 19+ data.
Subject level data: A few suggested that it should be made easier to find subject level data (for example, without having to select areas first).
More variables: One suggestion was that comparisons should be made with more variables, for example gender, ethnicity, SEN.
Trend data: one interviewee suggested addition of a facility to display trend data.
Miscellaneous: interviewees also commented upon:
· Statistical validity is questioned by some.
· The mis-recording of learner names during their learning experience which made use of the DEF difficult. One provider reported this as a potential risk area with less familiar names.
· Learners who undertake a one year course before A levels are often excluded from the system.
· Overseas learners, particularly on science A level programmes, predominate in some colleges, and a tutor might only see 2 learners on the DEF from a whole class.
· Providers targeting young people disengaged from learning. Many learners apparently begin 2-year programmes at the age of 17.
· Many learners come from overseas or are over 19 years of age, and therefore baseline data is missing.
· Data from small class numbers do not appear on the system.
· Limited inclusion of qualifications.
Questionnaire Section C: How the DEF is used in the respondent’s organisation
14. Have respondents used the information in DEF reports?
The majority of the respondents had used the information. Within the sectors, this situation only reversed amongst Work-based Learning respondents. No ACL respondents have used the DEF information.
Respondents who said that they had used the information were then asked to continue onto the other two questions (16 and 17) in this section. The others moved on to the next section.
15. How would respondents’ organisations be likely to use DEF outputs in the future?
Setting targets with students seems rather less popular than other options. This may be reDEFed to the reDEFive lack of popularity of the ‘Chances Charts’ noted previously.
Only the GFEs were more likely than not to use the DEF for Quality Improvement (62%). A minority of School Sixth Forms, SFCs and WBLs were likely to do so (38%; 27%; 33%). Specialist colleges were evenly divided. A very similar scenario applies to Self-Assessment. Only GFEs (57%) were more likely than not to use the DEF for Inspections. In other sectors, a sizeable minority were likely to use the DEF for this purpose.
Interviewees generally valued the functionality provided by the DEF.
“It’s added that level of sophistication that’s been needed for a long time.”
Most providers who have made significant use of the system found the reports simple and easy to understand. Several providers commented that the availability of comparison charts for level 3 programmes had excited curriculum heads and tutors.
Work-based providers valued access to timely and overall success rates. Providers also commented that the DEF counterbalances the weakness of simply relying on qualification success rates (QSR).
Interviewees from further education colleges were generally supportive of the DEF, but concerns over data and usability mean that considerable improvements are necessary before the DEF has widespread acceptance. On the whole, sixth form college representatives were highly critical of the DEF’s current overall function and the timeliness of data; most of those using ALIS or ALPS were content with these systems, but suggested a potentially huge impact for limited purposes. There was some guarded support from WBL who would like more information, preferably of a non-technical nature. Specialist providers, for example land-based, find that qualifications with small numbers nationally are not showing up on the system. In some cases qualifications are not being identified and others the numbers are so small that national comparators become unusable. In one case quoted there was only 1 learner identified out of 141 known.
Interviewees generally conveyed the impression that they had not yet explored the full range of facilities available via the DEF – and certainly not to the extent of producing their own customised reports, over and above the standard DEF outputs.
The main demand for additional functionality related to the capability to display trend data.
A number would wish to have access to software or algorithms to check data and manipulate to create what–if scenarios.
Of the different tools, however, interviewees considered Chances Charts to be the over-riding issue…
A radical change in proportions was observed for ‘Setting targets with students’. GFEs were now evenly split over usage. Other sectors are rather less likely than they had been to use the DEF for this purpose, and only 9% of WBL respondents were likely to use the DEF for this purpose. WBL, however, is likely to differ from other sectors in principle, given its involvement in vocational qualifications; target-setting is likely to take a rather different form.
Chances charts were a major issue for interviewees, particularly in sixth form colleges. These are viewed as useful tutorial tools for discussing and agreeing target grades with learners. The inclusion of prior attainment data in the DEF was identified as a strength by many providers as it removed the need to rely on learners bringing their exam results into school. Having said that, extremely broad banding of attainment and predicted achievement can be problematic: for example, 55% of the high-performing learners at one college were subsumed into one band, resulting in little meaningful discrimination.
Several interviewees suggested that the prior attainment calculator should be available without having to log into the provider gateway, 'it is of little use to me as a Quality Manager, yet very valuable to tutors'. Other providers reported that downloading a full summary of the data for 05/06 was no longer available.
16. Has the respondent’s organisation been inspected in the past 6 months?
A sizeable minority had been inspected; this proportion was roughly reflected across the sectors.
17. Did the VA/DT data help the organisation to prepare for inspection?
The majority responded negatively. Given the greater number of respondents than those who had been inspected, it would appear that some respondents have answered in terms of intentions for future inspections. The proportion of the response is roughly replicated across the sectors. One possible explanation emerged from the online survey’s qualitative answers was that the DEF was still not accepted by providers or inspectors. The latter viewpoint was supported by some interviewees, who spoke of inspectors visiting institutions and concurring with the belief that there are substantial needs for improvement in the data.
Questionnaire Section D: Development Resources
18. Have respondents contacted the LSC for support and
guidance in using the DEF and VA and DT?
19. If yes, how helpful was the support?
Of the large minority who contacted LSC about the DEF, 45% chose one of the two negative rating options, with 55% feeling that LSC did provide meaningful support.
Interviewees point out, however, that there are times when an actual person is required to explain more difficult issues. Where providers had experienced problems with data or needed further guidance they were not always able to obtain satisfactory responses from XXX. According to respondents, direct contact has been handled by local centres, where many valued contacts appear to be no longer available. Those members of staff who are available are often viewed as lacking in knowledge of New Measures and expertise in the usage of the DEF. Often messages have not been understood or, at least, there has been a lack of cohesiveness when passed on to colleagues. One respondent referred to the value of the XXXX helpdesk facility.
20/21. Did respondents receive a hard copy of the Quality Improvement Pack? Those who did rated the pack overall ( others moved to the next section).
Of the main sectors, School Sixth Forms were least likely to have received a hard copy of the QIP (35% did). None of the four ACL providers had received a copy.
83% of those receiving the Quality Improvement Pack were positive about it overall. There were no prominent differences among the sectors. A small proportion of interviewees were unaware of the QIP’s existence. There was also a lack of awareness of the self-directed learning PowerPoint presentations on the LSC website.
22. Frequency of use of the Quality Improvement Pack
66% of the respondents were fairly regular users (taking in ‘Some Use’, ‘Frequent Use’ and ‘Very Frequent Use’). WBL and GFE respondents were more likely to be regular users. The interviews indicate that specialist users had read the QIP, but it was not used by colleagues.
23. User-friendliness of the Quality Improvement Pack
There would appear to be a positive reaction to the user-friendliness of the Quality Improvement Pack. The existence of a sizeable minority responding negatively suggests, however, that this perception may prove to be far from universal. The online survey did not detect great differences across the sectors.
A WBL interviewee expressed the opinion that it was insufficiently user-friendly for staff, especially the less numerate. Many interviewees highlighted the need for more accessible and shorter guidance for practitioners and managers. It was also argued that the pack covered too many audiences.
24. Accessibility of the Quality Improvement Pack in
terms of readability
Again, the majority (72%) are positive about the DEF’s readability, but 26% consider the accessibility to be ‘Poor’. SFCs and WBL providers in the sample particularly likely to take a negative view; these are half of the SFCs and a third of WBL.
Interviewees generally viewed the Quality Improvement Pack (QIP) as an understandable document. Respondents complained that it was lengthy and rather repetitive, one requesting an executive summary and navigation aids. Various providers, however, pointed out the need for helpline back-up for detailed technical enquiries and, in the case of WBL providers without specialist data personnel, possibly more basic assistance. WBL respondents tended to find the terminology and ‘mathematics’ in the QIP to be incomprehensible.
25. Suggestions for improving the Quality Improvement Pack
There were 59 responses, 20% of the 294 respondents. Of these, 9 were either complimentary or non-committal about the Quality Improvement Pack. As, therefore, only 50 of the respondents (17%) had suggestions for improvements, it seems reasonable to say that there is a high level of satisfaction with the pack.
Overall trend: From the sub-sample who did see fit to comment, the overall trend is quite clear and this section reflects the relative uniformity of the opinion. The pack is considered by some to be too detailed, too long and to contain too much technical jargon; possibly aligned to this criticism is the perception of a few that the writers were not necessarily very conversant with the learner and consortia contexts. The interviewees tended to be of similar opinion. One online respondent said that the concepts of the new measures themselves were too complicated and preferred to discuss matters with ALI inspectors in terms of ‘starts and outcomes’.
Content: In terms of content, it was suggested that the pack
could be more practical, with teachers in mind. Purpose rather than function
was the guiding principle: examples could be given of what could practically be
done, for example in tutorials, subject review and target setting. Tutorials on
the reading and interpreting of the data were also suggested. It was recognised
by some, however, that needs and abilities of readers rather vary; this is
reflected in the following suggestion for a radical change of medium.
Online versions of the pack: One or more online editions were proposed by some respondents. This was seen as offering considerable cost savings compared to hard copies; the adoption of another recommendation, the timely updating of the QIP in line with the DEF, would exacerbate the costs of hard copy. The possibility of providing different levels of user information was recognised as the strength of such a medium. Busy people with selective needs should be able to find what they want and print it. Those wanting interactive tutorials, case studies, examples and step-by-step instructions could find them. Individuals in need of conceptual explanations could similarly find these. More technical information could be sought by those requiring it.
Larger font versions: The need for larger font versions was also mentioned.
Questionnaire Section E: Communication of the DEF
26. The organisation’s level and scope of awareness of:
Similar responses occur in all four cases, with ‘Some Awareness’ predominating; the DEF appears to impinge somewhat less on the horizon. This response does not appear to be sector-specific.
27. Ways of accessing the information about the DEF
The most frequently cited sources of information about the DEF are the XXX’s web site, the Quality Improvement Pack and the Gateway Portal. ‘Other methods’, cited at the bottom of the chart, included forums and work groups, networking (e.g. with other colleges), consultants, and participation in the piloting of the DEF.
28/29. If you have accessed information about the DEF and
the new measures of success, please indicate information source and rate the
effectiveness of that source:
The web site of the XXX, the Quality Improvement Pack and the Measuring Success Newsletters are the most common sources of information on DEF and the new measures. It should be noted, however, that each of these is only cited by 48-59% of the survey sample.
Two of the three most common sources of information, the Quality Improvement Pack and Measuring Success Newsletters, are rated positively (‘Good’ or ‘Very Good’) by 79% and 82% respectively of the rating responses. The XXX’s web site is rather less highly rated, with 54% of the ratings being positive. ‘Other XXX publications’, along with the local centres and the national office fare similarly, with positive ratings of 58%, 51% and 55% respectively. National Office received the greatest proportion of ‘Very Poor’ ratings, standing at 23% of responses to that particular response item.
Workshops and meetings organised by the XXX fare much better; these and ‘Conferences’ each have positive ratings of 73%. ‘Other organisations’ and ‘Through other methods’ each have positive ratings of 71%.
Interviewees commented that some method needs to be in place to ensure that users are aware of innovations on the DEF. This is likely to be of even greater importance if some of the changes recommended in this report are to be implemented. This has implications for whichever are deemed to be the most important forms of communication, with effectiveness and popularity being principal factors.
30. Additional comments about the DEF and the New Measures of Success
There were 118 online responses, 40 % of the survey participants.
Duplication: Many of the comments made about the DEF repeated those made in Question 13. Concerns about the accuracy of underlying data was most prominent (24 responses; 20% of the comments made); this was also a concern of interviewees. Also appearing again were concerns about timeliness of the data (14; 12%), a major issue for interviewees; the need to include Fails; too much complexity and the need for training; problems gaining access to the DEF and some positive comments about it.
Retaining other tools for New Measures: New comments about the DEF included 10 (8.5%) expressing a desire to continue to use other tools (e.g. GHI). One respondent felt that inspectors did not recognise the DEF as a valid tool, another specifically endorsing a paper by Xxxx Xxxxx of Xxxx Xxxxx College which broadly criticises the DEF. As will be noted in the next questionnaire section, on the treatment of fails, SFCs seem to have a separate perspective. This SFC perspective, including a desire to continue with other tools, definitely emerged from the interviews.
Support shortcomings: Other comments complained of lack of support from XXX or local centres when approached about data or access problems.
Specific information requests: One respondent cited a lack of information at provider level within Work-based Learning. Another respondent suggested that there could be more information about JKLMN. Two respondents felt that General Studies should be excluded because of potential distortion.
Updates: A few providers asked to be communicated with about updates. Interviewees suggested that emails or other methods should be adopted, as people were not always aware of changes which they indicated would be likely to improve their confidence in a tool with considerable potential.
New Measures: Comments about the New Measures included a desire for greater coordination with other frameworks, or more compatibility between measures (one respon