Introduction to Teaching Quality Control
The end of semester survey of students has become ubiquitous in tertiary education. The literature on its use goes back the 70s. That’s just the majority of the work too; there is some research into the use of surveys for teaching from the early 20th century. With 40 plus years of serious research effort it would be safe to assume that the system used to measure teaching quality and help improve it is well established and well developed. Therefore, if an academic knows how to use and respond to these surveys, then that academic can’t help but become a better teacher. The only gap then would be how to use the current survey system properly.
Unfortunately, the current survey system is incredibly far from being remotely close to a good quality control system.
The majority of research into the development of surveys for the measurement of teaching quality has been conducted within the context of assuming that statistical properties such as repeatability and consistency are the end goal. As a result the actual requirements of a good quality control system have been ignored.
It also seems that quality control and improvement is not actually the focus of the current system. The results of the surveys are used for numerous purposes:
- To help students choose subjects
- To influence the promotion of academics
- To provide an overview to the administration of how different study areas are performing
- To compare faculties and teaching institutions
I am not arguing that any of these are bad to do. However, when you try to develop one survey to do all of these things, you are unlikely to do all (if any) of them well. It will certainly make quality control (and improvement) more difficult to achieve.
Why is it Like This?
As I mentioned above, the research into teaching quality really took off in the 70s. This was before the quality revolution of the late 70s and early 80s where new quality principles developed in Japan flowed to the rest of the world. It was first in manufacturing and then it spread to services and other business areas. However, it seems that because research into teaching quality started before the quality revolution, the area developed without input from other areas that were developing quality systems. The end effect was that teaching quality took a very different path and because it was a much smaller community, did not benefit from the larger pool of ideas and research findings. The end effect is that when it comes to quality, teaching has been left decades behind the rest of the world.
What Needs to Change?
The biggest flaw with quality control in teaching is the focus on time. Many assume that having a single survey at the end of the semester, when the students have had a chance to see the whole subject, is the best time to ask them their thoughts. However, establish quality control theory says that a quality outcome can only be achieved by tracking the state of the respective system all the time. If you wait until the end, then you have simply produced a bad outcome (probably) and let it go out into the world – it’s too late. Instead, you must ensure that each step taken to bring about the desired outcome is executed correctly to ensure that the entire process is working correctly. Then the desired outcome (in this case a well-educated student) can be expected.
The next biggest flaw is the idea that all data collected must be tabulated and recorded for distribution to all members of the organisation. Much modern quality control is within the system being controlled and the operator(s) of that system. This was not easily achieved. Many factory mangers dislike the idea of a problem eventuating, that problem being identified by the machine operator, then being fixed by the operator, and the factory manager never knowing. The system works well, but many fear ignorance. This seems to have carried over to academia. Academics are not given the chance to see that an issue is forming and a chance to correct it. Instead, the results of surveys are often withheld until assessment is completed with not chance to correct the issues.
The third flaw that I will talk about is the focus on what is measured. Teaching quality control efforts have noted correctly that it is the inputs to the system that need to be measured and controlled to ensure a good outcome. However, whereas in technical system the inputs and output are easy to identify and relate, in teaching they are not. The end effect is that the typical quality control tool in teaching is large and tries to measure anything that could affect teaching. Ideally, the quality control system would be adaptable, and able to find the issue that needs to be dealt with at the time.
The final flaw to be covered in this article is that of feedback. To ensure quality is achieved, the effects of changes need to be quickly noted. Otherwise, there might be wasted effort or overshoot. Because of the large time delay mentioned above and that improvements are tried on a different cohort from the one the raised concerns, there is no real feedback and it takes too long anyway.
In summary the current system needs to operate more frequently, the data must be primarily for the academic so that they can improve, the measurement must be robust and academic needs to be able to try responding to findings to gain feedback.
Taguchi/Total On-Line Quality Control (TOLQC)
In my experience, the most advanced philosophy in quality is that of On-Line Quality Control put forward by Genichi Taguchi. The philosophy is that control should be conducted while a system is running and not after the system has completed its task. That way, any issue can be identified early and then corrected for before they become problems.
For example, instead of molding a part and then measuring it, the temperature of mold and the properties of the material would be measured before moulding even starts. This practically guarantees a perfectly molded part.
This makes sense in a technical system, but it seems more difficult for something like teaching. However, we can still try to apply the basic principles and come up with a better system.
On-Line Quality Control in Teaching
The basic system
A better frequency to measure student satisfaction with teaching is after each basic cycle (say lecture then tutorial). This is typically weekly.
The questions that are best to ask are:
- Do you like the teaching staff? This is actually unimportant, but it’s good to get that out of the way for later.
- Do you like the subject? This is not very important (not everyone likes everything), but it is good to get it out of the way too.
- Are you satisfied with the teaching quality? This is important; it tells you if the students think you’re actually doing your job right regardless of whether they like you your colleagues or the subject.
- What would you like kept the same? This question is important because it allows you to work out what you are doing right, and not accidentally lose it when making changes.
- What would you like to see changed? This question is a good guide for what to change.
The first three questions are simple yes/no to keep things moving fast. The last two are optional worded questions to find that balance between insight and speed.
Implementing it
By asking these same three questions each week through an auto renewing online survey, you can quickly see what students have issues with. You can then respond to try to improve issues.
The essential aspect of the attempt to improve is to communicate this to your students. This has two benefits:
The students see that you are trying to improve your teaching and their learning. This will be greatly appreciated regardless of the benefits that come from those efforts.
The same students can respond in the next survey to confirm (or not) if you have understood what they raised initially. Often you will find from this feedback that there was a misunderstanding. By correcting that misunderstanding, you will be better able to improve your teaching quality.
The better feedback
The feedback that results from this approach to teaching quality control goes multiple ways.
Not only do you realise that you might misunderstood your students, but you will also find out when your students misunderstood you. You will at times find that the issue is not with the subject, the students’ ability, your teaching method or the anything like teaching material. Instead, it will sometimes be a perspective. Because the students can come to a subject with a certain perspective about what it is meant to offer, they will find the teaching incongruent with the subject matter. This perspective becomes very clear with comments left in the early stages of the subject. By correcting these earlier, you will find that students can adjust their position and learning style to suit sooner, and get more from the subject.
If you are like me, and make the results (ratings and comments) available to the students, then you will also find that students will give feedback to each other – anonymously of course. When students start sharing their thoughts on each other’s positions, they will give deeper thought to their own attitude, expectations and abilities. This can encourage reflective learning and self-regulated learning. Both of which will produce better learning. Each week one student will make a comment about another student’s comment from the previous week – providing multiple perspective on the subject, what it offers and how to get the most from it.
The confronting nature of the feedback
Chances are that you already know what feedback can be like for teaching. In a large group, there is always someone very negative and keen to share their negativity. This can often be handled when it is in a confidential report summarising the results. However, if you choose to share the results so that you can better articulate to your students how you will be adjusting the teaching, then the results will be seen by the entire student group. Other students might take a lead from this or they might counter it. It’s hard to predict and the experience can be a confronting one that you need to be ready for.
Getting the most from on-line quality control
You should try on-line quality control at least for one semester. Even if it is the first time you should still try to get as much as you can from it to give it a fair trial. I have been using this system for some time now and have found the following helps get the most from on-line quality control systems in teaching:
- Share results – it makes the system look more genuine and lets students see other perspectives, which helps. I do this each cycle (weekly) and recommend you do to.
- Each week respond to the issues raised – this shows appreciation for the time the students took to fill out the survey. I make an effort to thanks them.
- Remember that sometimes your response can be advice – there have been times when I had to reiterate the nature of the subject and suggest that students try a different approach.
- Use the language used in the survey and respond with your understanding – at times you will not understand what was meant. When this is realised, students can raise the issue again in a different manner (this helps improve their written communication skills).
- Hang tough – as I mentioned above, this can be confronting. It is up to you to show that such criticism can be taken on board by an adult and used to bring about improvement. I have had to set such an example for tutors as well.
- Respond appropriately – I do at times get students having fun with the system. I have been told that I look a bit like Duke Nukem and that some students like my Manga t-shirts. My take is that if students are in a good enough mood to write such things, then the subject is probably going well. You can go along with that fun in any way that suits you. There is also the other extreme. When students have simply been personally abusive of any teaching staff I have called it how I see it. I do however keep in mind that some people think that they are simply expressing an opinion, and at times I have found that I need to be careful about approaching negative comments to extract what is useful and ignoring the rest. The most important thing is that you do not let negative comments induce negative emotions or behaviour within you. It’s all an opportunity to improve.
- Follow through in some way – it’s not always easy to do what students seem to want or what you think might be required to alleviate an issue. Do what you can, say that you have (in case students do not realise) and always ask for more feedback or even suggestions. I recall one case when a student said that just a few more images in the presentations would make it easier to pay attention. I tried it and it worked – I still keep this in mind now.
- Remind students – when things are going well, students can be less motivated to fill out the survey. This is especially so when there are assignments due. I send a reminder email with links to the survey and results each week.
- Keep it anonymous – students become suspicious if they think that there is a connection between you and the survey system. Ideally, use an external website to run it. If you do so it in house, then try to avoid using student identification as a means of access.
- Choose a short enough interval between surveys – a week is good unless you run in block mode or something like that. Just remember, that the key to success is speedy response and that means relatively short intervals.
What improvements can be expected?
I have found that on-line quality control in teaching can help a lot when you are unaware of the real issues that students face. However, it will at times simply allow you to confirm that what you’re doing is OK. Typically though, I have found that it often reveals a conflict between the learning needs of different groups of students. It is good to discover this, and on-line quality control can help you find the nature of the conflict. But it is then up to you to work out a solution to it to try.
On-line quality control in teaching can also provide a very rich source of data for action based research or other types of research in education. Thus, it can also help improve the efficiency and effectiveness of education research.
How can this be most easily tried?
As I said above, you should try on-line quality control at least once to see how it can help you become a better teacher. You can try doing it manually yourself or with an online teaching system that you already have. However, I have found this to be time consuming and cumbersome. You really want a system already set up for such a task. Look for on-line quality control websites on the internet. You should be able to find a free one that is easy to use. I personally use http://www.thequalityprofessor.com.