Watch This Trend – Accreditation

New York Times has an article about what is going to be a growing trend, people getting formal accreditation for things they learned in a casual manner.

This will impact us as teachers as well as our students.

Educational institutions have held a monopoly on accreditation and before, even if you knew the subject better than the instructors and perhaps the instructors may have never had any practical experience as you may have had, you had to go through the schooling routine to get credit.

New York Times Article

Advertisements

If a good-looking good-talking phony rates high, is there hope for us?

Well it is the end of the school year. The school is asking the students for an evaluation of the teachers. Frankly it may be hopeless for many of us.

The problem is not that we failed as a teacher to help our students. The problem is that we didn’t make our students think we helped them. You see, making our students “think” Continue reading “If a good-looking good-talking phony rates high, is there hope for us?”

Action research: Mom and Dad and grammar

Spot quiz. Ready? What percentage of students, after nine years of English training, can use the correct pronouns in a few sentences about their mother and father?

Write down your answer.

Pencils down. Thank you!

If grammar teaching works, why does it take years for students to follow the simplest rule with accuracy?

Don’t try this at home! Try it in your classroom!

Without any reminder of the rules, ask your students to talk to you about their mother and their father and see how they do. The grammar rule on pronouns of gender cannot be simpler. Mom = she. Dad = he. We’re not talking about complex grammar rules. This rule takes less than a minute to teach and if you teach it and then test it, all of your students will pass the test.

They “learned” it. Why do they get it so wrong?

In December 2010 and January 2011, I gave an oral speaking test to 120 Chinese college students. As part of the test, I often ask the students to speak of a family relative. As part of the test this time I asked two questions about parents:

1. Tell me about your mother.

2. Tell me about your father.

Each student answered the request with about 3-4 sentences for each parent.

In the first sentence they always used “my mother” or “my father” but in the following sentences they used the pronoun of gender.

The students also filled out a form so I could learn how much English training they have had. They have almost all had the same amount of training, about nine years. Let me remind you, Chinese teachers are not shy about teaching grammar. Grammar is hammered into the students. Often the English instruction is given in Chinese. Extensive reading or other forms of extensive input is not promoted making this a more ideal situation to test the effectiveness of grammar teaching.

Considering nine years of training plus the simplicity of the grammar rule of gender, our students should be 100% accurate in usage. So how did they do?

Out of 112 students tested so far, 80 have called Mom a “he” and/or Dad a “she” one or more times during this test.

The question was: What percentage of students, after nine years of English training, can use the correct pronouns in a few sentences about their mother and father?

Answer: After nine years of English training, only 28%.

Some languages like French or Spanish have pronouns of gender. It is possible that it is easier for French and Spanish students of English to use “he” and “she” correctly but this could be more a matter of language transference than language acquisition.

If after 9 years of English training only 28% of the students can use “he” and “she” correctly, we must doubt the ability of learning grammar rules to lead to grammar acquisition and accurate grammar use.

>Speaking evaluations made simple

>This is a very complicated subject. It is not easy to conduct a speaking test but I will go over just a few things about it and touch on them lightly. There are many ways to do speaking tests. I have studied them, tried some of them and have settled on this way. It is similar to the way I was trained as an IELTS examiner with a few differences.

QUESTIONS

Design three levels of questions.

(1) Easy questions which are answered with straight factual answers. “Where are you from?” “How long have you been here?” “What did you do yesterday?” “What do you like to do on the weekends?” These questions make little demand on the student and only very low-level students will have problems with these.

(2) Moderately difficult questions demand more from the student. These are questions asking a student to describe a city or restaurant, relate the story of a movie recently seen or a book recently read. “Tell me about your last holiday? “Describe your best friend.”

(3) Difficult questions are those that require the student to give an opinion and justify their opinion with reasons. “Should students be required to wear school uniforms? Why?” “Should smoking be banned in all buildings? Why?”

Be aware that some questions are not only difficult to discuss in English, sometimes they are just plain difficult to discuss at all. I once designed a question, “If you had two weeks to live, what would you do?” This question was so deep that the students became extremely thoughtful in trying to give their answers to the point that it interfered with any attempts to show fluency. Questions do not need to be so deep.

Although the question may be difficult at times, to understand the question should be simple. Remember, this is a speaking test, not a listening test. For example, “Given the opportunity to go on a round-the-world cruise or participate in a scientific exploration in Africa, which do you think could potentially be more beneficial for your career development?” Many low and mid level students would not be able to understand that question and therefore would not be able to speak on it. Make sure your questions are easily understandable.

I like to let the students ask each other the questions. This way I can focus on listening and evaluating. But I do not allow the students to prepare for the questions except for perhaps just a couple minutes before the interview.

Look me in the eye? In western countries we have no problem looking into people’s eyes when speaking to them but this is something that Asians do not do. Therefore, when you conduct the speaking test with Asian students it is best to not try to look deeply into their eyes or to hold their gaze. Look elsewhere, shift your eyes around or even just focus on your band descriptors or rubric.

EVALUATION

You can use a rubric or band descriptor to measure the student’s level such as the IELTS band desciptors or the Common European Framework.

You will notice in the IELTS descriptors that at Band 4 it says:

“Is able to talk about familiar topics but can only convey basic meaning on unfamiliar topics and makes frequent errors in word choice. Rarely attempts paraphrase.”

and then at Band 5 it says:

“Manages to talk about familiar and unfamiliar topics but uses vocabulary with limited flexibility. Attempts to use paraphrase but with mixed success.”

That is why it is important to design your interview questions with easy, moderate and difficult topics so that the student will have to try to produce a full range of English at different challenging levels to respond accurately. The English of many students will begin to break down at the higher levels and this will allow you to see the limit of their English.

I put the band descriptors and all the students names on an Excel spreadsheet. I give the student a score for each rating catagory (Fluency and coherence, Lexical resource, Grammatical range and accuracy)
and the program averages it out into a final band score. Depending on the situation I will add formulas to work that score into a grade, average all the scores to compare one group with another or other things. Click on the picture (above) to see it enlarged.

ACCURACY

The more realistic the task is, talking naturally about a topic the student may actually need to discuss rather than some sort of T/F or multiple choice, the more difficult it is to test. So this sort of test will always be subjective, affected by your personal judgment of the student’s performance.

One thing that helps is to be sure to base your judgment as closely as possible on the rubric or band descriptors you are using. You should never compare students to each other. This will lead you off the track. Always compare to your chosen rubric.

I always record my test interviews. A couple days later I will listen to some of the interviews and rescore them without looking at the score I gave the first time. If there is a strong correlation then that is good. If you find that you are scoring much differently the second time then you need to try to understand why and may even need to rescore all your interviews. It happens that you can be in a certain mood that will cause you to score differently. (Another good reason to record is to contribue to a record of the student’s progress.)

IELTS research has even shown that male interviewers will sometimes give attractive females a slightly higher score which leads to inaccuracy. If the interviewer is tired, sleepy, hungry or if the interviewer has scored several high level students in a row and suddenly gets a low level student it can affect his accuracy. To run an effective test you need to be aware of all of these things and try to guard against them effecting your judgement.

>Does China have the worst English teachers in the world according to international test results?

>Some teachers tried to tell me that China has the worst teachers and students based on the international test results of the Cambridge BEC tests. Is that true? Can we use those tests results and other tests like IELTS and TEOFL to see where the best and worst English teaching is being done?

Here are the statistics for 2004 sorted from lowest pass rate to best pass rate, or you could say from worse to best.

As you can see, China did indeed score the worst of all the other countries. Does this mean China’s teachers are the worst? There is an easy test we can apply to find out the answer.

If these statistics show us that China’s teachers are the worst in the world then they should also help us to see which teachers are the best in the world.

China, People’s Republic of 40%
Indonesia 41%
Vietnam 45%
Brazil 58%
Hong Kong 59%
Italy 67%
France 68%
Spain 70%
India 71%
United Kingdom 74%
Argentina 78%
Bangladesh 79%
Czech Republic 79%
Switzerland 80%
Croatia 81%
Poland 81%
Russian Federation 81%
Germany 84%
Austria 86%
Portugal 89%
Canada 93%
Slovenia 95%

Is it true that Slovenia has the best teachers in the world? They scored a 95% pass rate! Incredible! Even better than Canadian teachers. How did they do that?

And what about British teachers? Is it true that Argentinean teachers are better than British teachers?

I’m afraid we can only use the pass/failure rates of the BEC (and also the IELTS) to show us the rates of those who took the test and passed or failed. It does not reflect the language ability of students in general and the teaching ability of teachers in general.

People take this test for different reasons, different goals at different ages. Only if the BEC was given to ALL High School graduates or at least a true random sampling of students who were ALL at the same level could we use them to try to interpret teaching quality. These statistics from the BEC do not answer the question on the English teaching ability of teachers in China.

But this does not mean Chinese English teachers are good. It only means these particular statistics are not going to be useful in the question.

As I said, it doesn’t really matter if China came in last in the BEC tests as this sort of test is not going to tell us anything about the teaching or learning skills.

>Evaluating speaking and validity

>A teacher, who has his students do a class presentation as the sole English speaking evaluation task, defended his approach by saying: “I guess this just shows that there is more ways than one to skin a cat – the principles of public speaking are things that we can use every day in our lives and, if it helps to build up the confidence in the students to use their English, then it seems as good a technique to use as many of the others.”

Indeed, there is more than one way to accomplish a task. But there are also good ways and bad ways, effective ways and ineffective ways.

I can’t say for sure but I’ll make a guess that you haven’t taken a TESOL course or had any formal training to be an English teacher. I’ll admit my formal training is skimpy but I study as much as I can.

Brown & Yule explained some of the challenges of speaking. “…it should also be borne in mind that even native speakers of English find that a straight description is easier, in some sense, than telling a story and, in turn, that telling a story is easier than a justification of an opinion….This is a rather general guide to the level of difficulty. Naturally, a short narrative involving a single character and only two or three events may be easier than a lengthy description covering many details and relationships.”

Here is a list of examples of speaking that Brown & Yule point out:

1. Static relationships

a) Describing an object or photograph
b) Instructing someone how to draw a diagram
c) Instructing someone how to assemble a piece of equipment
d) Describing/instructing how a number of objects are to be arranged
e) Giving route directions

2. Dynamic relationships

a) Story telling
b) Giving an eye-witness account

3. Abstract relationships

a) Opinion-expressing
b) Justifying a course of action

If you are familiar with the IELTS speaking test you can see these elements in the test. First the examiner asks some basic information or a description of something. Then to tell a story of something like a holiday you took. Then finally he will ask for some opinions on some topics.

He works from easier tasks to more difficult tasks. Students should begin having difficulty at some point during the test by which the examiner can see the student’s level of competence.

Of course, as Karen pointed out, there are different kinds of tests and what we are talking about are not IELTS English competency tests but more like a pass/fail type test.

But my point is that there are a lot of different demands English speaking tasks make and it’s important to match the task to the need and to achieve validity.

Brown explains: “The general concept of validity was traditionally defined as ‘the degree to which a test measures what it claims, or purports, to be measuring’ (Brown, 1996, p. 231).

“Validity was traditionally subdivided into three categories: content, criterion-related, and construct validity (see Brown 1996, pp. 231-249). Content validity includes any validity strategies that focus on the content of the test. To demonstrate content validity, testers investigate the degree to which a test is a representative sample of the content of whatever objectives or specifications the test was originally designed to measure.”

So a test of General English would not be valid for Business English students. A Public Speaking test would not be valid for General English students. Would it be good for General English students to be competent in Public Speaking. Probably. Does the ability to speak publicly show competence in a general English speaking situation? No. Could the specific anxiety generated by speaking in public interfere with General English production? Absolutely.

General English requires the ability to negotiate meaning, explaining, understanding when they are not being clear to the listener and using appropriate strategies (circumlocution or defining terms), turn-taking, etc.

To this the teacher questioned: “Why??? What support do you have for this statement. In my years in China, the correlation is pretty strong, good english communicators give good speeches. This seems natural.”

I meant invalid as far as content validity is concerned. Are you familiar with content validity?

As explained by Underhill the question is:

“It is relevant? Do the items or tasks in he test match what the test as a whole is supposed to assess? Where the objectives of the programme are set out in detail, for example in a syllabus that lists skills or functions, then the content validity can be assessed by comparing the kind of language generated in the test against the syllabus. The questions then is whether the test produces a good sample of the contest of the syllabus.”

Using a public speech as a way to test general English is like giving a driving test on a motorcycle instead of a car.

Brown & Yule (1983) Teaching the Spoken Language, Cambridge University Press. Brown http://www.jalt.org/test/bro_8.htm
There is a lot of good information about testing at: http://www.jalt.org/test/news-title.htm