>The Bob System: Tracking students for formative assessment


I am often called on to teach oral English. Unlike teaching written English where the students will be submitting a lot of writing samples, oral English offers less opportunity to sample the students’ English ability.
My primary interest in using the Bob System and some sort of scoring system is in formative assessment.
When I have a clear understanding of how they are doing then I have the ability to try to make my training more effective in two ways.

First, are my students “getting it”? Am I helping them to learn what will be useful for them to know?

Second, I can customize my training more to my students’ specific needs. I may not be able to give each student individual training (that ability and technology will be coming in the future) but I could segment the class. I want to know who is doing well and who is doing poorly.

When I know this I can offer extra training to those who need extra help. What about students who are doing very well in the class? Sometimes there is an academic ceiling in the classroom. Bright students cannot go higher because the teacher is teaching to the “middle level” of the class. But if we know which students are doing very well and how many of them there are then we can focus on their needs better by providing extra challenge.
Tracking this sort of information can be very useful in other ways, as of Action Research in the classroom. If you are monitoring many aspects of the student’s performance in the classroom and you have a student who always participates correctly, does the pairwork, groupwork, homework, listens and doesn’t goof off but does not seem to progress in their English from one term to the next then that would raise some very good questions for the teacher.

Of course, finally, the data that is collected can help in summative assessment. The teacher does not need to simply rely on a final exam for a score. The teacher will have a multidimensional way to look at the students.
[Photo: Some of my 300 college students that I taught weekly last term doing pairwork. Next term I will have 400 college students each week.]

>More ideas for mLearning projects

>These are some things that have already been done. I don’t post these here as suggestions of what you could do but to give you some ideas of how other teachers are using mLearning in the classroom. As technology improves and becomes more efficient we will be able to do more things similar to this:

“Students have to carry out scientific enquiry in the context of their discovering and exploring of an environment. Pairs of 11-12 year olds explored a woodland and were presented at certain times with different forms of digital augmentations.”

“Students can send SMS that appears on lecturers laptop during the class. They can anonymously ask questions without interrupting the class. The lecturer can choose to respond immediately or wait until a number of questions arise. SMS are available after class.”

“Players use GPStracked handheld computers to experience a virtual savannah that appears to be overlaid on a football pitch sized grassy field. Players act collaboratively to carry out a series of lion missions (such as marking their territory, hiding their cubs and hunting). Aim of the game is to encourage players to understand the behaviour of lions though personal experience.”

“Students in a economics course could acess course mateiral (slides, PDF) and contribute on the discusstion board.”

“Composition students were asked to compose and perform music on PDAs. PDAs are equipped with MIDI module and portable keyboard.”

“Evaluate to what extent and how a PDA can help students learning. PDAs are issued for the project time with the aim of collectiing application log data. Learn more about how students use PDAs.”

“The project aims to test ‘just-in-time’ access to knowledge on mobile terminals for medical students. Especially when they are practicing in medical institutions.”

“Medical practitioners on ward are equiped with PDAs to build a portfolio of evidence. A detailed evidence of clinical activities, prior learning comopetencies, course materials, certificates. Furthermore they get acess to learning resources, clinical guidelines and a learning diary.”

“Informatics students were required to develop and eavluate their own interactive learning experiences in collaboration with fellow students. PDAs with internet functionality were lent to them to consider as their own.”

“Course materials were distriubted in e-book format. Followed by an investigation how students use these learning materials.”

“A collaborative treasure hunt game that requires co-ordination between spatially seperated team-members. The teams have to visit locations in an urban area to collect symbols in order to complete tasks.”

“A collaborative problem solving application that attempts to support learners in constructing their own understanding of tshare their decisions with fellows.”

“A application for percussion composition, allows users to create, manipulate, edit and save original pieces of percussion music through an intuitive interface.”

“Children collect with scientific data on spot. Devices can collect data and communicate with sensors that are in the field while also providing instant feedback through on the spot data analysis.”

“This project supports the participants in expressing their thoughts through a digital narrative. While the overall process is similar to other digital film projects, the tools used are different. The learners shoot all of their footage and record their soundtrack on smartphones. In addition, the smartphones allow the participants to make their multimedia available to collaborators by sending the images and sound via the multimedia messaging service (MMS) to a blog.”

>The Bob Project – Are we ready for mLearning?

>The idea of students interacting with an automated teaching system is central to the Bob project. Many other people are promoting an idea called mLearning or Mobile Learning:

“Given its definition m-learning could very well be a new form of personal learning that never ends, allowing more and more people to realize how much of our lifetimes on this planet are truly extended adventures in personal learning.

From MasterNewMedia.org
“The advocates of lifelong learning have been advocating this very change in how we conceive, design and deliver education. Individuals are constantly learning, searching, questioning and acknowledging new information from the environment they operate in, no matter what their interest or specialization is. Unless your work assignment is something that a computer or other automated machine could take over from you, an increasing number of work activities depend on your ability to learn and familiarize yourself with a continuosly growing array of new concepts and ideas.” [Photo: mLearning at a museum]

Some researchers are testing the use of SMS messaging in the classroom. That would be interesting. Instead of teachers telling the students to put their phones away they’ll be saying, “Turn to page 35, take out your mobile phone and send me a message on question #6.”

From Using short message service to encourage interactivity in the classroom:
“Interactivity in the classroom is reported to promote a more active learning environment, facilitate the building of learning communities, provide greater feedback for lecturers, and help student motivation. Various definitions of interactivity exist in the literature, alternately focusing on the participants, structure and technology. The PLS TXT UR Thoughts research project builds on existing definitions to define interactivity as a message loop originating from and concluding with the student. The authors chose to introduce mobile phones and short message service (SMS) within the classroom due to the ubiquity of mobile phones among students and the interactive potential of SMS. SMS is a low-threshold application used widely by students to quickly send concise, text-based messages at any time. The research presented involved students sending SMS in real-time, in class, via their personal mobile phones. Using a modem interfacing with customised software to produce SMS files, the lecturer can view the messages and verbally develop the interactive loop with students during class. The SMS are available online after class, allowing interactive loops to further develop via threaded comments.”

This is an idea rather close to mine about creating an audio tour for students at a popular student location like a shopping mall. It could even be interactive through SMS or MMS.

From Supporting Mobile Language Learning outside Classrooms:
“The continuous development of wireless and mobile technologies has allowed the creation of an additional platform for supporting learning, one that can be embedded in the same physical space in which the learning is taking place. This paper describes a computer supported ubiquitous learning environment for language learning, called LOCH (Languagelearning Outside the Classroom with Handhelds). In the environment, the teacher assigns field activities to the students, who go around the town to fulfill them and share their individual experiences. The main aim of this project, called One Day Trip with PDA, was to integrate the knowledge acquired in the classroom and the real needs of the students in their daily life.”

mLearning can be used by students outside the classroom to make an instant blog of what they see, feel and experience.

From Moblogging for ESOL
“M-learning is a powerful tool for ESOL (English for speakers of other languages). In a recent example, ten adult ESOL learners became ‘photo journalists’ for the college open day. They created a photo diary of events using camera phones and sent their pictures, along with captions, to an e-mail address that automatically published them to a publicly-available web site. To prepare for the event, learners looked at published photo stories on the Internet and analysed the language and content to learn about styles and structures that would be useful in writing their own photo diary. The project proved so successful in engaging learners that even the most hesitant members of the class (e.g. a lady in her 60s, who had never used a mobile phone before, and a visually impaired learner) not only took part, but also found the experience very rewarding.”

And of course, mLearning is being used to teach languages.

More ideas

Are these the actual tools that we will use in The Bob Project? No. These are actual wild and crazy ideas that will help us think out of the box and find the most effective way to build Bob. If we don’t get out of the box we won’t go anywhere.

>Speaking evaluations made simple

>This is a very complicated subject. It is not easy to conduct a speaking test but I will go over just a few things about it and touch on them lightly. There are many ways to do speaking tests. I have studied them, tried some of them and have settled on this way. It is similar to the way I was trained as an IELTS examiner with a few differences.


Design three levels of questions.

(1) Easy questions which are answered with straight factual answers. “Where are you from?” “How long have you been here?” “What did you do yesterday?” “What do you like to do on the weekends?” These questions make little demand on the student and only very low-level students will have problems with these.

(2) Moderately difficult questions demand more from the student. These are questions asking a student to describe a city or restaurant, relate the story of a movie recently seen or a book recently read. “Tell me about your last holiday? “Describe your best friend.”

(3) Difficult questions are those that require the student to give an opinion and justify their opinion with reasons. “Should students be required to wear school uniforms? Why?” “Should smoking be banned in all buildings? Why?”

Be aware that some questions are not only difficult to discuss in English, sometimes they are just plain difficult to discuss at all. I once designed a question, “If you had two weeks to live, what would you do?” This question was so deep that the students became extremely thoughtful in trying to give their answers to the point that it interfered with any attempts to show fluency. Questions do not need to be so deep.

Although the question may be difficult at times, to understand the question should be simple. Remember, this is a speaking test, not a listening test. For example, “Given the opportunity to go on a round-the-world cruise or participate in a scientific exploration in Africa, which do you think could potentially be more beneficial for your career development?” Many low and mid level students would not be able to understand that question and therefore would not be able to speak on it. Make sure your questions are easily understandable.

I like to let the students ask each other the questions. This way I can focus on listening and evaluating. But I do not allow the students to prepare for the questions except for perhaps just a couple minutes before the interview.

Look me in the eye? In western countries we have no problem looking into people’s eyes when speaking to them but this is something that Asians do not do. Therefore, when you conduct the speaking test with Asian students it is best to not try to look deeply into their eyes or to hold their gaze. Look elsewhere, shift your eyes around or even just focus on your band descriptors or rubric.


You can use a rubric or band descriptor to measure the student’s level such as the IELTS band desciptors or the Common European Framework.

You will notice in the IELTS descriptors that at Band 4 it says:

“Is able to talk about familiar topics but can only convey basic meaning on unfamiliar topics and makes frequent errors in word choice. Rarely attempts paraphrase.”

and then at Band 5 it says:

“Manages to talk about familiar and unfamiliar topics but uses vocabulary with limited flexibility. Attempts to use paraphrase but with mixed success.”

That is why it is important to design your interview questions with easy, moderate and difficult topics so that the student will have to try to produce a full range of English at different challenging levels to respond accurately. The English of many students will begin to break down at the higher levels and this will allow you to see the limit of their English.

I put the band descriptors and all the students names on an Excel spreadsheet. I give the student a score for each rating catagory (Fluency and coherence, Lexical resource, Grammatical range and accuracy)
and the program averages it out into a final band score. Depending on the situation I will add formulas to work that score into a grade, average all the scores to compare one group with another or other things. Click on the picture (above) to see it enlarged.


The more realistic the task is, talking naturally about a topic the student may actually need to discuss rather than some sort of T/F or multiple choice, the more difficult it is to test. So this sort of test will always be subjective, affected by your personal judgment of the student’s performance.

One thing that helps is to be sure to base your judgment as closely as possible on the rubric or band descriptors you are using. You should never compare students to each other. This will lead you off the track. Always compare to your chosen rubric.

I always record my test interviews. A couple days later I will listen to some of the interviews and rescore them without looking at the score I gave the first time. If there is a strong correlation then that is good. If you find that you are scoring much differently the second time then you need to try to understand why and may even need to rescore all your interviews. It happens that you can be in a certain mood that will cause you to score differently. (Another good reason to record is to contribue to a record of the student’s progress.)

IELTS research has even shown that male interviewers will sometimes give attractive females a slightly higher score which leads to inaccuracy. If the interviewer is tired, sleepy, hungry or if the interviewer has scored several high level students in a row and suddenly gets a low level student it can affect his accuracy. To run an effective test you need to be aware of all of these things and try to guard against them effecting your judgement.

>Organizing speaking tests for large numbers of students

>Yesterday, I did a speaking test for 200 students. It’s quite a big job and took me all day. But it would have taken much longer if I had never done this before and if I wasn’t organized.

Some teachers allow students to choose their partner and choose their subject. Sometimes they can do this days in advance. Consequently, some students will find a dialog and memorize it and then perform it for the test. I don’t do it this way. I don’t think it’s very realistic.

In the workplace, people need to be able to speak English to anyone and they can’t always choose the topic. So I tell my students they can choose any partner they want as long as student #1 chooses students #2 and student #3 chooses student #4, etc. I tell them they can choose their own topics to talk about. They come up to my desk and choose from several slips of paper which are facing down. The students cannot look at the paper before they choose. Consequently, they are choosing randomly. I do allow them 3-4 minutes to prepare before the interview. The topics are always things that we practiced discussing in class.

When one pair of students sits down to do their dialog for me another pair of students will come up to the desk, choose a topic and stand aside to prepare. I don’t allow them to use dictionaries, notebooks, textbooks or to talk with other students during the preparation period. I sit facing the two students who are talking but behind them I can keep an eye on the next two students who are preparing for their talk. This is important because they many of them can hardly keep themselves from a bit of cheating if it is possible.

The students talk to their partner on the topic. I used to be an IELTS examiner and found it is a bit extra work to have to also be asking the students all the questions. So I like to get the students talking with each other and I listen in. If I think a student can go higher or if their dialog was too short I will jump in with a few extra questions.

While they are talking I am recording them. They hold a cheap $2 clip-on microphone that I found which works really well and is fastened at the end of a ballpoint pen. I use free program called Audacity to record the interview. Although I’m sitting right next to them, I actually listen to the students through a set of earphones. This ensures that everything is being recorded. The interview is recorded for reference in case I want or need to go back and check something or if I need to justify a score I have given.

If their dialog is too short or doesn’t reveal their English skills well enough I will ask some questions to make them speak more. Answering”why” questions or questions where they have to explain or justify a viewpoint are some of the toughest questions and are good to push students to the limits.

I have some band descriptors and a list of all the students names and numbers on an Excel sheet. The band descriptors are divided into three areas of speaking (communicative range, overall fluency, accuracy & appropriacy) and four levels (levels 4-7 of a 10 level rating system) While they are talking, I scan the band descriptors and give them an initial score. I continue to listen to them and modify parts of my score as they perform better or worse.

It’s very important to have a clear set of standards that the students should speak to. And while they are speaking you should be constantly checking those standards and try to measure the student to those standards as best you can.

After all of the testing is done, I will use Excel to average out the band scores and assign grades on a curve.

>English Safari – 10 students at the mall

>I just got back from an English Safari to the mall this morning with 10 of my students. This time it was with a group of students from a training center. Most are adults with a couple young people.

I’ve been doing about one or two safaris with students every week. This was the biggest group. We attracted some attention. Some shoppers realized there was something special about us and some stood near to hear what we were talking about. The training center sent along a minder in anticipation of this. She gave out flyers about the school. Generally, I don’t think you would want to attract too much attention as the mall management may be disturbed if you have a big crowd or are blocking entrances or aisle ways.

What do we do on an English Safari?

Well, in a way it is odd to even ask the question. After all, we are out in the real world. We are surrounded by realia. The environment is dense with the necessities and even luxuries of life. As long as everyone is speaking English about the things they do and experience there then it is improving their English.

Look at a definition of Task Based Learning:

“A task-based approach assumes that speaking a language is a skill best perfected through practice and interaction, and uses tasks and activities to encourage learners to use the language communicatively in order to achieve a purpose. Tasks must be relevant to the real world language needs of the student. That is, the underlying learning theory of task based and communicative language teaching seems to suggest that activities in which language is employed to complete meaningful tasks, enhances learning.”[1]

If we are in a mall how can we not be talking about things of the real world? This is no book learning and there is no book.

But if you want to make a clear outline or have some goals to accomplish or check your students on here are some competencies taken from the “Common European Framework of Reference for Languages: learning, teaching, assessment”. In the scale given below, A1 is beginner and C2 is the most advanced.

Can speak well about…


Themselves & others

Simple routine
Limited work and free time activities
Simple directions & instructions
Pastimes, habits, routines
Past activities

Detailed directions
Accumulated factual information on familiar matters within their field


Where they live

People, appearance [they could describe people in the mall, even mannequins]
Background, job [jobs of people who work there, what duties do they have]
Places and living conditions [nice and poor conditions, luxury]
Objects, pets, possessions [their possessions compared to those in the shop]
Events and activities [what do you need/want on holidays]
Likes/dislikes [preferences]
Plans/arrangements [what big things do they plan to buy, when, why and how often] Habits/routines [how often do they go shopping, to the restaurant, to movies, what do they usually do when they go there]
Personal experience [did anything funny or exciting ever happen when they went to a mall]

Plot of book/film, reactions [the mall has a cinema, this is a good one to talk about there]
Experiences, reaction
Dreams, hopes, ambitions
Tell a story
Basic details of unpredictable occurrences, ie: accident

Clear detailed description of complex subjects

Want more ideas?

Here are some topics related to our society, also from the Common European Framework. Nearly all of these topics can be touched on or discussed in detail during an English Safari to a mall. You could get ideas from below and together with your own ideas make a checklist and check items off as you discuss them with your Safari group. If your students show difficulty in some things you can make a note and cover it more deeply in class. Sociocultural knowledge

1. Everyday living, e.g.:
• food and drink, meal times, table manners;
• public holidays;
• working hours and practices;
• leisure activities (hobbies, sports, reading habits, media).

2. Living conditions, e.g.:
• living standards (with regional, class and ethnic variations);
• housing conditions;
• welfare arrangements.

3. Interpersonal relations (including relations of power and solidarity) e.g. with respect to:
• class structure of society and relations between classes;
• relations between sexes (gender, intimacy);
• family structures and relations;
• relations between generations;
• relations in work situations;
• relations between public and police, officials, etc.;
• race and community relations;
• relations among political and religious groupings.

4. Values, beliefs and attitudes in relation to such factors as:
• social class;
• occupational groups (academic, management, public service, skilled and manual workforces);
• wealth (income and inherited);
• regional cultures;
• security;
• institutions;
• tradition and social change;
• history, especially iconic historical personages and events;
• minorities (ethnic, religious);
• national identity;
• foreign countries, states, peoples;
• politics;
• arts (music, visual arts, literature, drama, popular music and song);
• religion;
• humor.

6. Social conventions, e.g. with regard to giving and receiving hospitality, such as:
• punctuality;
• presents;
• dress;
• refreshments, drinks, meals;
• behavioral and conversational conventions and taboos;
• length of stay;
• leave-taking.

7. Ritual behavior in such areas as:
• religious observances and rites;
• birth, marriage, death;
• audience and spectator behavior at public performances and ceremonies;
• celebrations, festivals, dances, discos, etc.

I make note of the new words we learn together. I sometimes do this on my cell phone and after the class is finished I send a copy to each student.

[1] http://iteslj.org/Articles/Rabbini-Syllabus.html

>What to teach? What not to teach?

>A teacher in America said: “I asked my students to learn the 50 U.S. states, with capitals and what each state is commonly known for. There was some grumbling about ‘Yankee imperialism’ or some such comment, but it was important for context knowledge in conversation.”

I think such ideas should be tested with the “You don’t have anything better to teach them?” question. The same goes for teaching things like Shakespeare. All of this is great to teach students if you have taught them everything else that they need to know.

There are many Americans who cannot recite all 50 states (myself included) and don’t know all of the capitals (myself included). There are many native English speakers who have not read one complete work of Shakespeare (myself included).

I’m not saying that we shouldn’t teach these things to students. It’s just that we should have a priority list and I suspect there are a lot of other things that our students will use everyday, every week, every month or even every year that they should learn first.

It would be nice if we could equip our students for everything. But, as our students pack their bags for their journey through life, we must make sure that we only add to their burden the things that they will use the most.

>Know thy student

>Yesterday, I passed out a slip of paper to my college students and asked them to write two things, their English learning problem and anything else they’d like to say to me. I also asked them to put their name on the papers.

I have only done this with 200 of my students and I’ll do another 100 on Wednesday but I don’t expect the feedback to be very different.

The main thing they tell me is that they have a real hard time learning new words.

I believe this is because they believe when they “learn” a word then it should be “known”. I’ve been trying to steer my students away from deeply ingrained this idea but I can see how difficult it is for them to give it up.

At best, our students can only be “introduced” to a word in much the same way we introduce one friend to another friend. At that point, the friends will only know each other’s name and a couple facts like their jobs and place of origin, etc. They will only know each other about 1%. If these two newly acquainted friends got married and would spend a year together, work together, play together, go through hardships together, they would know each other more completely but still not completely know each other.

So the only way students can really learn words is by constantly coming in contact with them. That is why I favor English learning approaches that lean towards extensive contact with English such as Extensive Reading and also watching a lot of English TV, movies, etc.

The second biggest lament was learning grammar.

I am not the grammar teacher in this school so I don’t know how that goes but I can imagine. It makes a lot of sense that our students should be able to learn a lot of words and then learn all the grammar rules and put it all together into sentences and communication. It makes great sense but it doesn’t work. The mind is unable to mechanically put it all together at the moment of communication. Again, that is why I favor an approach which leads to massive exposure to the language as well as using language to communicate as per the Communicative Approach.

I was not surprised that my students had those feelings about vocabulary and grammar. I was surprised that despite my bringing this up with them a couple times already they are still thinking they have to do it the old way. This shows me that they haven’t bought into the better idea and I haven’t done a good job in helping them understand.

Photo: My students giving me feedback.

>How can we encourage autonomous television watching?

>Some teachers speak of autonomous learning. Some teachers feel their students are too lazy. Some teachers feel students need to always be pushed to learn.

Why is it difficult to get students to study English but it is not difficult to get students to watch television?

Is it because watching television is, to put it simply, brainless? Do people have an inherent need for brainless entertainment?

Are all television programs brainless? Do viewers never learn anything useful from the tube? Is there no useful educational content in television?

Or is it that the makers of television programming have learned to be “student-centric”? Do they work under the pressure that viewers can switch to another channel with one click? Does this propel them to captivating content?

If our students could get up and leave our classrooms at anytime with no negative repercussions would it change the way we teach?

Are there any teachers out there that could compete with television? Are television programs always more engaging than English lessons?

Is there anything we can learn from television?

I am not recommending television watching or movies here, although I think those are great tools. My point is that we can learn a lot from these people, like television producers, who must, every night, attract the attention of what is a fickle public.

Rather than take the approach of making a boring processes more palatable to students, what if we really challenge ourselves to present materials as interesting as TV to make our training thoroughly engaging to the students?

I think that would motivate students to be truly autonomous.

>Teaching, learning and "concept pods"

>One friend told me that, as a child, he mistook the line from the old Christmas song which goes, “While shepherds WATCHED THEIR FLOCKS by night…” and always ang “While shepherds WASHED THEIR SOCKS by night…”

But I think that we can say that incidental learning actually produces massive results. A tremendous amount of learning is taking place without it being taught, not only new language patterns but reinforcement and greater development of language already acquired. And I think we can say that nearly all of this learning is accurate.

This is where Mert Bland’s Concept Pod Theory comes in. It starts as a nucleus of one single idea of a word. Over a period of time more and more understanding and definition and application for the word is added through a vast number of contacts and encounters with the word and experiences attached to the word. It’s something like a snowball effect as more and more ideas get stuck on the word and our understanding of the word grows.

Christine Tierney is right. Some of it needs to be corrected if it is mistaken when acquired. Thus, direct teaching has a role to play in correcting bits and pieces of this massive amount of material we have not learned correctly.

Now Christine Tierney’s students’ word, “firstable”, was a concept nonetheless. They had a clear idea of what they were trying to express with that word and the concept was valid. Thus, I think according to Mert’s idea of a Concept Pod, the Concept Pod for that word was begun. They had ideas of how to use this word and how not to use it. It’s complex growth had begun. However, a bit of correction was needed to the Concept Pod to realize that actually the correct thing to say is “first of all”, not “firstable”. On the other hand, perhaps we are watching the birth of a new word in the English language.[1]

Here in China, almost all students have trouble with the word “colleague”. They want to say “col-lea-gue” as opposed to “col-league”.

As Krashen says, “The study of grammar has value, however: Even those who are well-read may have small gaps in their writing competence, and conscious knowledge of some grammar rules can be helpful in filling some of these gaps (e.g. the it’s/its distinction).”

Proponents of the value of indirect learning (Comprehensible Input, Extensive Reading, what I call “Extensive Contact”, etc) are not saying that all direct teaching should stop and all learning should be done indirectly. It’s just that we realize the massive amount of language our students learn without it being directly taught and also realize the difficulty students have to learn even simple language rules that are directly taught, so we raise the question if “direct instruction” should support efforts towards “indirect learning” and not the other way around.

[1] http://archiver.rootsweb.com/th/read/WORDS/2000-03/0952827010