Pilot Project on Asynchronous Learning
Summary Report to the Ed Tech Committee
Dianne Lynch
Introduction
In
the spring of 2003, I was considering the possibility of participating again in
a collaborative teaching project with which I’d been involved twice before with
colleagues in
I
proposed a “mixed-model” course because, while I have been enthusiastic about
the potential of online learning, I have always had real questions and concerns
about its limitations, particularly as they related to the critically important
relationship between student and faculty, and to the unique challenges the
format raises in terms of assessment and evaluation. In addition, my teaching style is very
dependent upon interpersonal interaction with my students; while chat rooms and
email provide an alternative to face-to-face relationship-building, I remain
unconvinced that they are equivalent.
Combining the two ‘platforms’ seemed to offer the best of both
environments, and it provided the opportunity to assess whether my ‘gut
instincts’ about the importance of physical presence were validated by
experience.
In
selecting the course, I looked for one that was taught frequently enough that I
would have ample opportunity to identify a control group; I also wanted to
teach a course familiar to me so that I could concentrate on the platform and
pedagogy as much as on the content and curriculum. I chose JO 102 Intro to Writing, a
required first-year course that emphasizes basic writing skills (grammar,
usage, syntax and style) through extensive reading and style emulation. I teach the course as an introduction to the
“Literature of Journalism”; students read the works of such writers as Mark
Twain, Annie Lamott, Hunter Thompson, Truman Capote, Jack Keroac and Annie
Dillard. That format was conducive to
Web presentation. In addition, I had
already built a course site on ecollege.com that I had used to teach the course
during the fall of 2002; I assumed that would provide a sufficient base for the
summer course (I was incorrect; more on that in ‘Lessons Learned’). And I decided to use the EdTech grant as an
opportunity to develop some core teaching tools that I had wanted to create for
some time: a set of online drill-and-practice modules to help students improve
their writing skills. While there are a
variety of such tools available on the Internet, I wanted to create a
proprietary set of materials that would be permanently available to our
students – across disciplines – and to have the ability to revise and update
them as time went on. So I built that
project into the course proposal.
The course
I
expected an enrollment of six. Instead,
there were 14 (eventually 13 after one student didn’t log on for the first
week, discovered he was hopelessly behind, and dropped).
I
expected that all of the students would be sophomore journalism majors who
couldn’t get into the course during their first year. Instead, three of them fit that description.
A fourth was a 1999 UVM graduate considering a career in journalism. The rest
-- three graduating seniors who needed a final course, and seven rising
seniors – enrolled in large part (I am certain) because they expected that it
was going to be a breeze: “It’s a summer course; it’s a 100-level; and it’s
online – how hard can it be?” None were
journalism majors; they came from sociology, business, economics, English, and
history. While the particular group was
obviously unique, it merits mention that students may come to an online course
– particularly in the summer - -believing that it will require less of them
than a traditional course. While it took
only a couple of days to disabuse them of that notion, there’s no question that
that perception played a role in their decision to take a distance course.
The
course was structured differently from many other asynchronous courses. As you can tell from the syllabus (if you cut
and paste the URL into a browser, it’ll take you to the course:))
it
was grounded in the assumption (indeed, the requirement) that all students
participate and complete the assignments every day. Rather than a ‘body of knowledge’ that
students could access and master on their own schedule (within predetermined
dates), the course was a process as much as it was a product; the expectation
was that students would meet daily deadlines and would participate five days a
week. That expectation enhanced their
learning, I believe, but it demanded much more of them and of me than a less
‘engaged’ model would have; some would certainly have preferred to wait until
Sunday afternoon, pound through the readings, write two or three essays, and be
done with it until the following Sunday.
That would have required only that I set up the course and collect the
work once a week – a far more efficient and less-time-consuming process than
the ongoing interaction we had every day. But it would have undermined their
learning, which required them to build every day on the lessons of the day
before.
I
incorporated into the course the same five key elements central to any writing
course:
- content delivery: Students were required to read a lecture
every day that highlighted a particular genre of writing or author;
- models of excellence: Students were required to read as many
as four articles or book excerpts modeling the particular writing style
under review;
- emulation: Students completed daily writing
assignments (some of those were revisions of earlier drafts);
- collaboration: students exchanged
papers and interacted in the chatroom and blog;
- practice: Students were required to complete up to
three daily grammar and usage quizzes, 20 questions each. On two occasions, they were required to
complete 100-question comprehensive quizzes.
(See
appended screenshot for the daily interface).
Assessment
The
course involved three very different kinds of assessment: 1) assessment of student learning; 2)
assessment of student experience (student evaluations of the course); 3)
assessment of the efficacy and effectiveness of the grammar modules.
Let’s
take them one at a time:
- assessment of student
learning. I came away from the experience with a
less-than-clear picture of its efficacy in terms of student learning.
There were several reasons for this: 1) as an experienced writing teacher,
I can intuitively tell you whether one pedagogical approach produces
better results than other, but the kind of rigorous assessment that would
prove such assertions was well beyond the scope of this project. My
general sense of it was that students who had the time to participate
fully learned a great deal in the course and truly enjoyed it; students
with poor skills, overbooked schedules, and expectations that didn’t match
the course realities did less well.
That said, even those students were enthusiastic about the course after
we’d met face to face; drawing students into the learning community
and identifying their individual points of reference are processes that
don’t happen through a computer screen, and they are critically important
for ‘marginal’ students who really need that kind of personal contact the
most. 2) The nature of a writing
course makes it difficult to draw valid comparisons across classes or
groups of students. The mix of students this summer was very different
than our typical JO 101; during the semester, the course is restricted to
journalism majors, most of them first-year students. This summer, they
were all over the map, in terms of class, discipline and writing
skill. In fact, I was astonished
(and appalled) at the poor writing skills of several of the graduating and
rising seniors; at least four of them could not consistently write an
intelligible sentence, much less express a complex idea or develop a
cohesive argument in written form. At the same time, two of the students
were extraordinarily gifted, insightful writers; both are now working
professionally as journalists. Such
a range of skill levels required a great deal of one-on-one attention to
each of the students; I often felt as though I were teaching 13 independent
studies more than a single course.
Final grades in the course ranged from two A’s to
several C’s and a D. The average grade
was a 2.7.
- assessment of student
experience. Student evaluations of
the course were generally positive, both about their learning and about
the online experience. (For a
transcript of their responses, see appendix A.) In particular:
§
All but one student agreed that they ‘learned a lot’ from the grammar
exercises. (One student completed only 40 percent of the exercises.)
§
Nine of the 13 agreed or strongly agreed that the face-to-face
interactions were valuable. That said, they were mixed in terms of whether
they’d prefer an all-online course; seven said no, four said yes, and two were
neutral.
§
Ten of the 13 said they would like to take another course online, and
would recommend an online course to their friends.
§
Students were enthusiastic about their learning experience in general
and their interactions with the instructor.
It was clear from their evaluations that they were pleased with the
course overall.
In general, it’s my impression that students liked
the course but found it difficult at times to adjust to the delivery system and
its inherent demands on their attention and time. Obviously those with more time were less
concerned about those issues than those who were trying to work one or two
summer jobs; their course evaluations don’t reflect the extensive interaction I
had with many of them around deadlines and expectations. The evaluations suggest, I think, that students
may have found the combination of high course demands and expectations, a very
compressed schedule, and the flexibility of online learning to be more of a
challenge than they had anticipated it would be. (Me, too…)
3. assessment of the efficacy and
effectiveness of the grammar modules.
During the spring of 2002, I worked with Matt Golec,
an adjunct professor who had taught several sections of Intro to Writing, to develop a very informal ‘control group’
against which I might be able to measure the performance of students who
covered the same material through a distance course. Let me make it clear that this was not a
rigorous assessment or comparison; validity would obviously have required much
greater time and resources than we had available. What we did do, however, was
give Matt’s class the same 200-question grammar pre-test and post-test that I
used in my online course to determine whether there were substantive
differences in performance between students who had access to (and were
required to complete) the practice modules and those who did not.
The results were as expected: Matt’s group of first-year journalism
students (n = 21) initially performed generally better on the test than did the
mixed group of students who enrolled in the summer school course. While the average score in Matt’s class was
52 percent, the average score in my summer course (n = 13) was 47 percent.
(Neither group was particularly strong in basic skills.)
At the end of the course, however, there was a substantial difference between the groups. While Matt’s group had improved a scant 8 percent over the 15-week semester, the performance of the online summer class jumped nearly 25 points, to just under 70 percent. Whether those results would or could be replicated and validated in a more rigorous setting remains to be seen, but initial outcomes suggest what common sense would confirm: students who are required to work through more than 400 grammar questions inevitably improve their basic understandings of the language.
Lessons Learned
- Teaching online during
the summer raises unique challenges but is probably our most promising
market for online delivery. Given
that, the Ed Tech Committee might want to explore the relative
challenges/benefits of teaching a ‘process’ course such as writing,
language learning, music composition, and mathematics vs. a ‘product’
course dependent upon content delivery and regular evaluation. Each has its own particular issues:
process courses require immediate feedback and evaluation if students are
to be expected to continue to progress on a daily basis; content courses
require creative approaches to presentation so that the online course is
more than a digital textbook with a few chat discussions thrown in. Effective models and “teaching tips” for
each course type would be a valuable contribution to the future of
distance learning at St. Michael’s.
- Teaching an online
course is astonishingly time consuming.
I had taught the course before and I had the framework of the
course site already constructed. I
expected to be able to build it out one week at a time, in order to be
sensitive to where this particular group of students was in terms of
skills and understandings. What I
found is that I spent at least 8 hours a day in front of my computer,
responding to their emails, grading their daily assignments, writing (and
rewriting) my lectures, revising the next day’s assignments based on the
problems they’d had with today’s, etc. etc. Because their schedules were so
different – some worked mornings, some afternoons, some evenings – I felt
the need to be available to them all day long if I expected them to
complete their assignments on a daily basis; that’s obviously not the way
to do this, but I am still uncertain as to what the best way is. One option would be to have all of the
assignments due on a single day of the week – but that takes a huge toll
in terms of their learning process: they need feedback daily if they’re
going to improve their writing.
- Faculty and students
must be made far more aware of the challenges of online learning. I have never taught a more
time-consuming course; it was the equivalent of 13 independent studies,
all running simultaneously. Students,
for their part, were astonished at the demands of the course; they truly
expected it to be “Internet fun” and an easy three credits. The college might even consider setting
a gpa limit for online learning; better students can more easily handle
its demands.
- Real-life interaction
is necessary. Students should be
required to come to campus at least twice during a four-week session; the
dynamic online changed dramatically after we had had the opportunity to
meet one another in real life. We
are physical beings; we need physical contexts. And as I said, marginal students really
need the kind of ‘face time’ and direct discussion unique to classroom
learning. Our seven-hour sessions
combined group discussion, peer coaching, and formal assessments; by the
time they were over, we had made connections, we had common experience, and
they were no longer isolated learners.
It made a difference. After
the second session, I spent more than three hours with three of the really
marginal students, hanging out in my office, just finding out who they
were and what they cared about. It
made an extraordinary difference in their commitment to and engagement
with the course.
- The grammar modules are
an effective tool for teaching basic writing skills; they represent
precisely the kind of drill-and-practice exercises for which the Web is
uniquely well suited. I am in the
process of transplanting them from the ecollege.com site to freestanding
Web modules; I am using software developed in the
Thank
you for the opportunity to teach the course.
It was an invaluable learning experience.
Appendix A:
A screenshot of the home page, just to give you a sense of what it looks
like and how it worked:
Appendix
B: Summary of the Course Evaluations, Intro to Writing Summer 2003
This fall, I will be a: Sophomore (3) Junior (0) Senior (6) Graduate (3) Other (1)
My Grade Point Average is range
from a high of 3.7 to a low of “Low” (a student’s self-reported GPA). Average was 2.6 – relatively low for Saint
Michael’s, where the average gpa is 3.1.
This course is: a course in my major 3
a general requirement 0
an elective 10
other
The grade I expect to receive in this course is: A (0) B(10) C(3) D F
(Actual grades: 2 As, 3 Bs, 2 B-s, 1 C+, 2 Cs, 2 C-s,
1 D -- course average: 2.3)
Overall, I would rate my technical computer, email, and Internet skills as follows:
1 2 3 (5) 4 (5) 5
(3)
Novice Average Expert
The online format made
access to this class convenient.
5 (5) 4 (5) 3 (3) 2 1
Strongly agree neutral strongly disagree
I enjoyed taking this
course online more than I would have if I had taken the course in the
classroom.
5 (1) 4 (2) 3 (6) 2 (2) 1 (2)
Strongly agree neutral strongly disagree
Overall, I enjoyed using
the technology in this course.
5 4 (4) 3 (7) 2 (2) 1
Strongly agree neutral strongly disagree
The ecollege.com site worked (with the possible exception
of the Dropbox).
5 (2) 4 (8) 3 (2) 2 (1) 1
Strongly agree neutral strongly disagree
The assignments posted on the web pages adequately
explained what I needed to do.
5 (8) 4 (4) 3 2 (1) 1
Strongly agree neutral strongly disagree
Contacting the instructor by email or telephone was an effective way to get help when I needed it.
5 (12) 4 (1) 3 2 1
Strongly agree neutral strongly disagree
The instructor’s lectures and instructions on the site presented the material clearly.
5 (11) 4 (2) 3 2 1
Strongly agree neutral strongly disagree
The instructor used examples from his/her own experience or research.
5 (12) 4 (1) 3 2 1
Strongly agree neutral strongly disagree
The instructor was helpful
in accommodating my summer schedule and work load.
5 (10) 4 (3) 3 2 1
Strongly agree neutral strongly disagree
The instructor was sensitive to the fact that this was a new learning experience, and accommodated students’ needs accordingly.
5 (11) 4 (2) 3 2 1
Strongly agree neutral strongly disagree
The instructor demonstrates a commitment to high standards of professional competence.
5 (13) 4 3 2 1
Strongly agree neutral strongly disagree
The instructor seems to care about my learning.
5 (13) 4 3 2 1
Strongly agree neutral strongly disagree
The punctuation and grammar exercises were the least valuable part of the course.
5 (3) 4 (2) 3 (5) 2 (3) 1
Strongly agree neutral strongly disagree
I contacted the instructor by email or telephone when I was having trouble or had questions.
5 (13) 4 3 2 1
Strongly agree neutral strongly disagree
I was able to access the web-based material when I needed
to.
5 (5) 4 (8) 3 2 1
Strongly agree neutral strongly disagree
Handouts, lectures, grades, homework due dates, etc., were available on the website.
5 (7) 4 (5) 3 (1) 2 1
Strongly agree neutral strongly disagree
The amount of one-on-one interaction I had with the instructor (including electronic communication/email) was more than I’ve had with instructors in my traditional “real-life” classes. (This refers to actual interaction/conversation, not just being in the same room/lecture hall.)
5 (4) 4 (5) 3 (3) 2 (1) 1
Strongly agree neutral strongly disagree
The amount of work in this class was about the same as in other three-credit summer-school classes.
5 (6) 4 (3) 3 (1) 2 1 (3) – “was more”
Strongly agree neutral strongly disagree don’t know
I found it easy to find things on the course site.
5 (3) 4 (7) 3 (3) 2 1
Strongly agree neutral strongly disagree
I learned a lot in this course.
5 (8) 4 (4) 3 (1) 2 1
Strongly agree neutral strongly disagree
I completed all of the
punctuation/grammar/usage exercises on the site.
5 (13)* 4 3 2 1
Strongly agree neutral strongly disagree
*because the quizzes were auto-graded, I know that this was not even close to being accurate. In fact, a few student completed fewer than 40 percent of the quizzes.
My punctuation/grammar/usage skills improved over the past four weeks.
5 (3) 4 (9) 3 (1) 2 1
Strongly agree neutral strongly disagree
I found it harder to learn new concepts from the Website materials than from listening to my professors’ lectures.
5 (4) 4 3 (6) 2 (2) 1 (1)
Strongly agree neutral strongly disagree
I found the face-to-face interactions on campus twice during the course to be worthwhile.
5 (5) 4 (4) 3 (2) 2 (2) 1
Strongly agree neutral strongly disagree
I’d prefer a class that was all-online, without the ‘real life’ sessions.
5 (2) 4 (2) 3 (2) 2 (2) 1 (5)
Strongly agree neutral strongly disagree
I would have liked to have had regularly daily chatroom
sessions with the instructor.
5 4 (3) 3 (1) 2 (6) 1 (3)
Strongly agree neutral strongly disagree
I put more effort into this course compared with other 100-level courses I’ve taken.
5 (8) 4 (5) 3 2 1
Strongly agree neutral strongly disagree
I would like to take another course online.
5 (2) 4 (8) 3 (3) 2 1
Strongly agree neutral strongly disagree
Rank in order of importance from 1-5 (if one doesn’t apply to you, just cross it out):
I took this class because:
__________ It was online, which meant I could fit it into my schedule.
_________ I’m really interested in developing my writing skills.
_________ It was a required course.
_________ I needed the credits to graduate.
_________ I thought it was going to be easier than it was J.
I would recommend an online course to my friends.
5 4 (10) 3 (3) 2 1
Strongly agree neutral strongly disagree
Other comments?
It was definitely interesting.
I thought the course was very challenging and it took a lot of effort although I ddi not have a lot of time on my hands. If I could have taken this course in a regular classroom, I would have.
Thank you!
It’s been a pleasure.
This was a great experience for me. I enjoyed the reading and didn’t mind writing papers. This is a first for me, which is why I loved this class. Dianne was an inspiration to myself and to the other people in the class. I would recommend this class to anyone who enjoys reading and writing.
Great experience, especially in summer. Perfectly suited for a writing class (though we did miss out on a lot of workshopping, which would have been helpful)
You taught me a lot about my writing. I’m really glad I took this course. You are really a great teacher. Thank you so much!
Awesome professor, the most caring I’ve seen yet.
I thought I worked very hard for a three-credit course.