Introduction

This document addresses another set of changes to our teaching that we believe have the potential to contribute to a first year course that is exciting, effective and innovative compared to traditional computer science courses.

Our earlier documents proposed a shift to teaching first year in an object-oriented language and a Problem-Based Learning (PBL) style. We will not repeat their summary of our motivations and plan. Instead, this document briefly describes how we taught in our PBL style in 1996, how we would recommend this to change in 1997 and a summary of the overall results of our semester 1 evaluation.

We are gaining an increasing sense of the timeliness of our proposal, not only from our growing dissatisfaction with the current style of teaching but also from the increasing evidence of similar activity. In the Vice Chancellor's Forum on Teaching: The First year Student Experience at this University (6 September, 1996), the Keynote Speaker, Richard James, of the University of Melbourne, reported on a CAUT commissioned project which identified major problems experienced by first year students and summarised several recommendations, including the following (paraphrased from his presentation) which are relevant here:

• value teachers and co-ordinators of first year
• unpack the curriculum - decide what is important and create a coherent course focusing on that
• create learning environments that build a sense of student identity and affiliation focused on the course (for example, the small course effect).

This department has had many fine teaching initiatives, including several at first year level, and the authors of this document take considerable pride in having spearheaded a number of these in the past. We are also delighted at our success in achieving the move to an object-oriented programming language for first year, one of the central tenets of our original proposal. We now present the outcome of our evaluation of PBL for teaching CS1.

Executive summary

• will students, especially the weaker students, learn as well in a PBL format as in our current cs1 format?
• Can we resource PBL, especially the teaching effort?

Will students, especially the weaker students, learn as well in a PBL format?

Analyses of the examination results show no statistically significant difference between the PBL students and conventionally taught cs1 students in overall performance. Details are available in Question 2 of the Appendix which currently includes two sets of analyses.

In addition, we are aware that some staff believe that weaker students will be unable to cope with PBL. We have produced a scatter diagram of the performance of all CS1 students against their TER, the only readily available measure of `ability'. In the diagram below, we show the PBL student's marks as a square and other cs1 students as a dot. There is no evidence that weaker (low TER) students perform less well in a PBL learning situation, this inspite of the fact that five of the PBL students have TERs below 70, our usual cut-off.

Can we resource PBL, especially the teaching effort?

There are two major roles in teaching PBL: the section leader and the class tutor. A section leader will be a senior member of staff and their role carries similar levels of responsibility to that of a lecturer for a module. The section leader must prepare and deliver the 1-hr class attended by all students in the section and maintain administrative details for students in the section, teach one class of approximately 20-24 students, with 5 hours contact for the first four weeks of semester, 4 hours contact for the next four weeks and thereafter 3 hours contact per week.

Overall, our calculations show the staff costs of teaching PBL are similar to those of the current cs1:

• PBL section leaders cost about 20% less than current cs1 lecturers (PBL 375 hrs - compared with 540 hrs)
• Pbl class tutors cost 15% more than the current cs1 classes (PBL 2652 hrs - compared with 2333 hrs)
• Total marking costs are higher with some administration time has been traded for actual marking (PBL 300 hrs - compared with 221 hrs)
• Additional administrative tasks are well below current cs1 (PBL 150 hours - compared with 300 hrs)
• Overall totals were similar (PBL 3477 hours - compared with 3244 CS1 calculation).

Inherent in these calculations is the fact that costs predictions are harder than summarising actual costs and even that is not easy! Having said that, we conclude that costs of the two systems is comparable. We also note that if budgets are tight, PBL costs could be reduced by 560 hrs simply by having 6 in each group: this group size was quite workable this year and is widely used for PBL groups elsewhere.

Overview of cs1-pbl 1996

Writing a homepage engaged the students in an exciting aspect of computing, placing the Internet at their feet (something of an expectation in today's CS students). It also introduced a useful skill that we could rely on during the semester, asking students to attach reports to their home page, for example. It was also part of establishing skills in electronic communication to be used throughout the semester. Importantly, it served as an excellent way for students in the class and the tutor to learn about each other.

Writing a minimalist UNIX manual ensured that students were in a position to manipulate the environment in which they would be programming.

An important feature of this phase, in view of the fact that it occurs in a period of potentially volatile student enrolment, is that it uses fluctuating group membership; i.e., the tasks are essentially individual ones, but are tackled in a group environment which is deliberately varied in each class. This fits in well with concerns about group membership in this period, as well as providing a period when students could be introduced to the group learning environment.

Phase two consisted of a large problem in which groups were asked to choose between a (classical) cryptology scenario and one involving a video-shop database. The period lasted from week 4 to week 11, inclusive. At this stage, long term groups were established and students were required to develop and implement plans for researching, designing and coding solutions to the chosen problem. Pace was maintained by a requirement to provide reports on their progress and activities like code reviews where all members of the group and the tutor did a typical software engineering code review. This introduced a technique that is currently acknowledged as the most effective means of improving code quality, one that is not available in the conventional learning style where this activity would constitute cheating. Cryptography groups were encouraged to challenge each other by placing encrypted text on their home page as incentive for other groups to try and break them. Groups in this period were observed to be highly resilient to any adverse conditions. Note that each individual was required to take responsibility for some specific coding aspects.

Phase three was a period in which students reflected on their learning, and presented their work to the rest of the class and for assessment. The presentation consisted of reports documenting their work (which included individual reflective statements), submission of code (tarred with demo shell scripts and test data), and an actual demonstration of their product to the class (and to invited members of staff). During the lead-up to the demonstration, other activities were introduced as a basis on which their presentation might proceed; for example, cryptography groups were required to submit encryption code which was used to apply to large pieces of previously-unseen text, with the resulting pool of encrypted text being made available for them to attempt cryptanalysis. This was effectively a period in which their code was tested on further "real-world" problems, and was generally a rewarding period in which students were motivated by the opportunity to make use of the code they had written.

The practical work just described formed the motivational basis for student progress throughout the semester, and was directly supported by a weekly three-hour workshop period. A further (non-contiguous) two hour period was used for practical activities designed to facilitate the development of certain skills important to success in the main problem-based activities. This time was used to reinforce a range of useful skills, and not constrained to technical ones; for example, self- assessment activities took place in these sessions. Students were encouraged to be less teacher-reliant via a number of activities which involved working together in groups (these groups being different from those used in their problems). Technical material was often combined with this work; for example, one of their first exposures to Pascal was to be asked to do the first prac exam in small groups. This is typically PBL, in that the problem drives the learning rather than concludes it; for many, it de-mystified the programming task ahead, and strengthened their commitment to group processes as they jointly managed to work through a problem which initially they seemed to assess as being beyond their capabilities. Further support to the main problem-solving task was given in the 1 hour seminar slot used each Friday. This was sometimes used to present subject-matter - PBL is not discovery learning, and a seminar on certain aspects of coursework is just another resource that student would use. However, more often this one-hour session was used to discuss either conceptual matters which provide orientation to the task or consolidation of major concepts that had been the focus of the two-hour period of work. Such topics included things like problem solving, group work, programming style and testing strategies. Syntactical issues were not explicitly dealt with by a lecturer, although guidance was given as to which aspects students should be familiar with. For example, rather than lectures on procedures and functions, a session was devoted to modularity as a concept, and this was motivated by and linked to working as an individual in group contexts.

Modifications we recommend for 1997

More self-assessment tasks would be included in future programmes. We knew that this was important since many PBL courses identified this as a need. We included a number of these (such as a Pascal "checklist" which allowed students to self-assess the Pascal coverage they had achieved, short quizzes and longer take-home exams and self assessment forms for submitted work, in which they applied the marking scheme to their own submissions). These proved valuable, but in retrospect we do not feel that we extracted maximum value out of the device. As well as providing a useful mechanism for encouraging some degree of metacognition from the students, they offer a means for introducing structure to the course in a way that does not overly impose on student ownership of the learning task.

Another example of a change that would be incorporated in a 1997 implementation of PBL is to strive for a greater degree of incorporation of supplementary learning tasks. For example, we did a nice UNIX-based activity which was problem-based and students found a useful task; however, we later realised that we missed an opportunity for this to be framed in a wider context, such as program testing. A greater degree of task amalgamation would provide more realistic learning scenarios and could be used to reinforce the value of certain concepts (such as testing).

Finally, we have to emphasise that this was a first trial of PBL for a first year computer science course. Just as the shift to Blue will require a good deal of learning by staff and some blind-alleys, so this year's trial has given us considerable and valuable experience in applying PBL to first year computer science. At the very least, the administrative aspects will be improved in light of that experience. For example, the students were very confused about what was required of them: we had provided a document that gave detailed assessment criteria and we had devoted one class to having students apply the code assessment to some of their own code. But students were still unclear. We can see simple administrative and organisational ways to improve this. Similarly, we managed some of the marking rather inefficiently this year and only as the course progressed did we develop ways to give better feedback more efficiently.

Certainly, one important aspect of the approach will change in 1997, and that will be the change of language to BLUE. This is not insignificant on its (positive) impact on PBL. Our initial proposal included a proposal for change to an object-oriented language as a natural basis on which to deliver a course oriented towards groupwork (as PBL must be).

Acknowledgements

We also thank Bob Kummerfeld for his support through semester 1 and especially during the writing of this report when he took over teaching duties to permit the considerable time needed to prepare this report.

This report includes calculations by Ross Quinlan and graphs by Greg Ryan. We are grateful to them for helping in what has proved to be a large task.

Appendix: Detailed Answers to the Questions Defined by the Courses Committee

1. The 1996 semester 1 trial is a fair approximation of what is envisaged for 1997

The trial was substantially similar to the envisaged 1997 course except that the overall numbers were far smaller, timetable constraints meant that additional classes were run for the 1-hr session and we made more use of the PBL-lab than in our initial proposal.

However, we should note that we were constrained to match the coursework for the current cs1 course; we believe that other changes agreed for 1997 (adoption of an O-O language and greater emphasis on practical skills of programming as discussed in recent meetings) will mean that the 1997 course will be better matched to PBL parts of the trial.

We also note that the trial operated under difficult conditions that operated against the effectiveness of the teaching. The worst problems were in the first weeks when timetable problems (including changes) caused chaos for many students. A set of quite serious problems resulted from our inexperience in designing PBL-style courses: we did not include all the right check points at the right places and we sometimes designed materials in old-style tutorial format (with students seeing them as irrelevant at the time they did them). The students also knew this was a trial and frequently compared themselves with the other class and delayed engaging with their work.

Similar class sizes

The 35 cs1pbl students attended

• one 3-hr class: 20 (Thursday) and 15 (Friday) students. This is similar to the 1997 plan of 20.
• one 2-hr class: similarly, the class was split into two groups for this with similar class size to the 1997 plan.
• one 1-hr class on Fridays: the 1997 plan has a full pbl section in a single class. Timetable constraints meant that we had to give this class in all three of the normal cs1 Friday slots. Presentation of material was lecture-style and as the cs1a students also attended, we trialled this with classes up to 40.

Student TER profile

Staff and other resources

Other resources

Assessment regime

2. PBL students learn what we intend the regular CS1 class to learn as well as the regular class does

Analysis for a matched group of cs1 students is shown in the table below. Overall, there was no statistically significant difference on the semester 1 exam as a whole (using any of the methods of calculation). The students in the pbl trial were matched on the basis of TER, previous background and responses to a week 3 questionnaire about approaches to learning. Data entry problems meant analyses did not take account of people volunteering for pbl as intended.

The table below gives all statistics calculated for the semester 1 exam. The first column shows the page number. The second column shows Ross Quinlan's assessment of how much problem solving is assessed on that page: ** are regarded as largely problem solving and * have some problem solving.

The three right hand columns show all the statistical calculation so far done on this data. The third column gives the means and Z value (using Wilcoxon rank test - also known as the Mann-Whitney test) The fourth and fifth columns give 2-tailed t-scores for Null hypothesis: that the change to pbl-style cs1 had no effect on exam performance.

The differences between the three columns are:

• Column 3 uses the Wilcoxon rank test which is appropriate for data that is not in a normal distribution (which is the case for several of the exam questions) and the values shown for the individual questions are for all students for main part of the paper (P2-P9) and exclude non-attempts and zeros for the optional, advanced parts of the paper (H1, H2)
• Column 4 gives t-scores for populations matched for TER, computer experience at school and scores on the first questionnaire on approaches to study and learning (so it excludes all students who do not have any of these, eg no TER available).
• Column 5 gives t-scores on the pbl group with the whole of the cs1 population

The t values are given for two tailed t-tests. The fourth column t-scores generally have 74 degrees of freedom and this means that the two means are significantly different if |t| >= 2.00 (where |t| is the absolute value of t). The exceptions are H1 and H2, questions that were not completed by all. These have degrees of freedom of 35 and 39 respectively, so in both cases |t| >= 2.042 in order for the means to be significantly different. Thus the values near one are definitely not significant.

In the fifth column t-scores, the sample sizes were larger (as the whole of CS1 were being compared. Degrees of freedom for the total exam mark - S1E were 396. Here to be statistically significant |t| >= 1.98.

The differences marked are significant based on p>0.05.

The last row of the table shows that there was no statistically significant difference in overall exam results between the PBL group and either the matched control or the whole classes.

As already noted in Question 1, the groups do not differ significantly in TERs.

On all three measures, the cs1 class performed significantly better in page 7. The question on page 7 required students to draw `an expanded diagram of a typical' nested record structure, something that had been done in lectures but was not seen by PBL students.

The third and fifth columns shows the PBL group performed significantly better on page 9.

The results for the practical examination show are:

• for the PBL class, % passed = 48.5
• for other cs1 students % passed = 48.1

• 10.4 (sd 7.7) for the PBL class
• 11.0 (sd 7.2) for the other cs1 students.

3. PBL students are better than regular cs1 students at some additional skills that will be useful to them in later years, such as group work, problem solving, and independent learning.

Since PBL students were required to do many things unrelated to the main class assessment, these developed skills not normally part of courses until cs3. Students were required to write reports, devise their own convincing testing of code, put together code written by different students, demonstrate their work and, importantly, develop skill in planning and critically self evaluating their work and their planning. All these contribute to the generic skills that the University requires us to nurture in students.

In particular, the PBL course developed several skills our initial proposals identified as important to long term success in computer science as well as those in the University's generic skills. These include:

• Software development skills - we encouraged improved design, good style, documentation and re-use of existing code by doing code reviews here each group studied the code of individuals in the group with a tutor present or reviewing the outcomes.
• Group software development skills - students participated constructively in planning, co-operative testing and integration of software.
• Problem solving - specific activities in the 2-hr classes as well as the whole process gave students practice in the various strategies for tackling problems and planning work on them and the students wrote a reflective statement about these processes to consolidate learning from mistakes as well as successes.
• Group work - there were specific activities in the 2-hr classes to aid students in learning to work effectively in groups and there was considerable practice as well as the requirement to self-assess their success in this area.
• Report writing - students developed these skills in the process of writing a series of reports that were critiqued by tutors and developed through the semester.
• Verbal skills in the discipline - the need to discuss the problem with group members created a natural context for students to discuss technical issues. In addition, the fact that groups were doing different tasks meant that students had to be able to explain what they were doing to tutors. Finally, the groups gave presentations of their work at the end of the semester (although we would like to include specific activities in the 2-hr classes to develop these skills)
• Testing - in both the group demonstrations and the individual submissions, students were required to demonstrate the testing strategy they used - these skills were developed during the semester by specific activities in the 2-hr classes.
• Planning - a long term project such as the students did for the bulk of the semester requires much more planning than the relatively small assignments in the current CS1 - we provided the students with specific activities in the 2-hr classes to develop planning skill and the PBL context provided practice.
• research and library research skills - the nature of the cryptography problem made use of the library and other resources like the WWW natural and the one group that took the other problem used the library to find search techniques.

4. Staff delivery cost per student is comparable

These costs do not account for the initial set up cost but do account for the continuing long term costs.

The following pair of tables show calculations for the cost of delivering cs1 in PBL-style compared to the current style for semester 1. It largely follows the original proposal submitted to the courses committee but has been amended to reflect our experience this year.

Some terminology needs clarification:

• The full cs1-cohort is split into Sections, each of about 100 students and each Section has
• 5 classes, each of about 20-24 students at the beginning of the year
• and each class would organise students into groups of about 5 students.

• the setting and running of the practical examinations - although in PBL, we use the first practical exam more as a friendly diagnostic tool and do not apply the full practical exam set up and associated costs (currently 0.1 load to Tony and 0.1 load to Jim for a full year).
• managing the marking of the written examination (currently 0.1 load to Jim for a full year).

Those who have lectured cs1, giving 3 lectures per day, 3 days per week for just over 4 weeks will attest that this is an onerous load. This would essentially be replaced by the role of Section leader who will take the 1-hr Section's combined class plus one Class.

5. Space and equipment costs will not exceed lg41/2/5 plus two PBL labs like F46.

6. Students are not disadvantaged by changes in group membership.

There were no changes in group membership. This might be attributed to the process we used to form groups and the way the running made it manageable to maintain groups. We note that some groups had students who suffered illness and misadventure and there was great diversity of background, commitment, competence and performance.

Moreover, there were many situations were the groups meant that students were resilient through various difficulties. For example, in one class two of the eight monitors failed: groups just reformed where in a mainstream class, a similar loss would have been a major problem. Similarly, some students had serious personal problems and the group members helped them get back on track.

Note that groups for the main programming task were not formed until week 4 of semester. before that, we encouraged groups that changed membership for each main activity. This was done primarily because there is so much change in the cs1 class in the first few weeks, typically with around 10% of starters leaving and a similar number of late people coming.

7. The course can be fairly assessed.

The assessment was:

Individual practical work was assessed on a portfolio containing the work done by the individual (code, testing etc) as part of the group task. This was assessed much like any other programming assignment might be.

The groups work was assessed from a sequence of reports through the semester and at the end plus a demonstration in the last week. Both these were assessed by the section leader. In a full implementation this would mean that all the group assessment was done by one person for each Section of 100 students.

Full details of the assessment criteria and marks as used are available.

8. Students do not dislike PBL

Studies of students who dropped out of the PBL class.

• Student One PBL -> normal

  This was the student's first year at the University of Sydney.

  During enrolment this student selected to do PBL without reading the form. After reading the form and discussing the option with friends, the student changed to the normal stream in the first week of term. The student's friends were doing the normal stream. The student is finding the normal stream "hard but OK".

• Student Two PBL -> normal

  This was the student's first year at the University of Sydney.

  This student described himself as having no previous knowledge or experience with computers and low confidence. From the form given out during enrolment this student formed the impression that the PBL course would have a strong practical base. The student chose to do PBL thinking this practical approach would help him gain confidence and learn fast.

  During the first week of term the student felt lost and left behind. The student's confidence was worse and he felt that a lot of work was required to catch up. The student enjoyed the format and group emphasis of PBL but felt there was insufficient instruction.

  The student spoke to his tutor about his concerns and was reassured and encouraged. The student spoke to other students taking the normal course and read their lecture notes. By the end of the first week the student decided to change to the normal course. The student feels more confident and secure with the lecture/tute/workshop structure.

• Student Three PBL -> normal

  This was the student's first year at the University of Sydney.

  This student speaks English as a second language and found working in groups difficult and confronting. The student didn't enjoy the PBL style of teaching and much preferred to attend lectures. The student spoke to friends who were doing the normal course and decided to change. The student finds computer science "very hard".

9. Sufficient staff members a willing to participate

The roles in teaching PBL are:

• curriculum developer. This could be a small team of people who design problems, support materials and assessment guidelines as well as examination questions. In the first year, they will have additional staff development work.

  We estimate a steady state load for this position as 0.1.

• section co-ordinator. This person is responsible for overall administration of a group of approximately 100 students, who would study in 5 groups of approximately 20. They also take the 1 hour class that is attended by all students. This person will collate marks, brief tutors and standardise assessment.

  We estimate a steady state load for this position as 0.1.

• group tutor. This person takes the students for their remaining practical class time, starting with 5 hours per week for the first half of semester 1, 3.5 hrs in second half of semester, 2.5 hrs in semester 2.

It is hard to miss the fact that most of the staff who have had a long term commitment to teaching programming in first year are `keen' to teach using PBL. Only two staff members report being `keen' to teach cs1 in the non-PBL format and one other shows willingness only to teach non-pbl style.

We conclude that the 12 staff already indicating willingness (or keen) are sufficient to run pbl in sem1 1997, with no more casuals than at present.

It was clear from the return mail that several staff do not feel that they know enough about PBL to comment or are awaiting the outcomes of the trial.

We conclude that there is a substantial part of the staff who are keen to teach using PBL and a significant group that needs more information.

10. Other conditions

The only new matters to present are:

• Staff at the CTL have agreed to help in staff development that will be needed.