I'm just catching up from the excellent Game Education Summit earlier this week. Here are my notes:

Donald Marinelli, keynote:

• Much of today's educational system is obsolete. Summer vacation exists to let young people go home and help their families with farm chores. How many K-12 students do you know that are planting wheat right now?
• If you are building a game for a class, build it for someone. Give it a purpose.
• ETC's "secret sauce" is that they let students teach each other.

Terrence Masson, on building Northeastern University's curriculum:

• Interesting way to structure a Capstone course with 10 people: first day people give their project pitches (most students pitch several alternative projects). Second day, students narrow the pitch list down to the two projects that the class will work on; students choose their teams (split into two teams of 5); each team assigns roles and chooses their project lead. Essentially, the students drive everything.
• Another interesting thing about this program is the requirement of two non-adjacent semesters in internships/co-ops. The benefit is that students keep the faculty honest: "What do you mean we don't have Zbrush on campus? That's what everyone is using now!"
• Note to prospective students: at this particular institution, the program is called "game design" but it is actually "game development". This points to the importance of schools and industry using a unified set of terminology.

Jessica Hammer, on how to teach creativity:

• First, you have to define what "creativity" is, because it is an overloaded term, and there are different kinds of creativity. She defined it as "appropriate novelty" -- something that is new, but within a given context/domain. (If you ask students to design a game and they write an essay instead, and try to define an essay as an innovative new type of linear-narrative game, this is not what we are looking for.)
• Creativity happens within a context or domain (i.e. within a set of constraints). There is a virtuous cycle within a field, where the domain influences individuals; the individuals produce creative work within the domain; and the gatekeepers who see this work then influence and redefine the boundaries of the domain to compensate. In the case of teachers, the classroom is the domain.
• One problem in practice is that we often measure creativity after the fact: we look at the final product and decide if it is creative. Unfortunately, this tells us nothing about the process used to create it... and if we want to teach creativity, we want to teach the process!
  There are three aspects to the creative process that students need to understand: the generation of novel ideas, the ability to decide what ideas to pursue (since ideas are a dime a dozen, once you learn how to generate them), and the motivation to follow through on your chosen idea and do the work to turn it into a final product. The class should focus on these.

Jessica's hints for course design:

• Begin with outcomes. "The goal of a course is not to deliver content, but to transform your students."
• Consider the length and pacing of the class. If there is not enough time to generate ideas, fail many times, and still finish, students will take fewer creative risks.
• "Personal attention is valuable currency." Keep class sizes small when possible. Group work can enable larger class sizes by having you deal with a small number of groups rather than a large number of individuals.
• Recruitment is rarely thought about, but is important. The more diverse your class (or, um, game studio), the more creative the ideas you're likely to see. When approached by a female and/or minority student, be supportive and ask if they have friends who would also be interested in taking your class. Also, consider the accessibility of your classes: if students can choose between written or verbal assignments, you will see higher enrollment among those for whom English is a second language.
• Use a lot of class time on playtesting and peer review. Professor should model appropriate feedback, to show what it looks like.
• Encourage uncertainty, in projects, classes and life. "Your game design education does not end when you leave this class. It has just started."
• Don't just have students solve problems that are handed to them, because this is not how real life works. Have them create and seek out their own problems to solve.
• There is a negative relationship between the time and emotional investment in a project, and willingness to take risks. In the middle of larger projects, consider giving smaller-scale "lightning round" design challenges that encourage creative risk-taking -- for example, email students with constraints of a challenge at noon one day, and they have 24 hours to post a short concept in an online discussion group. These are not a major component of the course grade; they are a chance for students to show off. Examples: "Design a game to be played in the waiting room of an ICU while you're waiting to see if a loved one lives or dies." / "Design a game for NASA that can keep astronauts alert and interested on a 3-year mission to Mars." / "Design a game for Obama's cabinet to help improve their effectiveness as a team."
• How do you assess creativity? Note that you get what you measure; students will game any system. If you want to reward risk, you have to give grading opportunities for it. Jessica splits the final project grade into three equal parts: the game itself (the final result of the process), the theory (students write a companion paper that shows the connections between the theory learned in class and its expression in their game), and the process (students submit a "process paper" that includes everything that was part of the project but not visible in the final form: raw data, early playtest results, early versions of the game, mechanics that were tried and abandoned... whatever the student wants the instructor to see).
• Divide larger projects into many feedback cycles / milestones. Iteration is part of the creative process, and class projects should reflect that.
• The nature of instructor feedback is important. If you just give a grade, that carries very little information. Extensive written feedback is much better, but can take a lot of time; to manage this, favor group projects or smaller numbers of submitted projects per-person.
• As the instructor, you are a strong influence on the culture of the classroom. You want students to feel comfortable taking risks, both in their projects and by raising their hand to make suggestions/comments in class. How you react when students say something "stupid" has a huge impact. Suggestion: draw from the "Yes, and..." technique of improvisational theater -- accept everything in class, refuse to shut down an idea or say that it's wrong, and instead challenge yourself to find the nugget of truth in there.
• Give students a sense of mission. People are more creative under stress when they believe in the importance of the final project. Because of this, fewer projects (reduction of workload) can paradoxically lead to students spending more time and doing more work... as long as the projects they have are the right ones.
• Self-efficacy is important: students must believe they can perform well in the class. Corollary: we as teachers must believe in our students. Research has shown that a teacher's belief in a student's ability to perform is often self-fulfilling.
• Praise students not only for their projects, but also for exhibiting personal qualities that we want them to continue: hard work, persistence, etc.

Walter Rotenberry (Wake Tech), on the challenges faced by Community Colleges:

• The ideal case for a Community College is that you are based in a "hub" of the game industry, so that your graduates have immediate local employment and internship opportunities. What if there are no game companies in a 100-mile radius?
• An alternative: focus on entrepreneurship. Require your students to take classes in business, enough that they would be comfortable building their own startups. Give students the tools to start their own local studios.
• Wake Tech's approach to a two-year program is interesting: cover a little bit of everything (at least one or two courses from programming, design, art, production, audio, business, game studies, etc.) to give a well-rounded background. This provides a foundation for transfer to any four-year school. I thought this was an interesting approach -- in my experience, usually with only two years to work with, Community Colleges focus on art or programming. I'm not sure that one approach is "better" than the other, but I can see the use of both.
• Encourage students to take courses in other relevant areas and departments: theater/drama, history, storytelling, etc. - the bonus is that in many cases there is no need to add specialized "game" classes, you can work with what is already there.
• Wake Tech got an 800ドルK grant from NSF to develop their curriculum. This money is not allowed to go to new hires, but can be spent on curriculum development and new equipment. Other schools may be able to get similar money, so it is worth looking into.

G. Michael Youngblood on Computer Science-focused game research:

• Students can get involved through an NSF program called REU (Research Experience for Undergrads).
• It's easy to get academics involved; this is what many of us do. Biggest challenge is collaboration across departments, since games are so interdisciplinary.
• If you're working in industry and want to get involved, the easiest way is to visit. Invite some local researchers to lunch. Look at their stuff, read their papers, ask questions on what you don't understand.
• You can support students for your own benefit! If you have an idea you'd like to test out, 1100ドル per month for a grad stipend x 5 months = 5500ドル for a prototype and white paper. This is a pretty good deal if you're a large studio with an R&D budget! Note that some schools and some researchers will ask to charge overhead (to cover costs of building maintenance, utilities, etc.) that is as much as 50% of your grant. You do not have to put up with this; operations costs are not required for non-governmental grants, and you can offer the funding on a take-it-or-leave-it basis. Most universities would rather accept money than turn it down.
• Be on a university's Industry Advisory Board. Suggest that they research difficult, interesting problems.

Michael's list of things that the industry should keep in mind when dealing with academic researchers (particularly in Computer Science):

• Academics are extremely "paper-focused". If there's not a publication in it, then it doesn't matter.
• Academics are always behind and have too much to do.
• Like any programmer, academic researchers will overstate their ability to deliver for nearly everything.
• If a study involves humans in any way (such as, say, using college students in a playtest of your game), learn about the IRB process.
• The field of games research has matured quickly. Two years ago, "I'm working on a game" was good enough to get published. Today, you must also be able to show why your game research is cool or useful in some way.

Random tidbits from side conversations:

• Games and learning are both negative feedback loops: once you have learned something, you don't want to learn it again. This drives students to learn something and then stop. We need to find a way to counteract this by including a positive feedback loop, so that great students will want to keep learning and to learn more.
• I wonder if a school has ever hired an entire small development studio. Granted, not everyone has teaching skills, but you would get complete coverage of all subject areas and you'd be hiring people who already know how to work together as a team.
• Giving students a general literacy of classic games is important. One approach: have students write "reviews" of classic games. How do you get them to play older arcade or console games in the first place, when the original hardware is hard to come by? Several alternatives: first, many companies are repackaging their classic games for sale on modern systems (Atari Flashback, Midway Classic Hits, original NES games downloadable on Wii, etc.); second, with questionable legality, you can download emulators such as MAME; and third, particularly useful in class, you can find short gameplay videos of just about everything on YouTube to show what some of these games looked and played like.

There is an inherent conflict between the nature of classes and course objectives, when it comes to designing a game as a class project.

The best way to learn to design games is to make a rapid prototype, fail miserably, figure out what you did wrong, and try again. Repeat until you get it right. In order to do this, the student has to feel like it is okay to take risks, that it is perfectly acceptable (even expected) to try crazy stuff that may simply not work out.

But of course, this is for a grade. Enter the fear of failure. Or, it's not for a grade at all. No threat of failure, but likely no effort put in by students on an "optional" project. Is there a way around this paradox?

Here's the method I'm currently using:

• My non-digital game design project has four milestones. The first is just a high concept, target audience, basic information (number of players, etc.) and some core mechanics. The second is a rough but playable prototype. The third is a playtested prototype, with the mechanics finalized or close to it. The final milestone is a polished product.
• All milestones are graded. Early milestones are easy points -- just turn in something, anything, as long as it works. Later milestones are graded based on the quality of the design -- you'd better have done some iterations.
• For the future, I'm thinking that early milestones should be worth fewer points than later milestones. This puts less importance on early work and more focus on the final product.
• On the days where milestones are due, students bring their works-in-progress to class and present the work for peer review. This also gives me a chance to see how the projects are progressing. In the future, I should probably just give a grade right then and there for the early milestones.
• Make it clear to students from the beginning that the more they iterate on their project, the more they playtest, the more they fail and then change, the better their final project will be. Unfortunately, this is one of those things they might just have to find out the hard way for themselves. I'll try bringing in a student work from an earlier course (with permission) in its various stages of completion, to show just how much difference playtesting can make.

People who study education will talk a lot about the differences and similarities between assessment and evaluation. I'll spare you the details, but basically we're dealing with the question: how do we know that learning actually takes place when a student attends a course?

The traditional response is to give the student tasks (often in the form of a final exam or project), and hope that their ability to perform well correlates with their mastery of the subject material. But there's the rub: what we measure (performance on a task) is still different from our goal (learning). There's no way to measure an abstract, transparent concept like "learning" directly, so we have to find indirect ways to take a guess.

It strikes me that there's a parallel here with game development, and metrics-based playtesting in particular. During playtesting, what we'd really like to do is know if the players are having fun, but there's no way to measure "fun" directly. So, we take measurements of things that we hope will correlate: how long did each tester spend on this level, how many players finished the game, how long was the average play session, and so on. But it's still indirect measurement.

In both cases, we still live in an imperfect world. Someone tell me when we solve one problem, because it probably means that we'll have solved the other one as soon as someone puts two and two together.

Having just graded another midterm, I realized something.

My favorite part of grading is when I ask a question that I know is difficult (but meaningful), and I see a student just totally nail the right answer on the head. It makes me feel... validated, like here's someone who was paying attention, here's something that I was able to teach.

My least favorite part is when I see an answer that's totally unintelligible, like the student was answering a question that I didn't ask, and it's clear that they either misunderstood the question or else that I'm misunderstanding their answer. On the one hand, I teach game design, not communication, so if the student understands the question and has the right answer and just has difficulty communicating then I feel bad about taking off too many points for it. On the other hand, I can't justify giving points for an answer that I don't think is right. So, I have to dock the points and hope that if I'm wrong, the student has the guts to call me on it (which actually happens a lot less often than I imagined it would). But I just hate the uncertainty.

Game designers love tradeoffs. Tradeoffs are one kind of "interesting decision". Life is full of all kinds of tradeoffs (it has in fact been argued that life is simply a massively-multiplayer game... with extremely realistic graphics). Teaching has its fair share as well.

Here's one that I'm discovering: the more useful an experience that some exercise (an in-class presentation, homework problem or project) is, the more subjective the grading is.

It's very easy to create a multiple-choice exam where each question has exactly one, clearly correct answer. The grading is easy and the grades themselves are inherently fair. But students don't actually learn anything by taking the exam, they merely show you what they've memorized. The same is true for homework questions that are direct enough to have exactly correct answers.

By contrast, essay questions, open-ended student projects and creative exercises all lend themselves to both student and teacher interpretation. Even if you think you know what you're looking for in order to grade objectively, some students will surprise you with unexpected answers and you'll have to revise your grading methods. Sometimes a student answer will, on reflection, be better than your own. Sometimes you will simply not understand what a student is saying, and they will be completely correct and it's your own fault for misinterpreting their answer (you can weasel out of this by saying "if I didn't understand it then it was poorly written" but I would consider that the height of arrogance in most cases).

And yet, these subjectively-graded exercises are the most valuable because they force the students to actually design something.

In terms of teacher evaluations, this means that any class where I score well on "relevance to the real world" is probably also a class where I score poorly on "fairness of grading"...

I prefer my exams to contain essay questions, because they offer greater opportunity for expression; they're harder to copy without cheating being obvious; and they give me a better idea of just how much my students grasp the material. The down side is that they take forever to grade, so I can certainly understand the practical consideration of a multiple-choice exam -- especially since the professor is under time pressure to turn in final grades at the end of the course.


Brenda, another game-designer-turned-professor, says to me about multiple choice:

For a multiple choice exam, you can create the silliest answers, and people will select them. It's like a mini-game. My favorite silly answer is "Costikyans". I use it a lot. We talk about Greg's articles regularly in the quarter, and it still surprises me when someone selects his name as an answer. In this case, the correct answer should have been "Semiotics."


I'll have to try that some time, just for laughs.

And now that she mentions it, if I ever make an RPG, I'll be tempted to name the currency the Costikyan. It doesn't sound that much more odd than, say, Gil, Potch or Zenny. "A sword of flame? That'll be 300 Costikyans, please."

During the process of grading a final exam, one student's answer to a particular question was very different from my own "correct" answer, but persuasive enough to make me question my own assumptions.

After looking a few things up, I think the student is right. I now have to change my own answer.

Naturally, this was the last exam I graded, so now I get to go back through every single other one and redo the grading on this question. (I won't take points away from students who made the same mistake I did.)

I have to wonder how common this is. I know that every teacher says they learn as much as their students, but I didn't expect to be learning things after the class was already over.

A long time ago I was thinking about different ways to grade a class. For the intro class, Game Industry Survey, I ultimately decided on scoring out of 1,000,000 points -- it's more accessible, and any given assignment feels uplifting even if you only got 50% (which most students don't, since it's not supposed to be a difficult class). It served its purpose: it amused the class on the first day and set the stage for this being more about fun and learning than traditional skill-and-drill.

In the interest of science, I decided to use the Zelda Heart method for my Game Design Workshop-like class this Winter. The students are already (mostly) familiar with me as a teacher, they're ready to take a step towards being a game designer, so I wondered if they'd be more open to alternate grading styles. So far, the reaction is very positive; students chuckled when they saw a "life meter" on the last page of the syllabus, and they realized (without me having to point it out) that turning in assignments late amounted to "poison damage". So far this seems to be working... at least for an advanced class.

One of my upcoming classes basically amounts to a group of students working together in a team to make a game. I expect them to have some great experiences, and the classroom is a much better place than the first industry job to make some serious beginner mistakes. So, I'm really excited about this one.

Then there's the question of grading. If I have a programmer, a couple of designers, a producer, an artist, maybe a sound guy, all working on different aspects of the same project... how do I grade them?

So far, I've come up with several possible methods:

1) The "I know it when I see it" method. No quantitative grading at all, I just assign whatever grade I feel the student deserves. While this would probably give the grades that are the most in line with what students have done, it's a terrible stress on the students themselves, and it puts a lot of burden on me to Get It Right. And if a student complains that they got an unfair grade, I have no defense.

2) The project-based method. I grade the final project, not the individual students. Everyone gets exactly the same grade. I don't like this, because it doesn't allow for variation in student abilities.

3) The discipline-based method. I grade the programming on the final project, and anyone who was a Programmer gets that grade. I do the same with design, art, audio and production. Everyone gets graded on their own work... except that you quickly realize that everyone's work affects everyone else. Maybe the game has a brilliant design, but it just can't be implemented by the programmer(s) in the amount of time we have; this is an issue with design and programming and production, so who gets a lower grade?

4) The method I'd like to try is a hybrid between project and discipline. I grade the final project on its various aspects, and each student gets a different weighting of those aspects based on their role:

• Programmers: 40% programming, 5% design, 10% art, 10% audio, 15% production.
• Designers: 15% programming, 40% design, 5% art, 5% audio, 15% production.
• Artists: 5% programming, 10% design, 40% art, 10% audio, 15% production.
• Audio: 10% programming, 5% design, 10% art, 40% audio, 15% production.
• Producers: 15% programming, 10% design, 10% art, 5% audio, 40% production.
• Everyone also has 20% class participation. Class time is roughly the equivalent of team meetings, and not showing up for work is obviously bad.

Where the five categories are roughly defined as:

• Programming: does the game work? How buggy is it?
• Design: is the game fun?
• Art: does the game look good?
• Audio: does the game sound good?
• Production: did the game get finished on time with all of the intended features?

I'm hoping to impress on the students that their work affects the rest of the team, and the rest of the team's work affects them, and that in the industry they're ultimately "graded" on the sales of the final game above all else.

Now I understand why so many homeworks I had to do as a student involved right answers: specific numbers, computer programs that either ran or didn't run on test data, multiple choice. It's because it's much easier to grade; in some cases it can even be automated with those ScanTron things.

Even the essays I had to write were typically on a given subject, and everyone in the class was writing a different variation of the same essay. These are harder to grade since you actually have to read them, but it's still relatively easy because you know the kinds of things you're looking for.

Well, I made the mistake of starting my first assignments with "choose any game"... so everyone's assignment is going to have different content. I also didn't specify an exact format, so some people write paragraphs while others give bullet points (and both are equally valid). So for each student I first have to look at the game on Mobygames and Wikipedia and IGN/Gamespy/Gamespot to understand their game, then look through their description and see if it's reasonable. With 30 assignments total between my two classes, that's a bit of work.

Next time I give these classes it will be easier. I won't need to prepare each lecture in advance (I can re-use most of the content that I'm creating now) so I'll have extra time for grading. I'll also give an example of what I'm looking for so that students have a format they can follow.

I'd like my classes to feel distinctly game-like, so it's only suitable that the grades themselves are game-like. So far I've considered two ideas, and I'm trying to decide between them.

Method 1: Dance Dance Revolution scoring.
The class is out of a maximum of 1,000,000 points. A typical assignment might be 100,000 or so. Giving out awards of a few thousand here or there for class participation is easy, and it's more points than they'd get in all their other classes combined.

Method 2: Zelda scoring.
You start with a full health meter of 100 hearts. When you lose points on an assignment, your grade "takes damage". Extra credit provides "healing". You can keep track of your own grade on a life meter provided on the syllabus.

Zelda scoring is more realistic in that students always know exactly what their maximum possible grade is, and overall where they stand. But it can also be more demoralizing and intimidating since you're always losing points instead of gaining them, and it sets up an adversarial culture between me and my students since I'm dealing damage with every assignment.

Do you have a favorite? I'd love to hear your thoughts.