GameLoop: Scoring Creativity

Back in August, I attended Boston GameLoop, an annual game development unconference. I’ve been going for years, and I’m always impressed by the sheer wealth of knowledge available.

I take notes. Lots of notes. And I’m writing up my notes from this year (in fleshed-out form) for everyone who wasn’t there.

GameLoop participants: Apart from the discussion leaders, I have not attached anyone’s names to these writeups, and I’ve mostly scrubbed personal anecdotes out to maintain the privacy of attendees. Please contact me (carolyn at if:

  • I included something you said that should not be included, or
  • I quoted you without attributing (when you would prefer attribution), or
  • You think I got something from the discussion wrong

…and I’ll make a correction. Thanks!

Scoring Creativity

Discussion leader: Dan Bruno

Dan Bruno is a designer at Harmonix, where he’s been working on Rock Band 4.

Harmonix has repeatedly tackled the question of how to score player creativity – shipped examples exist in Dance Central (freestyle sections), the Rock Band franchise (drum solos in prior iterations, freestyle guitar sections and new vocal improv in the upcoming Rock Band 4), and Fantasia. There was also a cancelled internal prototype called Rock Band Sessions, in which players attempted to build a song from scratch together.

Encouraging creativity can open up exploits – for example, it used to be possible to just hold a single note in RB4 vocal improv and get a perfect score.

Covet Fashion is a fashion design game in which users create outfits to a theme. Over the next 8-12 hours, the outfits are scored by a voting system from the player community. (F2P – you get coinage by voting on others’ outfits.)

Kids game app – players upload tile-based art, and then other players show when they like your work by adding connecting pieces of art onto the edges of yours, promoting it collaboratively. (No one remembered the name of this, though there was apparently a GDC talk on it.)

Approach to scoring creativity: have multiple goals – variety, consistency, etc – and acknowledge players for demonstrating any of them.

In SpaceChem, there’s a pass/fail on whether or not you completed the puzzle, but there’s also a view into how everyone else did the puzzle (statistics about how the number of cycles required, the number of instructions needed, and the number of reactors used.) The latter part is creative feedback.

One way to reward creativity is by requiring players to experience their decisions. Example: a band game where you pick potential band members to play together. The reward + punishment are the same – these are your band members, you’re stuck with them. (I think this was a real game as well, but I didn’t get a name.)

The Axe (Harmonix’s first title), which was really a creative toy. The player could press buttons and move the mouse to change the way the music sounded.

Important consideration: how are you framing what you ask the player to do?

Another approach: try to reward the player for the amount of effort put in.

Another approach: give the player a target. You’re not making art for you but for someone else (a profile provided by the system), who has specific tastes. Then reward the player on matching those tastes.

Objective comparison of creativity: is this better or worse than something else?

Pokemon Snap challenges you to take photos of Pokemon and then rates you on how good your photos are. This feels like a creative game, but it actually isn’t, because the game is using specific criteria about how centered the Pokemon is and whether it’s engaged in “interesting” activity.

If you mask the criteria for scoring, then you can avoid people gaming the system.

Another approach: have the player tell the system what their intention was, and then score the player based on whether or not they succeeded in their intention.

Elegy for a Dead World (a creative writing game) has no actual scoring in-game, but there is a player-driven system for people to read each others’ writing and leave a commendation. This didn’t work out well for them – there are 3 achievements that nobody has, because people are most interested in writing, not evaluating each others’ writing. (The achievements require you to get 1,000 commendations on a single piece of writing. No one has done this yet.)

The problem with scoring creativity: systems score on rules, but people can break every rule and make something amazing.

Views of Minecraft walkthroughs on YouTube and world downloads can be considered a creative “score” for a Minecraft world.

There are many ways to reward creativity that are not actually score-based.

Approach: The game is not telling the player that something is good. The game is allowing other people to tell the player that something is good.

People want to feel that they’ve been creative. That’s a big reward in and of itself.

Telltale Games titles show percentages for your decisions, so you can see what percentage of the community chose what you did. This gives players the feeling of being special when they did something unusual.

It’s valuable to acknowledge player choice, either in the solo mirror context or the social feedback context.

Creativity is best judged by people.

Automated essay scoring systems are surprisingly good at scoring essays and producing believable teacher feedback. Possibly this tech could be harnessed somehow for games.

Approach: Machine learning. Backlog information and deduce criteria for “being creative”.

Approach: The system passes judgment on type, not quality (example: “That was an intense solo!”)

People get confused by unbounded creativity. They need guidance and a frame for onboarding.

TIS-100 is similar to SpaceChem in that it presents a puzzle that must be solved, but can be solved multiple ways. (TIS-100 teaches modified assembly.) One person in the room was so inspired by TIS-100 that he went on to write a prime number generator in assembly.

A game that encourages people to be creative can then convert into a toy that allows them to continue being creative.

Sentris is a music/puzzle game that allows the player to solve songs in multiple ways.

Scribblenauts! The inherent reward of this game is solving problems in the most interesting ways possible.

You can encourage creativity by scoring the player on something other than creativity, while requiring them to work in a creative way.

Crayon Physics was especially good at the goal/creativity partition.

Having a goal is a good way to onboard players to the creative process – but sometimes the presented goal works in opposition to player expression and creativity. Examples of games where the announced goals actually work against player creativity: Dance Central 1 freestyle mode, Angry Birds, Alphabear (kind of), Minecraft (kind of), Dwarf Fortress.

Twitch and streaming allow players to show their creativity with games that aren’t inherently creative. Speedrunning can be an expression of creativity.

How can we make games recognize “that was really cool?”

Creativity is intrinsically valuable. Scoring is extrinsically valuable.

Creativity is present when players not only overcome obstacles, but create those obstacles in the first place.

Bulletstorm is very good at acknowledging when the player does cool stuff (but it’s disappointing when the system fails to notice.

Rocket League asks players to choose between doing cool things and clever things.

The ability to share you creativity/awesomeness with other people is really valuable.

The latest Mario Kart is really bad at picking the highlights reel. Suggested replacement from the room: turn on a Kinect-style audio sensor and pick the highlights based on when the volume in the room was loudest (because people were cheering and yelling).

MarioMaker is coming out soon! Everyone was excited about this.


This site is supported through Patreon. Please consider becoming a patron!

Bookmark the permalink.


  1. I was cranky about Crayon Physics’s goal/creativity partition. TLDR, I felt like if the goals were more varied then they would encourage creativity themselves, instead of the game needing to give extrinsic rewards for creativity. Although I do like the idea that you yourself are the best judge of your creativity.

    In fact I’d say this might be another game where announced goals work against creativity, sometimes. Scribblenauts (one of only other two games in the post I’ve actually experienced) handles it more to my satisfaction because the creative solutions aren’t prohibitively more difficult than the boring ones, where there are boring ones. Well also because the person who plays it mostly wants to make stuff to blow up. Come to think of it this is also what that person wanted to do mostly in Crayon Physics, but there wasn’t as much scope to do so.

    • Hmm. I poked Crayon Physics once, but it was something like 5 years ago, so I don’t actually have an opinion here. Maybe I should re-investigate.

      • That may sound more negative than I feel–it’s a cool concept and does a lot of neat things. It can be kind of frustrating (for me at least) because very small changes make a big difference–if your line hinks up a few pixels it can mess you up. There may also be a bit of “Why on earth are you using a touchpad to play this?” at work here, which is probably not an issue you’d have. And maybe it’s worth thinking of it as a semi-hardcore trial and error game, despite what the music and art are telling you. My complaint is more that I was hoping for something more awesomer. More rockets and stuff.

        Anyway, I’d definitely check it out if you have a copy or can get one cheap.

        • OK, having gone back to Crayon Physics for a while, I think I can articulate what bugs me about the creativity rewards.

          The way progression in Crayon Physics works is that you get a star for finishing a level, and an extra star for finishing it three ways, “Elegant,” “Old School,” and “Awesome.” All but the last set of levels can be unlocked without getting any extra stars. The last set of levels takes 120 stars to unlock, and there are only 75 levels up to then (if I’ve done my math right), so you need fully 45 extra stars to unlock it–that’s 60% of the levels.

          You are free to mark any solution as “Awesome”–that’s the self-checked creativity reward, which IMO does it right by letting you judge. “Elegant” solutions are ones where you can only draw one object, which often forces you into creative solutions, though a particular kind of creative solution and obviously one that’s judged by a rigid rule. “Old-school” solutions are ones where you can’t use certain resources, which I find does not push me toward creative solutions, but toward certain hacks that tend to be kind of tedious to implement. Like, I have a fast way to get the ball up a hill using pins and ropes. I have a slow way to do it involving drawing dozens of tiny objects under the ball. Old-school solutions prohibit drawing pins and ropes, so I fall back on the slow way.

          (Think of it like the difference between a nethack conduct, illiterate or foodless maybe, that makes you run through the dungeon and take lots of risks, and another conduct that people do by spamming Elbereth and letting their pet take every fight. Elbereth spam is the worst, isn’t it?)

          So there are tangible rewards for creativity, which you’re allowed to judge yourself. But in order to collect the rewards, you have to do two other special conducts, which sometimes feel like achievement-hunting and one of which can turn into a joyless grind. (I changed my metaphor: Think of it like microscale pudding farming.) Furthermore, to collect the full reward you have to do an awful lot of this achievement-hunting and self-judged creativity; maybe not all the levels inspire me to creative heights.

          And the rewards aren’t just a nice hat, or an extra ending, but access to the full content of the game. (Adding to this, the way the gating is presented makes it feel like “the rest of the game” rather than “a bonus after you’ve completed the game.” The last normal level is called “The End?” but it doesn’t give a credit roll or ending cutscene, and the remaining levels are sitting there on the world select screen like any others.) So in the end, if I have 119 stars and all I need is to mark a solution awesome, I expect I will, even if I secretly don’t think it’s that awesome.

          Whereas all along, the game lets me watch my old solutions, and that’s what I do with the ones I really think are awesome. (Which often are spurred by the elegance challenge, although sometimes they’re just ones where weird things happened.)

          OK, that went really long! TLDR: if you’re going to reward creativity, don’t tie those rewards too tightly to things that may oppose the sort of creativity you want to reward. Especially don’t tie it in to what may seem to be essential progression in the game. And extrinsic rewards may not be necessary–you can just design your game in a way that spurs creativity and do something that lets the players (and others!) appreciate what they’ve done.

          Or maybe I’m overthinking it–the person I sometimes play with doesn’t care about unlocking the last levels and he just wants me to mark things as awesome whenever we’ve done something he likes. Often that involves rockets.

          • Thank you for the detailed explanation!

            I see the problem – you’re being encouraged to fib about how creative you think you were. I agree that does conflict with other design intentions.

            Love the NetHack analogy, which of course spoke to me. (I confess, I usually forget about Elbereth. Some of my characters would live longer if I didn’t!)

          • Late addendum: I wound up accumulating enough elegant/old-school solutions that I didn’t feel I needed to fib that much about how awesome my solutions were, though I felt about weird about marking eight solutions awesome when they used pretty similar tricks (an awesome trick, though).

            But, more relevantly, that center island turned out to be one special level that gave you an ending and a credit roll. So really a lot of my complaint comes from the presentation on the level select screen–if it somehow weren’t presented as like the rest of the levels then it wouldn’t have felt as much like necessary work to get to it.

  2. Ooh, I skipped this talk because I saw the word “scoring” and assumed it was about audio design, but it sounds really awesome! (Though it also conflicted with the multiplayer educational games talk, so I don’t have too many regrets.)

    Was the focus solely on digital games? I’ve played a tabletop RPG where players score points based on how much the other players like their roleplaying. On the one hand, it’s nice to be complimented, but on the other hand it felt a little tacked-on, because there wasn’t really a need to declare an objective winner.

Leave a Reply

Your email address will not be published. Required fields are marked *