So You Want To Be A Video Game Tester

So you’ve watched Grandma’s Boy a gazillion times and think you have what it takes to be a video game tester?  Think again robot vagina!

grandmas_boy_pic

Software Quality Assurance testing can be very grueling and tedious.  I am here to share some of my experiences and knowledge working in the video game industry as a tester.  During my gaming career, I worked on major titles such as Command and Conquer 3: Tiberium Wars, Medal of Honor: Airborne, 007: The Quantum of Solace and The Saboteur.  I hold fond memories of my time spent, but the journey through those projects was not all rainbows and unicorns.

The movie Grandma’s Boy is a great classic comedy film, but besides taking place in a game studio, it’s far from the truth of testing. We’re talking about 80+ hour work weeks, overtime catered dinners starring glue-like tasting mac and cheese, extremely repetitive tasks and fist clenching annoying co-workers. These are just a small taste of things to come along this journey.

Not all is bad however.  The job can be very rewarding if you take it seriously.  Instead of being the guy that’s trying to rack up kills and smack talking those they just head-shotted during an MP play-test, be the guy who analyzes balancing issues or is looking for sections of the map where the player can become permanently stuck.  This can be the difference between your name in credits and getting shit canned.

video_game_tester_quote

So what does it take?  First, you need to be somewhat educated and pretty good with a computer.  Having some basic programming knowledge is a definite plus as you can better understand code spit out during crash dumps.  Excel skills greatly help as many of the smoke tests and test cases you run will require updating spreadsheets.  If you want to be a senior tester (and beyond) at some point, you will also need to be able to create those test cases from scratch.  Having descent typing skills and proper grammar is a must as you’ll be writing a lot when describing how to reproduce the issues you found.

Still with me?  Ok, now the BIG question.  What is a bug?  My definition of a bug is an issue that takes the player away from the immersion of a game whether it is in the form of a crash, a miswrapped texture or an accessible hole in the level’s geometry.  That was the answer I used when I interviewed for my 1st QA testing position and the same answer I used for subsequent interviews.  You can look up the official definition in a dictionary if you want something more technical — the principle is the same.

Mercenaries_3_bug

Now, if you find a bug, you need to make sure no one else already found that issue.  Not in the database?  Awesome!  You’re not writing a “dupe!”  Before you even start to write up the issue, you need to redo what you just did to cause the issue to reproduce.  Do this three or more times to see how reproducible the issue is.  Once you’re able to successfully reproduce the issue, take a screenshot, video capture, etc of the issue and prepare to write up your bug report.  Now think, how would you describe this issue for someone who has never played this game and have them reproduce it?  This is where your documenting and general computer skills come in.

Every game studio is different on how one submits issues into their database.  I will describe the method I used primarily during my career to submit bugs.  Below is an example of a bug report ready to be submitted.

Summary:
Player encounters a crash when clicking “accept” to exit sound options menu in pause menu after adjusting slider.

Description:
Player encounters a crash when clicking “accept” to exit the sound options menu while in pause menu after adjusting slider.  This does not occur when the player accesses the sound options from the shell, only from in game pause menu.  Also, game does not crash when player does not adjust slider before clicking “Accept”.  Only occurs on PC version.  Progression stops.  Note: See attached dxdiag for tested PC specs

Steps to Reproduce:
1.  Load any level of game (level xxx used in submission)
2.  Once in game and out of level cut scene, press “start” button
3.  Go to “Options” and press “A”
4.  Go to “Sound” and press “A”
5.  Select any sound option and adjust slider left or right to make a change
6.  Go to “Accept” and press “A”

Result:
Game crashes when player selects “Accept” inside of Sound option of pause menu.

Severity:
A – Crash – Progression Stopped

Reproducibility:
5/5 times – 100% reproducible

Solution:
Game does not crash when player selects “Accept” inside the sound options of the pause menu

Type:  Code/Engineering    Location:  Shell/HUD    Build:  2093.14    Assign to:  Code Lead    SKU:  PC

Attachment:  BugXXX.dmp, soundoptioncrash.jpg, dxdiag.txt

In case this write-up sounds like a bunch of gibberish to you, I’ll explain the fields a little:

The Summary is where you describe the issue found to its most basic description.  This is what will pop up when searching DevTrack (industry standard bug tracking software) for you and others to locate your bug.  Make sure the words used are universally understood to make the submission easier to find.

Description is where you can go all willy nilly and describe the bug with all relevant info you find to help the developer fix the issue.  Sometimes you can mention what video card you have installed, OS, or if the issue occurs on different SKUs (XBOX, PS3, PC, etc.)

Steps to Reproduce is a pretty important one.  Most developers that work on games are too busy to actually play the game they are working on.  You have to assume that they’ve never played the game and you have to describe step by step exactly how to reproduce the issue found.  This can become complicated in some bug submissions. For example, on a certain floor, around a certain corner, of a certain building, on one of the largest maps in the game, there’s a bug.  If the issue gets that extreme, you may need to add quadrants so the dev can easily ghost to that location.

Result is where you state the bug that occurs when following the steps.

Severity is how serious the bug is.  The example used above is a crash bug — a progression stopper.  This is as serious as a bug gets and must be fixed before the game ships.  Severity is usually categorized as: A- Must Fix, B- Major, C- Minor and D- Suggestion/Not a bug.  Many games ship with B and C level bugs as many of those do not break immersion or progression.  An example of a B level bug is invisible collision that does not block the path for progression.  It is annoying, but not necessarily game breaking as you can go around it.  C level bugs are generally art bugs such as stretched textures or assets floating in the air.

Reproducibility is how many times you were able to reproduce the issue found. A 100% rate is ideal, but sometimes it’s hard to reproduce getting stuck between geometry as the user needs to jump at a certain angle to get stuck between a wall and a hard place.  Those types of bugs would usually require the tester (you) to go and reproduce the bug on the dev’s computer.

Solution is where you state what the outcome of fixing this bug should be.  Simple and straight to the point is good.

Type is what kind of bug it is.  It can be code, design, art or sound (I hope I didn’t miss any).  Knowing the type will help direct you to assign the bug to.

Location is where the issue occurred.  Was it in the Shell, level XYZ or is it something that can occur globally across any level?

Build is what current build number you found the bug on.  Every time a dev makes a change, they upload their affected file changes via Perforce and it updates the working build to a new change list.  Usually you will test on the most stable build from the previous night when you get in, but will probably work off of 2-5 new builds throughout the same day.  Often the issue is already fixed by someone else’s fix when it actually reaches the dev.

Assign to is who you assign the bug to depending on what the Type was.  Some studios let you directly assign the bug to a developer or simply assign it to a “dump” where the lead of that department will divvy out who works on it.

The SKU is what system you found the bug on.  Depending on your title and studio size, you may be working on a game that is PC only or could be released on PC, PS3, 360, Wii, DS, etc.  Most of the bugs will occur on all systems but some can be sku specific such as a game crash involving pressing the Xbox blades button.

Attachment is stating what files you have attached related to the bug.  Most of the time a screenshot is all you need, but on more complicated issues, video capture is needed and crash dump files to extract data.

Borderlands2_Bug

Now that the bug has been written and submitted, a few things may occur.  If you are a low level floor tester, your bug is sent to a QA dump that the senior testers will go over and evaluate.  They may send it back to you as an “NMI (Need More Information)” and you will need to revise and add more information and resubmit.  It can also be sent back to you and closed as a “dupe” because you failed to locate the bug already entered in the database.  The bug can also be sent back as a “NAB (Not a Bug)” or “WNF (Will Not Fix)”.  This usually occurs closer to the launch date of the title where there simply isn’t enough time to fix the issue.  This is usually why games ship with bugs.

If the bug is not sent back to you, it was successfully submitted to the proper channel where someone will eventually fix it.  Once the dev states that they “fixed” the issue, the bug will be sent back to you to confirm it is fixed.  You will need to check on a build that is the same or past the build number that the dev stated the fix was on.  If the issue wasn’t fixed and still occurs, you send the bug back to the dev.  If fixed, you can set the bug to closed and pat yourself on the back!

So, how many bugs would one write?  During my last QA position, I had a bug count of over 2,000 during a 6 month period, but generally an average per day is around 7-10 bugs.  Some days it can be zero to 1, another 35+.  Really depends on what you are assigned to focus on.  You can only find so many bugs when you are assigned to test AI.  The NPCs can only path so many ways!  Don’t worry about your bug count.  Lots of testers like to brag about their count, but they’re usually filled with poorly written C bugs.  Focus on quality, not quantity.

Still think you have what it takes to be a tester?

hot-shots-part-deux-fight-bug

Well, my previous post generated some positive buzz and I’d like to say thank you to all who read my post! I am ready to dish out more experiences and info about Software QA testing! I collected some valuable feedback from readers (and trolls) on what they wanted to read and will no doubt address those aspects now and in the near future.

In my previous post, I scratched the surface, focusing primarily on preparing to be a tester, writing out a bug with details and the process that occurs during the life of the bug submission. Today I would like to focus on the difference in working as an embedded (developer side) tester vs. publisher side. I have worked in both environments, primarily developer side.

ACT I: PUBLISHER SIDE
My first QA position was at EA Los Angeles. This contract position was located on the main testing floor that basically was endless rows of ultra-small cubicles. How small is this space you ask? Well, imagine having one work computer to the left of your feet, one massive heat exhausting gaming tower to the right, three monitors (two for the main computer and one for testing), an Xbox 360 test kit, PS3 test kit and oodles of wires, KBM/KBM switches, VCR (whua?), controllers, headsets, etc. Now imagine all of this equipment accompanying you on a 4 ft wide desk with the back of your chair banging the tester directly behind you when you rocked back.  It was a tad worse if you had a PS3 dev kit.  Those are the size of web servers!

If you were testing on a console and required video capture, you would need to turn on your archaic VCR and record your segments.  Next you would need to go to the video capture station and convert your video to a digital format to attach to your report.  Luckily for me, I tested primarily on PC where FRAPS was my best friend, but always dreaded having to use the VCR when needed.  This was an ass backwards method of recording, but it was what we had to do at the time.  Even more stupid was the fact that we were unable to capture screenshots for PS3 from Sony’s ProDG Software.  We had a community digital camera to take a picture of our screen until an engineer was able to update our QA tools to capture PS3 images.

Being a floor tester publisher side also meant that you had to deal with varying personalities of your team.  I have made numerous friends, some of which I still keep in touch with, but not everyone clicks with each other. Imagine being in a large classroom where every type of personality is present.  There are the slackers, smelly kids, the socially inept, class clowns, egotistical bastards, “girl” gamer, bad joke teller, elitists, etc.  Couple that carnival with burned out senior testers who originally got the position because of bug count and not managerial skills.  Sounds like a recipe for grumpy cat, but it doesn’t end there. Tack on “crunch time” mandatory overtime for 3 months straight and kiss your already nonexistent social life goodbye. But at the end of the day/project, despite this rag tag group, all of you have a love for video games and that will keep you united… that is until layoffs.

The really crappy part of the job is that it’s a contract position unless you have a brown enough nose to stay on permanently.  Typically your contract is good for 6-12 months.  It doesn’t matter how good you are or how many crash bugs you found, but when you reach your 12 month anniversary, the company by law (maybe just California) has to provide you with benefits.  That just does not happen.  Wave goodbye to your colleagues as security escorts you down the walk of shame.  It is a harsh truth, but there is a glimmer of hope called Funemployement (a future story!)

ACT II: DEVELOPER SIDE (IE:)  EMBEDDED TESTER
A few months into my time testing on Medal of Honor: Airborne, I was promoted to an embedded tester.  They sat me right smack between the Art and AI team.  Being with the devs was like night and day.  For one, you had more physical space to move around and didn’t have people hovering over you.  My bug count did go down a little as I was doing more specific tasks and numerous smoke tests.  Having to come into work an hour earlier kind of sucked, but it was needed to complete the morning smoke for your fellow publisher side testers to have a stable build to work on.

There were some small perks to enjoy such as your security badge upgraded to allow access to the elevator and being able to shoot the shit with the devs during “Beer Fridays.”  You could also sneak in after late “Margarita Wednesday” lunches not worrying about your bosses catching a whiff of you celebrating Cinco de Mayo.

The best part of being an embedded tester was that you were able to experience the entire wheel turning.  You felt more like part of the team as you were involved in daily Scrum meetings, being in the middle of successfully hitting milestones, being seriously heard for suggestions and concerns… that experience alone really opened my eyes to seriously pursuing video games as a career.

Once I was eventually let go from that prestigious tester position, I had interest from Treyarch and Insomniac Games for work.  I chose Treyarch even though Insomniac was closer to my home. I figured they had more upside seeing that I would work on more than just a PS3 and PC SKU.  Plus Treyarch was in Santa Monica and Insomniac was in hot ass Burbank!

Treyarch was very similar to when I was an embedded tester at EA.  My work station was smaller and test lead was an elitist tomboy, but she did fight for us and never fed us to the sharks. If you are curious, Treyarch’s perks were pretty nice.  Fully stocked kitchen including unlimited fountain drinks, beef jerky, trail mix, candy bars, cereal, chips… oh man I’m making myself hungry! Oh and I can’t forget the free catered In-N-Out Truck and Pink’s Hotdogs once a month and mandatory “Beer Friday Team Meetings”.  The “Breakfast for Dinner” OT dinners were also the rare times when I ran to be in the front of the line!  Come late and no bacon for jou!

Food aside, Treyarch is where I was able to excel and lead the team in bug count for a majority of the time.  The dev team listened to my suggestions and implemented many of them including numerous changes to the UI/Shell.  Since I was a graphic designer by trade in my previous endeavors and currently, I was pursuing UI art as a viable career destination.  I even bought a book on UI I glanced over a few times!  I first caught the eye of our executive producer when I created a custom badge sticker for my ID card that had a James Bond silhouette and my name and title along with Treyarch and Activision logos.  He gave me 20 bucks for materials to make him some to pass out to some of the higher ups.  He sported his and always flashed his to me when we walked past each other in a hallway.

My big break came when a temp UI artist position became available where I applied faster than a BMW M8’s 0-60, but that is another story (keep your eyes peeled for that upcoming blog!). Eventually my time at Treyarch came to an end as well, but my career was not over!

testing_with_a_view

I landed back on my feet in the trenches of QA developer side with Pandemic Studios.  This office was plush. For one, there was a bar in the lobby, but even nicer was that you can work with a view as opposed to being shackled in “the Dungeon” of Activision HQ.  I never had the pleasure to work in that cutthroat environment, but was thrilled this location was in beautiful Westwood just a few blocks from UCLA. I was placed on the Saboteur team and was graciously welcomed by an elite squad of testers in our own spacious room to comfortably shoot black and white Nazis and deeply analyze 1940’s digital boobs.

Pandemic felt much like Treyarch, but there was an invisible line between QA and the dev team.  Many of my old colleagues I worked with at EALA were transferred to Pandemic when EA purchased Bioware and Pandemic Studios from U2 Bono’s holding company for $860 million. With that move, they brought the whole “publisher side” mentality to a dev side studio which made for some frustrating moments. I had to get all of my bugs “evaluated” again like I was some n00b. Despite feeling like I was publisher side again, I made sure to keep my voice heard and pushed my way forward, making strides in suggestions for the UI while helping document as many bugs as possible.

One thing to note that was interesting we did was that we had “the Pit.” This is where we would go to play test and reproduce nasty bugs with devs and hear quality jokes from QA Analyst/Stand-Up Comedian Marty Quinn. You should check him out. The Pit was where everything went down and was also in the open room where we held team meetings even though you can’t efficiently fit 120 people in that area.

pandemic_shutdown_QA

Fast forward through crunch time and a week til launch, EA dropped the bomb and decided to shutdown Pandemic Studios forever.  It was interesting to read about a “rumor” on Kotaku about Pandemic being shut down an hour before it happened.  It was a mix of emotions and actions.  Confusion, yelling, crying, nervous laughter, pillaging, denial, the whole bag was present.  We were first split into 2 groups.  One group was contract employees and the other perms.  The perms got presented on their severance packages and benefits while the employees on Visas were up a creek royally. I was contract so I was told that I am being paid till the end of the week and to drop off your badge with security. The next 5 hours were spent getting drunk at the local pub where our ex bosses paid the tab.  Funny because we were in the middle of fixing the PC only bug that made the game super unstable for users playing the game on multi-core processors and Radeon GPUs.  Stay classy EA.

EPILOGUE
Well, that is embedded vs publisher testing as seen through my eyes.  There are other differences I didn’t go over such as the types of bugs you look for as the publisher has their own criteria vs. developers.  That can include Sony’s TRC and Microsoft’s TCR compliance testing and black vs. white box testing.  Generally publisher side tests “Black box” (external testing perspective such as the end user) and embedded works “White box” (internal testing perspective more at a code level), but they both overlap each other somewhat.  Another one is that the publisher side will test off of discs of the build that the embedded testers “approved” via smoke tests.  Their builds will be a day or two behind embedded who don’t generally work off of discs too often.  They sync the most recent game files and compile a build on their comp/console to create a working build to test off of.

I hope you enjoyed this read as I was happy to recall my fond memories of QA.  Look for my next recollection in a blog post near you!

Comments

comments

by Steve Cantu, on March 9, 2015

I work at iBUYPOWER