Testing Computer Games

Gamers can also use this forum to chat about any game related subject, news, rumours etc.

Moderator: maddog986

Post Reply
User avatar
Greyshaft
Posts: 1979
Joined: Mon Oct 27, 2003 1:59 am
Location: Sydney, Australia

Testing Computer Games

Post by Greyshaft »

I was going to post this in the Beta Tester Forum for MWiF then I thought the rest of the MWiF readers might want to read it to prepare themselves for when the MWiF Testing Machine drags them in, and then I finally thought it might have a universal audience for anyone who is, or wants to be, involved in testing computer games.
***********************************



Here is a summary of an article from the February 2006 edition of CIO Magazine. It lists 25 steps to better software testing. Not all of it is applicable to what we are doing here, but there's still plent of food for thought...

1. Respect your Testers - Testing is not an entry level job. If it is treated as such then the quality of Testing will suffer but if it is treated as a professional skill then testers will tend to rise to the challenge.

2. Co-locate your developers and Testers - Well, THAT'S not going to happen!

3. Set up an Independant Reporting Structure - Neither will this... everything goes straight back to the Developer.

4. Dedicate Testers to specific Systems - That's already planned for the MWif beta-test. Some MWIF testers will become experts on the Production system and some will become experts on naval movement and interception etc etc. That doesn't mean that you ignore the other subsystems, but it makes life easier for Tester and Developer when they are tracking down a bug if they both understand how a subsystem is supposed to work. Hopefully testers will spread themselves across the subsystems in an even pattern but I may call on people to pick up additional areas (Don't send me your preferences yet!)

5. Give them Business (or in this case... MWiF) Training - We have already set up our Tutorial Teams.

6. Allow Business Users to Test - Periodic new blood in the Test team should address this issue and give us an understanding of what MWiF looks like to a new purchaser.

7. Involve network operation - We will organise a few sessions to test multi-player operation.

8. Build a Lab - Not really applicable because every user's configuration is different.

9. Develop Tests during the Requirements Phase. - That's happening. The Specification / Development / Testing are all happening simultaneously and will provide continual feedback for the other tasks.

10. Test the Old with the New - Standard Regression Testing. Even if a module (say, Production) has been completed and successfully tested, it must still be tested again when another module (say, Surrender) is deployed. That's in the plan.

11. Apply Equivilence Class Partitioning - I would if I had the tools or knew the math. Its a method of determining the number of test cases you need to run in order to complete the functional testing of a system.

12. Involve Testers early in the Development Cycle - Just about now seems right.

13. Establish Quality Checkpoints - More applicable to larger groups. Since Steve is the source of all decision making then we need to rely on him to make his own sanity checks.

14. Write a Tech Guide - read "Tutorial'. Testers must know what results to expect before they can meaningfully test any software. uh... my unit just turned purple... is that supposed to happen?'

15. Centralise your Test Groups - Ensure that Testers are communicating. More of a problem with large organisations who don't have a cohesive test team.

16. Raise Tester's awareness of their value - Ever played a game with a bug in it? How did you, as a purchaser, feel about the development/test team? If you were a reviewer, what would you have written about the Game Publisher? None of us can single-handedly destroy Matrix's reputation, but if we collectively fail to do our job as a Test Team then MWIF and Matrix and the chances of seeing more strategic level games in the future will suffer.

17. Don't forget about negative testing - Not all bugs generate a Blue Screen of Death. Sometimes its the things that don't happen that are just as important. Hey! I produced a sub four turns ago! What happened to it?

18. Tell Programmers to chill out
- Programmers? Sensitive about their code? Never!

19. Cross train developers and testers in each others roles - Steve has gone above and beyond in the public Forums with his efforts to explain how MWIF will work. As testers we need to be reading what he writes. If an article is too techie for you (as some of them are for me) then ask your Tutorial Team for an explanation.

20. Test in a locked down environment - I'm pretty sure Steve can't log into our computers.

21. Advise Testers of upcoming code changes - Steve will notify us of what to test for in each code release.

22. Code Coverage - That's my job. Ensure that testing covers all aspects of the code.

23. Scan your source code for known problems - That's Steve's job. We will report the bugs to him eg. Hey Steve! MWIF gets reeeeeeeal slow after about 1943 ... and he will scan for the memory leaks etc.

24 Identify patterns - Difficult to do in a single purpose test team with just one piece of software to test.

25.Develop a Plan 'B' - Well if Steve walks under a bus then I plan to sign up to test the Matrix remake of 'Battle of Britain' and 'Bombing the Reich'...



/Greyshaft
User avatar
sterckxe
Posts: 1897
Joined: Tue Mar 30, 2004 8:09 am
Location: Flanders
Contact:

RE: Testing Computer Games

Post by sterckxe »

ORIGINAL: Greyshaft

I was going to post this in the Beta Tester Forum for MWiF then I thought the rest of the MWiF readers might want to read it to prepare themselves for when the MWiF Testing Machine drags them in, and then I finally thought it might have a universal audience for anyone who is, or wants to be, involved in testing computer games.

Thanks for posting this - I posted the url into another beta-testing forum because it's so very appropriate to all of us [;)]

Greetz,

Eddy Sterckx
User avatar
Widell
Posts: 890
Joined: Wed Apr 27, 2005 2:25 pm
Location: Trollhättan, Sweden

RE: Testing Computer Games

Post by Widell »

ORIGINAL: Greyshaft
3. Set up an Independant Reporting Structure - Neither will this... everything goes straight back to the Developer.

8. Build a Lab - Not really applicable because every user's configuration is different.

All are good comments, and it would be interesting to see the "version" of the article if it had been written from the perspective of the testing that goes on here at Matrix, or even more so at a place like Sourceforge. Not sure if the conclusions would have been the same ;-)

I also had some specific comments to the two issues I quoted:

- Independent Reporting not happening: Why not? It shouldn´t be that hard to set up an environment for bug and issues reporting. I have used Mantis for one project and the stuff included in the Sourceforge package for another. Understandably, Matrix doesn´t do Open Source stuff, but something like Mantis should be simple to set up for the beta testers of any of Matrix's projects

- Build a lab: Maybe we should see this as building a virtual lab? Most of the testing I have been involved in here (COTA mainly, and I agree, I haven´t been contributing much, mostly because the other testers are sooooo much faster than I am.....) goes on in a very open and "peer-like" environment. Assignements and files are handed out and tested by the team on different machines and config's. In essence a lab environment, although not very formalised

Well, that was my 2 cents
User avatar
Monadman
Posts: 548
Joined: Tue Dec 06, 2005 3:33 pm
Location: New Hampshire

RE: Testing Computer Games

Post by Monadman »

ORIGINAL: Greyshaft

16. Raise Tester's awareness of their value - Ever played a game with a bug in it? How did you, as a purchaser, feel about the development/test team? If you were a reviewer, what would you have written about the Game Publisher? None of us can single-handedly destroy Matrix's reputation, but if we collectively fail to do our job as a Test Team then MWIF and Matrix and the chances of seeing more strategic level games in the future will suffer.

And there you have it. Well said.
User avatar
Erik Rutins
Posts: 39324
Joined: Tue Mar 28, 2000 4:00 pm
Location: Vermont, USA
Contact:

RE: Testing Computer Games

Post by Erik Rutins »

ORIGINAL: Widell
- Independent Reporting not happening: Why not? It shouldn´t be that hard to set up an environment for bug and issues reporting. I have used Mantis for one project and the stuff included in the Sourceforge package for another. Understandably, Matrix doesn´t do Open Source stuff, but something like Mantis should be simple to set up for the beta testers of any of Matrix's projects

In fact we are already in the process of setting up Mantis, specifically, for the MWIF testing, but Greyshaft may not have realized that was coming. [8D]

A virtual lab of sorts does exist on every test team and we also have a number of different system configurations here at Matrix that we test on before a title goes into production, but it's not the same as a full lab.

Regards,

- Erik
Erik Rutins
CEO, Matrix Games LLC


Image

For official support, please use our Help Desk: http://www.matrixgames.com/helpdesk/

Freedom is not Free.
User avatar
Hard Sarge
Posts: 22145
Joined: Sun Oct 01, 2000 8:00 am
Location: garfield hts ohio usa
Contact:

RE: Testing Computer Games

Post by Hard Sarge »

I have used Mantis before

it works well :)

one thing, and I have tried to tell people, if you see something in a game that looks like a bug, or the way the game/program does something, the normal response is, dang testers, don't alway "blame" the testers, there job is to find and report, and maybe explain why it is wrong

one beta I was in (for another company) almost every tester complained about how something was going to be modeled, it was just wrong, the designer said, no, he likes it that way, it will bring in more people to play the game, then when it was released, all we heard was, how did the testers miss this, the designer later ended up changeing it, due to players complaining

(bad part, the number 2 man at the company, stood up for the testers and agreed with us, he was fired when the game was released, for disagreeing with the number one man)

Image
User avatar
Greyshaft
Posts: 1979
Joined: Mon Oct 27, 2003 1:59 am
Location: Sydney, Australia

RE: Testing Computer Games

Post by Greyshaft »

ORIGINAL: Erik Rutins

In fact we are already in the process of setting up Mantis, specifically, for the MWIF testing, but Greyshaft may not have realized that was coming. [8D]

Steve had mentioned that possibility but I wasn't holding my breath [:D]
/Greyshaft
Post Reply

Return to “General Discussion”