NOTICE: Account creation on the wiki has been temporarily disabled until the wiki is moved to OpenID. If you need an account before then, please request one here: http://forum.xbmc.org/showthread.php?tid=165868
HOW-TO:Help with quality assurance testing
|This page or section may require cleanup, updating, spellchecking, reformatting and/or updated images. Please improve this page if you can. The discussion page may contain suggestions.|
XBMC Media Center is a FOSS (Free and Open Source Software) project, driven by the community of users, and you and anyone else who uses XBMC are more than welcome to help us make it better! There are many different ways that you and your friends can help by contributing to the XBMC project and Quality Assurance (QA) is one of them.
QA is regarded as an essential part of making XBMC into a better software and people helping with QA is in a sense just as important as developers to ensuring the continued success of the project. Taking a QA role is to help take ownership and responsibility of the bug reporting and tracking process, ensure that projects developers have as much time available for fixing things, and to ensure that users get prompt replies to their queries, and that things are fixed in a timely manner.
1 What a Quality Assurance Specialist expected to do
Those people that show an enthusiastic contribution to the project and the community on the forums might be a good candidate for the official XBMC QA Team (also known as the "Bug Hunter Squad") that is a part of Team-XBMC. Invitations are sent out to community members who stand out and show that they could be willing to take on the role as an official QA Specialist on Team-XBMC.
1.1 Specifically, the QA team is expected to
- Step in as soon as a problem is reported on the forums or on our trac tracking-system, and attempt to help the user fix the problem, or confirm whether the problem is a bug by reproducing it. Once confirmed, your role is to ensure it gets properly reported in trac, either by the original user or by doing it yourself.
- Ensure all relevant information has been reported on trac tickets, and that it is reproducible. You should close any reports in which the user doesn't respond with needed information in a timely manner. You should then ensure the bug gets assigned to the appropriate developer, and track its progress so that it doesn't slip through the cracks. The report is considered in your hands until it has been confirmed as fixed by either yourself or the original user. When the developer needs more information, the QA team makes sure they get it.
- Perform regression testing. You should have a media library of various formats to test against XBMC, (a media library which of course could be a shared team resource), that can be used to regression test.
- Testing new and changed features or functions in XBMC, and the main application for consistency. You should be willing to test things that you might not normally use (as well as things you will of course be using all the time) as they are developed to ensure that XBMC stays at the leading edge in terms of both features and stability.
If you where to get invited to become an official QA Specialist on Team-XBMC then the expectation is that it will require an hour or so a day in checking up on the forums and trac, (so approximate 7 hours a week), possibly with additional time taken to verify bugs, and identify ways to reproduce them. It will be important to also either know or be willing to learn how to compile your own builds of XBMC from SVN, for one or more platforms.
2 Eight Quick Steps to Triage Nirvana
The main job of Quality Assurance is to triage bugs to our trac tracking-system.
2.1 Triaging a bug involves
- Making sure the bug has not already been reported before
- Making sure the bug has enough information for the developers and makes sense
- Making sure the bug is filed in the correct place
- Making sure the bug has sensible "Severity" and "Priority" fields
- Making sure the bug is versioned correctly
2.1.1 First steps for newbies
- Register an account on XBMC Community Forum.
- Read the Stock Responses further down this article. These stock responses are general examples of the kinds of comments that triager's should add to bug reports. There are even a surprising number of bug reports for which you can simply use these responses.
- Pick a bug to triage. Any bug report will do. There are also some tips on finding bugs to triage. Once you have picked a bug report, read it carefully.
- Find what should be changed. Read the Steps of Triaging further down this article to determine what should be changed (if anything) and how.
- Ask someone to verify your changes.
- Make the changes. After someone verifies your changes (and likely refines them and/or points out other changes), either make those changes yourself or make sure that the reviewer makes them.
- Repeat steps 2-6.
3 Steps of Triaging
This guide should make triaging a bug as easy and simple as possible. Remember, if in doubt, ask! Also, remember that you may not be able to find anything to change for some bugs since not all bugs are filed incorrectly or incompletely.
3.1 Is it part of XBMC Media Center software from Team-XBMC?
Some pieces of add-ons (like third-party skins, plugins, scripts, and more) that users think of as part of the XBMC Media Center software from Team-XBMC have their bug tracking systems elsewhere. If the bug relates to one of these components, please close the bug with the resolution as "Not part of XBMC" and ask the reporter to report it to the correct tracking-system. Note that there is a stock response for this situation.
3.2 Does it make sense?
Does it have enough information? Common sense rules here - "this sucks" or "this crashed" should be be asked for more information polite message as change the resolution to "Need more information", as should other things that are too ambiguous to be useful. It is not appropriate for bug volunteers to mark a ticket as "'Will never be fixed/added", that is up to the maintainers. The exception is crasher bugs that have a XBMC debug log (and stack trace if Linux). See below for details; in general you should not change a bug to "Need more information" if it's got a good XBMC debug log (and stack trace if Linux). If you mark a bug "Need more information", "Invalid" or "Incomplete", it is useful to add yourself to the CC field; if the reporter adds more information, you can then reopen the bug and re-triage it. If a bug report has been set to "Need more information" state and if there has been no response for 4 weeks, feel free to close the report as "Incomplete" by using the "incomplete" stock response.
3.3 Is the bug report a duplicate?
Many bugs ported onto our tracking-system have been reported before, and finding them is more of an art than a science. If the bug is a duplicate, change it as a duplicate in the resolution box and use the "Duplicate" stock response for this situation. If the original report (i.e. not the one that you currently triage) is in "Need more information" state, please let the reporter know by asking him/her to answer the questions in the original, "Need more information" bug report. You don't need to carry on triaging it if it's a duplicate; press the Submit Changes button and go onto your next bug!
3.4 What platform and version of XBMC?
If the bug is in a version of XBMC prior to 8.10, ask the reporter to upgrade and close it as "Obsolete". There is a stock response for this situation. Otherwise, select the appropriate version that the bug appears in for the XBMC Version field.
- There are two exceptions. If fixing the bug requires breaking string or UI freeze, you can mark the bug with the version beyond the one that will be released next. If the bug is a feature request the you can change it to say so.
3.5 What severity/priority should be set?
Finding the correct severity and priority takes experience, and you will get a feel for it after triaging several bugs. Here are some guidelines for the most commonly reported bugs. You should read the severity and priority guidelines at least once.
Please note that only changing severity and/or priority slightly may do more fuss than good. It causes an extra email to the maintainers and consumes their precious time and attention. If other changes are done as well then there is only little additional impact when changing severity and/or priority.
- Crasher bugs
- Should have Severity=Critical, Priority=High
- Mark them "Need more information" if they don't have a XBMC debug log (or stack trace if Linux), and ask the reporter to get a XBMC debug log (and stack trace if Linux). Otherwise, do not mark them "Need more information" since that is the first step to them being closed as "Incomplete" and the stack trace often has enough information for a developer to fix the bug. However, if there is no information about how it is triggered you should ask the original reporter how to reproduce the bug.
- Should have Severity=Critical, Priority=High
- Something not working as expected
- Severity=Minor/Normal/Major depending on how much it breaks the application (see the guidelines). If XBMC would suck badly if we shipped another version with this bug, set Priority=High; otherwise Priority=Normal.
- Usability issues
- Severity=Minor, Priority=Normal.
- Add the "usability" keyword.
- Accessibility issues
- Severity=Minor or Normal depending on the extent of the bug. Priority=Normal.
- Add the accessibility keyword, and possibly the keynav keyword if it's a keyboard navigation bug.
- These really require expert assessment to determine their severity, so the accessibility keyword is the most important part of our triage.
- Incorrect strings, typos, etc.
- Severity=Trivial, Priority=Normal or High depending on how user-visible it is.
- Set the string keyword.
- Feature requests
- If it really is a new feature, and not an improvement to something already in the product, set Severity=Enhancement, Priority=Normal.
If you're unsure about your Severity/Priority triage, add a comment in the bug explaining why you triaged it as you did. You can also add yourself to the Cc field, so if someone re-triages it differently you can find out why.
3.6 Add yourself to the CC list
By adding yourself to the CC list of bugs you change, you will receive e-mails with comments and changes others do to the individual report. You learn what further investigations others make and learn more rapidly about triaging bugs in the XBMC tracking-system.
4 Stock responses for incomplete bugs
Here are some stock responses to cut and paste into XBMC tracking-system for common situations.
- No XBMC debug log is provided
- Please collect and attach a XBMC debug log that shows the problem, see http://wiki.xbmc.org/?title=HOW-TO_Submit_a_Proper_Bug_Report
- If the bug is a crasher and no stack trace is provided
- Thanks for taking the time to report this bug. Without a stack trace (also known as a backtrace) from the crash it's very hard to determine what caused it. Thanks in advance!
- If the bug is not described well
- Thanks for taking the time to report this bug. This bug report isn't very useful because it doesn't describe the bug well. If you have time and can still reproduce the bug, please read http://wiki.xbmc.org/index.php?title=HOW-TO_Submit_a_Proper_Bug_Report and add a more useful description to this bug report.
- If the bug is not described well and it's a crasher
- Thanks for taking the time to report this bug. This bug report isn't very useful because it doesn't describe the bug well. If you have time and can still reproduce the bug, please read http://wiki.xbmc.org/index.php?title=HOW-TO_Submit_a_Proper_Bug_Report and add a more useful description to this bug report. Also note that without a stack trace (also known as a backtrace) from the crash it's very hard to determine what caused it. Thanks in advance!
- If the stack trace is incomplete or without debugging symbols
- Thanks for taking the time to report this bug. Unfortunately, that stack trace is missing some elements that will help a lot to solve the problem, so it will be hard for the developers to fix that crash. Can you get us a stack trace with debugging symbols? Thanks in advance!
- If the bug is in an obsolete version of the application
- Thanks for taking the time to report this bug. However, you are using a version that is too old and not supported anymore. XBMC developers are no longer working on that version, so unfortunately there will not be any bug fixes for the version that you use. By upgrading to a newer version of XBMC you could receive bug fixes and new functionality. You may need to upgrade your Linux distribution to obtain a newer version of XBMC. Please feel free to reopen this bug if the problem still occurs with a newer version of XBMC.
- If the bug is not with the XBMC software
- Thanks for taking the time to report this bug. However, this third-party skin/script/plugin/software does not track its bugs in the XBMC tracker-system. We kindly ask you to report the bug to the third-party skin/script/plugin/software authors. If the affected third-party skin/script/plugin/softwarehas a bug tracking-system you should investigate whether a bug for the reported issue is already filed in this system. If it has not been filed yet please do so. Also ensure that both bug reports contain a link to each other. Thanks in advance!
- If the bug is a duplicate
- Thanks for the bug report. This particular bug has already been reported into our bug tracking system, but please feel free to report any further bugs you find. or Thanks for taking the time to report this bug. This particular bug has already been reported into our bug tracking system, but the maintainers need more information to fix the bug. Could you please answer the questions in the other report in order to help the developers?
5 Triage FAQ
- What are Team-XBMC Quality Assurance Specialists?
- The QA Specialist team is the Quality Assurance (QA) team for XBMC. We keep track of current bugs in XBMC software and try to make sure that major bugs do not go unnoticed by developers. It is not our job to fix the bugs, although there is nothing to stop you from submitting patches if you have hacking experience. It is a great way to return something to the XBMC community whether you can program or not.
- What skills are needed in order to help triage bugs?
- Common sense and a bit of pragmatism is all that is needed to join the QA Specialist team. Coding skills might help you evaluate or fix particular bugs, but triaging most bugs will not require knowledge of programming. Bug triaging is a great way for people who do not know anything about programming to return something to the XBMC community.
- What software is needed in order to assist in triaging bugs?
- All you need is a web browser and an internet connection. Some people have triaged bugs using Firefox or Internet Explorer on a Windows desktop at work. It is usually best, though, to be running a reasonably recent version of XBMC yourself as well.
- How long does it take to learn how to triage bugs?
- Triaging bugs is a great way to help out with XBMC if you do not have much time. It does not take very long to get started, and on average, the simpler bugs can be triaged in anywhere from 2-10 minutes.
- Why should I join/help the QA Specialist team?
- There are several reasons:
- Maintainers will not have to spend as much time reading bug reports and will thus be able to concentrate on fixing bugs, making XBMC better for you.
- You will learn more about the capabilities of XBMC (e.g. when you look at a bug that someone files about a specific feature in the Music or Video Library Mode which you did not know about...).
- You will get a better feel for the different areas of XBMC and will potentially be able to find other areas that you can also help (e.g. documentation, graphics, translation, web design, user/human interface work, assisting with developing certain applications, etc.).
- There are several reasons:
- What if I can not find any changes to make to a bug? What if all the fields look like they are set correctly?
- Then simply move on to a different bug.
- What if a certain bugs look like a duplicate of many different bugs?
- Try to find the "original" one and mark your bug as a duplicate of that one. (i.e. if bug A has been reported and bugs B & C have been marked as a duplicate of A, then try to mark future bugs as a duplicate of A and not as duplicates of B or C).
- I can not reproduce a certain bug. Can I close it?
- You have to take into account why you can't reproduce it. Many bugs happen only under certain rare circumstances, and you might not be able to reproduce it due to a different system configuration, or you might be running Linux instead of Windows or Mac.
- If the bug is sufficiently general, is filed against an old version and looks as though it is now fixed, you can close it. If you're not sure, ask in forum and IRC channels if someone else can replicate it. You can also add a comment saying that you can not reproduce the bug (along with any relevant information about your system) if you think it will help.
- I am a programmer. Are there more advanced things I can do related to triaging?
- Yes, there are several things you could do that someone without programming skills might not be as adept at or would not be able to do at all. Bug triaging is a great way to find a small part of XBMC to start on while making your first XBMC baby steps. Here are a few of the things you can do:
- Obtain easy directions to duplicate bugs (sometimes it is easier for those with a programming background to find these)
- For bugs that are reported but do not contain debugging symbols, try to obtain debugging symbols for them. Some stack traces can help a developer find a problem even without debugging symbols, but often times they are not very helpful.
- Find and mark duplicates of crasher bugs (those without a programming background may take longer to get used to dealing with stack traces).
- Look for bugs with the "love" keyword. Fix them.
- Look for bugs which look interesting. Contact the maintainer to let him/her know you would like to work on it and verify that no one else is working on it. Then fix it.
6 Encouragement from Team-XBMC
Remember that this is only the first step. At least a couple other people will read and have the opportunity to correct your mistakes. That isn't to say "make lots of mistakes" but it is to say "don't be afraid to make them", especially once you get used to it, your work will make the lives of the XBMC developers- and as a result XBMC itself- a heck of a lot better.