In the world of software development, applications pass through many phases before being released, some of which include designing, planning, coding, testing, Alpha Release, Beta Release, Release Candidate, and finally Release. This guide is intended to help the people involved in public/private beta tests that do not have experience programming, but would like to give valuable feedback to the developer.
Disclaimer
This guide is specific to programming for the Windows Mobile Operating System, specifically for the .net Compact Framework 3.5. However, the main concepts should translate over to other platforms and frameworks.
In the beginning...
First off, programming is hard. Building applications that run smoothly and are bug free require considerable amounts of time and energy. In short, software development is an emotional investment on behalf of the developer.
Because software development is an emotional investment, developers will easily take offense to criticism. This is where rule number one of giving constructive feedback comes in:
Rule number #1, When Giving feedback, be polite!
During my time on XDA-Developers I've been very happy with level of professionalism demonstrated by its members, but every once in a while I'll see a comment like, "This app sux!" or "Eff Dat shiz!".
Always remember, rude comments only distract other users, cause contention, and cause the developer to become less motivated (less emotionally involved) to produce software. If the application is not working on your device, remember the developer is the first person who will be able and willing to help you resolve your problem, so they should be the last person you want to offend. [1]
This is where the term Troll comes from. A troll is not the ugly monster that lives under the bridge, but is someone who is like the fisherman that "trolls" the river for fish (or heated response in the case of internet trolls)
Rule number #2, be constructive!
What's interesting about this rule is that the phrase "be constructive" in itself is not constructive. So how do you "be constructive"? Labeling a comment as constructive is not deterministic, or in other words there is no formula to test whether or not your comment is constructive. The rules in this guide will attempt to give you some pointers to giving constructive comments, but in the end, only a mix of common sense and human reasoning can determine if a comment is constructive.
Some examples of non-constructive comments
Let's be honest. Programmers aren't perfect. In a list of common excuses given by programmers in response to different bugs, the number one response was "Well, it works on my machine!". If you are a developer you know this is true! I catch myself using this phrase all the time! But, what can we learn from this? Well, first, the crash/bug you are having might not have ever been seen by the developer (if it had, chances are they would have fixed it!). Because of this, you must include as much relevant information about the bug as possible.
What is considered relevant? That is the million dollar question. I will attempt to give some guide-lines on how to determine what is and what isn't relevant.
Rule of thumb: When in doubt if info is relevant, include it.
It is hard to include too much information (hard, not impossible), but it is very easy to provide insufficient information to the developer.
What did you do?! Is it reproducible? What did you expect to see?
The crash isn't your fault. But if the developer doesn't have the steps to take to reproduce your error, they can't find the bug!
When you get a crash, first thing you need to do is try and remember as many steps as you can to reproduce the bug. Then, restart the app and follow the same steps to try and get the crash screen again.
You need to tell the developer if the crash is Always reproducible or only sometimes reproducible. Then you need to create a list of steps to get the crash. Try and always use an ordered list when doing this. It makes it easier to read and follow.
Finally, tell the dev what you were expecting to see and what you actually saw. Some bugs don't crash but instead produce results that aren't exactly what was expected. For example, "I clicked on the 'Read' button, it won't let me read the page!" Well, the bug in this case could be that the word "Read" is ambiguous and really means "have Read" as in "I have read the page already, so mark the page as 'Read'". This is a "What I expected and what I got" error. All the info that is required is the steps to reproduce, what you were expecting and what you got.
What is your device/ROM/Windows version?
Sometimes this is necessary, so if your device is not in your signature, include it! If it is in your signature, include it anyways!
The Crash Screen
If you are lucky enough to get a crash screen, most likely you will be able to supply enough information from the error message to the developer for them to figure out what went wrong.
If you receive the following message (or a translation there of), you must take an additional step in order to get the real crash message:
This error message means you don't have the right software installed to display the crash message. To fix this, load the appropriate cab onto your device. Message cabs are located at: C:\Program Files (x86)\Microsoft.NET\SDK\CompactFramework\v3.5\WindowsCE\Diagnostics
If that directory does not exist, install the Power Toys for .NET Compact Framework 3.5 and check again.
Anatomy of the Crash screen.
Here is a crash details report from my XDAFacebook application. I will describe each part and what they mean, what's important and what the programmer is looking for:
The first line: XDAFacebook.exe is the application that crashed. This is important, because if the application relies on multiple .exe files, the one that crashed needs to be known. Always include which .exe crashes.
The next line is the type of exception that was unhandled: ObjectDisposedException. This is also very important. Always include this in your report. If anything, this is the most important piece of information.
The next part is called the Stack Trace. In 2 sentences, the stack trace is an ordered list of methods (or functions) that pin-point where the in the code the crash happened. Think of it as turn-by-turn directions to the line that caused the crash.
The stack trace lines have 2 parts, the NameSpace and the Method. The NameSpace is the series of "Strange word followed by a Period". The method is the "Strange word followed by parenthesis that might have strange words inside". In XFControls.XFPanels.XFPanelList.PanelPaint(Graphics g), the NameSpace is XFControls.XFPanels.XFPanelList and the Method is PanelPaint(Graphics g)
Some of the lines aren't important. Most of the lines are. The most important lines are usually at the top and will have NameSpaces that looks like the application that crashed (or don't look like they come from Microsoft). The best thing to do is include all of the stack trace, but it is also good to know what exactly you are telling the programmer.
Here is basically how I would read the stack trace above:
If any of those lines weren't in the report, I wouldn't be able to get the full picture. So, unless you have a good understanding of the application's source, include the full report!
Developer Defined Messages
Developers try to handle as many errors as possible. In the case that you get a Message from the application signifying some kind of error has occurred, supply the message the occurred, the steps you took to get the error and your "What I expected and what I got". If the message isn't enough information, the developer needs to change his message.
Also, check for any debugging options which the developer may have included, such as a log file. These either tend to be automatically generated files in the install directory, or they exist as a viewable panel within the application. Check with the documentation or ask the developer where these log files exist [1]
Rule number #4, be smart.
I should put this higher in the list, because it really should be the first thing you need to do.
First thing you need to do is make sure you have the latest version of the software installed. Also do a quick search of the thread to make sure that the bug hasn't already been reported. If it has been reported, I personally don't mind a quick "I have the same problem, here are the steps I had to reproduce..." with a link or quote of the original error.
This guide will be updated as I come up with new observations and as people respond with questions/suggestions. Please tell me what you think of the guide and always try and be constructive when you are beta testing software!
Thank you
**updates**
[1] 12/7/2010 (Thanks meltwater!)
Disclaimer
This guide is specific to programming for the Windows Mobile Operating System, specifically for the .net Compact Framework 3.5. However, the main concepts should translate over to other platforms and frameworks.
In the beginning...
First off, programming is hard. Building applications that run smoothly and are bug free require considerable amounts of time and energy. In short, software development is an emotional investment on behalf of the developer.
Because software development is an emotional investment, developers will easily take offense to criticism. This is where rule number one of giving constructive feedback comes in:
Rule number #1, When Giving feedback, be polite!
During my time on XDA-Developers I've been very happy with level of professionalism demonstrated by its members, but every once in a while I'll see a comment like, "This app sux!" or "Eff Dat shiz!".
Always remember, rude comments only distract other users, cause contention, and cause the developer to become less motivated (less emotionally involved) to produce software. If the application is not working on your device, remember the developer is the first person who will be able and willing to help you resolve your problem, so they should be the last person you want to offend. [1]
This is where the term Troll comes from. A troll is not the ugly monster that lives under the bridge, but is someone who is like the fisherman that "trolls" the river for fish (or heated response in the case of internet trolls)
Rule number #2, be constructive!
What's interesting about this rule is that the phrase "be constructive" in itself is not constructive. So how do you "be constructive"? Labeling a comment as constructive is not deterministic, or in other words there is no formula to test whether or not your comment is constructive. The rules in this guide will attempt to give you some pointers to giving constructive comments, but in the end, only a mix of common sense and human reasoning can determine if a comment is constructive.
Some examples of non-constructive comments
- The app is broken
- "It" crashes all the time!
- I get an error message, plz help!
- When will the app be finished?
- etc
Let's be honest. Programmers aren't perfect. In a list of common excuses given by programmers in response to different bugs, the number one response was "Well, it works on my machine!". If you are a developer you know this is true! I catch myself using this phrase all the time! But, what can we learn from this? Well, first, the crash/bug you are having might not have ever been seen by the developer (if it had, chances are they would have fixed it!). Because of this, you must include as much relevant information about the bug as possible.
What is considered relevant? That is the million dollar question. I will attempt to give some guide-lines on how to determine what is and what isn't relevant.
Rule of thumb: When in doubt if info is relevant, include it.
It is hard to include too much information (hard, not impossible), but it is very easy to provide insufficient information to the developer.
What did you do?! Is it reproducible? What did you expect to see?
The crash isn't your fault. But if the developer doesn't have the steps to take to reproduce your error, they can't find the bug!
When you get a crash, first thing you need to do is try and remember as many steps as you can to reproduce the bug. Then, restart the app and follow the same steps to try and get the crash screen again.
You need to tell the developer if the crash is Always reproducible or only sometimes reproducible. Then you need to create a list of steps to get the crash. Try and always use an ordered list when doing this. It makes it easier to read and follow.
Finally, tell the dev what you were expecting to see and what you actually saw. Some bugs don't crash but instead produce results that aren't exactly what was expected. For example, "I clicked on the 'Read' button, it won't let me read the page!" Well, the bug in this case could be that the word "Read" is ambiguous and really means "have Read" as in "I have read the page already, so mark the page as 'Read'". This is a "What I expected and what I got" error. All the info that is required is the steps to reproduce, what you were expecting and what you got.
What is your device/ROM/Windows version?
Sometimes this is necessary, so if your device is not in your signature, include it! If it is in your signature, include it anyways!
The Crash Screen
If you are lucky enough to get a crash screen, most likely you will be able to supply enough information from the error message to the developer for them to figure out what went wrong.
If you receive the following message (or a translation there of), you must take an additional step in order to get the real crash message:
This error message means you don't have the right software installed to display the crash message. To fix this, load the appropriate cab onto your device. Message cabs are located at: C:\Program Files (x86)\Microsoft.NET\SDK\CompactFramework\v3.5\WindowsCE\Diagnostics
If that directory does not exist, install the Power Toys for .NET Compact Framework 3.5 and check again.
Anatomy of the Crash screen.
Here is a crash details report from my XDAFacebook application. I will describe each part and what they mean, what's important and what the programmer is looking for:
The first line: XDAFacebook.exe is the application that crashed. This is important, because if the application relies on multiple .exe files, the one that crashed needs to be known. Always include which .exe crashes.
The next line is the type of exception that was unhandled: ObjectDisposedException. This is also very important. Always include this in your report. If anything, this is the most important piece of information.
The next part is called the Stack Trace. In 2 sentences, the stack trace is an ordered list of methods (or functions) that pin-point where the in the code the crash happened. Think of it as turn-by-turn directions to the line that caused the crash.
The stack trace lines have 2 parts, the NameSpace and the Method. The NameSpace is the series of "Strange word followed by a Period". The method is the "Strange word followed by parenthesis that might have strange words inside". In XFControls.XFPanels.XFPanelList.PanelPaint(Graphics g), the NameSpace is XFControls.XFPanels.XFPanelList and the Method is PanelPaint(Graphics g)
Some of the lines aren't important. Most of the lines are. The most important lines are usually at the top and will have NameSpaces that looks like the application that crashed (or don't look like they come from Microsoft). The best thing to do is include all of the stack trace, but it is also good to know what exactly you are telling the programmer.
Here is basically how I would read the stack trace above:
If any of those lines weren't in the report, I wouldn't be able to get the full picture. So, unless you have a good understanding of the application's source, include the full report!
Developer Defined Messages
Developers try to handle as many errors as possible. In the case that you get a Message from the application signifying some kind of error has occurred, supply the message the occurred, the steps you took to get the error and your "What I expected and what I got". If the message isn't enough information, the developer needs to change his message.
Also, check for any debugging options which the developer may have included, such as a log file. These either tend to be automatically generated files in the install directory, or they exist as a viewable panel within the application. Check with the documentation or ask the developer where these log files exist [1]
Rule number #4, be smart.
I should put this higher in the list, because it really should be the first thing you need to do.
First thing you need to do is make sure you have the latest version of the software installed. Also do a quick search of the thread to make sure that the bug hasn't already been reported. If it has been reported, I personally don't mind a quick "I have the same problem, here are the steps I had to reproduce..." with a link or quote of the original error.
This guide will be updated as I come up with new observations and as people respond with questions/suggestions. Please tell me what you think of the guide and always try and be constructive when you are beta testing software!
Thank you
**updates**
[1] 12/7/2010 (Thanks meltwater!)
Last edited:
