Performance Tool Usability Survey Questions

1. Which performance tools do you use? Why?

2. How long have you used these tools? Why did you stop using a particular tool?

3. In your opinion, what are the names of some of the more popular performance tools?

4. Why do you think they are popular?

5. Which tools are the hardest to use? Why?

6. What specific aspects of these tools do you find particularly frustrating?

7. What suggestions can you provide that would make these tools more usable?

8. How often do you use performance tools?

9. Profiling versus tracing

Profiling refers to the process of collecting statistical event data at runtime and performing data aggregation and filtration when the program terminates. In tracing, however, events are often time-stamped or at least ordered with respect to program execution. This enables the user to see exactly when and what occurred. Tracing is able to provide a more accurate view of the program behavior. However, with a long running program, tracing requires an enormous amount of storage space and because of this it can also complicate the analysis process.

Is the information gained from profiling or tracing more helpful to you? Why?

10. Do you prefer a GUI or a command line interface?

11. How much does a tool's interoperability with other tools affect whether or not you use it?

12. How useful do you find the low-level information provided by hardware counters (such as instruction count, cache miss rates, and hardware interrupts count)?

13. How important is it for a tool to support a wide variety of platforms?

14. Rate the importance of these features from 1-5:

Item Importance rating
The ability to provide a best case performance time based on the current system
The cost of the tool
The ease of installation
Support for a wide variety of platforms
Support for a heterogeneous environment
How easy it is to learn to use the tool
The ability to provide hardware counter information
The ability to provide automatic instrumentation (i.e. the tool automatically inserts function calls for the user)
The ability to provide profiling support
The ability to provide tracing support
Multiple analyses/views (i.e. multiple ways to analyze/view the data)
The ability to interoperate with other tools
The ability to filter and aggregate (i.e. summarize) data
The ability to correlate performance data back to the source code
The ability to identify performance bottlenecks
Documentation quality
Technical support
Stability (i.e. how often the tool crashes)
The ability to search through the data

15. Which is more important to you, measurement accuracy or quick response time?

SHMEM/UPC users

Please fill this information out if you currently use SHMEM or UPC.

16. What are some of the most common performance problems you encounter using UPC/SHMEM? (For example, too much implicit communication or unnecessary synchronization)

17. Are you interested in seeing all of the implicit communication for a node or just its total communication time?

18. What would be a typical execution time for your applications?

19. Which machine(s), OS(s), and compiler(s) are you currently using?

20. What type of information do you want from a performance tool?

21. How long have you been programming in parallel languages?