Cheap and Secure Web Hosting Provider : See Now

[Solved]: Scientifically acceptable method for comparing performance of algorithms

, , No Comments
Problem Detail: 

I am comparing several different algorithms for navigation applications. Since I may possibly be publishing a paper comparing the performance of these algorithms with each other, I was wondering the best way to go about measuring performance.

As far as I know, there are two ways to compare performance: writing the code in a compiled language like C and measuring the time of execution, or counting the number of FLOPS. However, I am aware that different processors have different capabilities which results in different execution times on different platforms.

These algorithms will be implemented on low-cost embedded systems (the main application is satellites, where low power is key).

So in summary: what is the scientifically accepted way of comparing the performance of different algorithms?

Asked By : dgreenheck

Answered By : D.W.

There is no single answer. The answer depends upon the specific situation you are in. It's not that there is a single scientifically accepted way of evaluating performance. Instead, a paper should be driven by the claims you want to make. First, figure out what claims you want to make about your scheme. Then, figure out what evidence is needed to support those claims.

For instance, perhaps you would like to claim that your scheme is fast enough to deploy on low-cost embedded systems. Well, then your next step is to make that claim more precise: what counts as fast enough? how do you know? what specific systems do you mean? Then, you can evaluate performance on those systems, by that metric, and assess whether it is fast enough.

Or maybe you would like to claim that your scheme is faster than previously published approaches. Your next step would be to make that more precise: which previous approaches are you going to compare to? on what platform? with what workload? And then you'd evaluate on those platforms.

The right metric to use for performance will depend on your specific situation. For some, maybe it is wall-clock time. For others, maybe it is energy consumption (power). For still others, maybe it is guaranteed low-latency, or memory usage, or something else entirely.

In short: there is no single scientifically accepted method. Instead, think of science as (a) being precise about what claims you are making, and then (b) providing appropriate evidence to support those claims. What constitutes "appropriate evidence" will depend upon your specific situation. You can often look to other publications in your field to see what evaluation method they used, as initially guidance, but ultimately this is a matter of critical thinking: evaluating evidence in a logical, careful, thoughtful manner.

Best Answer from StackOverflow

Question Source :

3.2K people like this

 Download Related Notes/Documents


Post a Comment

Let us know your responses and feedback