Should I worry so much about computer speed when designing my algorithm?

Explanation

If it is just a scientific exercise, then no.  Computers are pretty fast nowadays.  So, you do not need worry about performance.  However, there are problems that are complex in nature or needs to process a large volume of data, or deal with real-time responses.  In such cases, you need to make your design in light of the right computer capabilities including memory and processor