Use Apache mod_expires and be sure to set how long files should be cached. You could use the example below for your Apache virtualhost config:
# Turn on Expires and set default to 0
ExpiresActive On
ExpiresDefault A0

Use a Content Delivery Network (CDN) for parallel transfer of static content. There is a Magento extension that can help you do this with category and product images: the One Pica Image CDN. But… (see next tip).

Don’t use too many different external sources (for images, iframes, (twitter/facebook)feeds etc.) because every DNS lookup takes extra time and you create an extra dependancy (on some 3rd party server) for your site to load properly.

Enable Apache KeepAlives: Make sure your Apache configuration has KeepAlives enabled. KeepAlives are a trick where multiple HTTP requests can be funneled through a single TCP connection. The setup of each TCP connection incurs additional time, this can significantly reduce the time it takes to download all the files (HTML, JavaScript, images) for a website. More info at Apache.org. Be carefull though, I’ve heard from some that this create (a lot of) extra load on the server and might crash the server on high traffic moments!

Minimize redirects.

Make your output W3C compliant. Errors slow down the browser.

Swap Apache for NginX or Litespeed. If this isn’t an option, then go for Zend Server (ideally the non-CE version so you can use full page caching).

Turn off or at least reduce web server logging (reduces disk writes).

Disable Access Time Logging. Magento Wiki: For Linux servers, if you have access-time logging enabled on any of your mysql, web server or cache partitions, try turning it off for a performance boost. If you’re using ext3 or reiserfs there may be faster journal write methods you can use. For more info see Linux.com.

Compile MySQL from source instead of your OS’s package manager.

Always upgrade to the latest Magento version. Not only will you get more features and bug- and security fixes, but with every update Magento performs better.

Query Cach size: Magento Blog: Modify the configuration for your MySQL server to take better advantage of your server’s RAM. Most Linux distributions provide a conservative MySQL package out of the box to ensure it will run on a wide array of hardware configurations. If you have ample RAM (eg, 1gb or more), then you may want to try tweaking the configuration. An example my.cnf is below, though you will want to consult the MySQL documentation for a complete list of configuration directives and recommended settings.

set ‘php_value memory_limit 128M’ in your php configuration or add it to your .htaccess file to ensure you don’t run out of memory.

Use a memory-based filesystem for dynamic data. If you store dynamic data (var/cache, var/session) on RAMdisk or tmpfs, the disk I/O is decreased.

Change realpath_cache_size in php.ini.
realpath_cache_size=1M (careful, this is per apache process) realpath_cache_ttl=86400 (ok for production site)

Optimize all your (template) images- Most if not all should be at least below 10kb.

Crop the white space using your image editor.

Use PNG8 Files or GIF files rather than Jpegs and don’t use transparency (depending on how many colors you use and how large the image is, but try for yourself).

Scale images: make images in the dimensions you need and not resizing them in the editor.

Use image compression (you can use smush.it to do it for you).

Use CSS Sprites, there even are CSS Sprite Generators.

Minify your Css, remove unused code.

Minimize Javascript use.

Use a lightweight template as a basis for your template. For example: the Yoast Blank SEO theme.

Specify Image dimensions.

Use Block cache and HTML output in your extensions.

Apply Javascript Lazy Loader for prototype.

C) Magento configuration

Uninstall any extensions that you don’t actually use.

Disable modules that you don’t use: System -> Configuration -> Advanced -> Advanced.
This is an example setting

Enable all Magento Caches: System -> Cache Management.

Use an offsite Stats Tracker like Google Analytics and not an onsite one. Most of this will use Javascript, host the Javascript yourself.

Combine Javascript and Combine CSS files: System ->Configuration ->Advanced ->Developer -> ‘Javascript settings’ and ‘CSS Settings’. You can also consider using an extensions to do this like the Fooman Speedster extension, whichever works best for you.

Try some of the Magento performance extensions.

Enable the Magento Flat Catalog, only helpful for very large catalogs to feel the improvements.

Use Magento’s Compilation feature. It’s reported to give you a 25%-50% performance boost: System > Tools > Compilation.Edit: Yoav Kutner (Magento CTO) let me know that “in later versions since we optimized the catalog EAV, Magento Compilation is not really needed if you have byte code caching and if it is configured correctly“.

Use the correct session storage, choose file system or database (during setup). Most installations should use “file system” because it’s faster and doesn’t cause the database to grow. But if your site will run on multiple servers, you should select “database” so that a user’s session data is available regardless of which server his/her request is served from. More info about this from Ashley Schroder at Magebase.com.

Limit the number of products on a product overview page.

Set only those attribute frontend properties to ‘Yes’ that you’re actually going to use. Set all other to ‘No’. Don’t use in quick search, advanced search compare, etc etc.: Catalog -> Attributes -> Manage Atributes -> Frontend Properties.

(because it doesn’t actually speed up the frontend but only the backend):

Use K-Meleon if you are on Windows for your general Admin work. It renders Magento’s heavy JS back-end significantly faster than any other browser.

Use the GoogleGears extension from Yireo.com to cache static files locally.

Use a local pc/mac application to manage Magento (like mag-manager.com).

I do realize that you probably can’t use all of the above tips but it’s not about using them all, and sometimes you just make the choice to give a functionality a priority over performance and that’s OK. If you can only apply some of them you will still increase speed and gain in user experience. And remember: because speed depends on many variables, some of the tips might not have an impact on your website at all while others can have a huge impact. So always test the impact of every step you take and take actions accordingly.