Optimization is the process of identifying and eliminating bottlenecks in your TurboGears application and the infrastructure around it to increase performance. You should start optimizing only when your application works, not earlier, and when performance problems arise, not without need. A well-known statement is that “we should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil”.
Process of optimization:
Discovery: Generally, Pythonistas say, Python is “fast enough”. However as a developers we tend to get carried away. Web applications need to be fast! Users will generally let you know if there is a problem or it will arise in testing.
Isolation: Finding the actual location of the performance issue can sometimes be very difficult. You are dealing with so many variables, networks, disk speeds, algorithms, oh my!
Analysis: There are many possible solutions depending on the problem. We will mostly talk about caching because it is an easy solution that can be a applied at many levels of a web applications.
Implementation: This could be anything, from changing a configuration setting, to writing new algorithms and data types to implement your solution.
Testing: There are many an optimizations that have been done that never actually fix the problem. From experience it is important to replicate your user experience though testing and to be able to measure optimization.
Rinse and repeat: Optimization is a constant process of refactoring problem areas.
To find specific bottle neck locations in TurboGears application you can use the Python profiler. You need to change the last part of the start() function in the <yourpackage>/commands.py module to the following:
from yourpackage.controllers import Root import profile profile.run("turbogears.start_server(Root())", 'myapp.prof') # turbogears.start_server(Root())
Start your application to replicate your user traffic. Then stop the application and open a Python shell:
>>> import pstats >>> p = pstats.Stats('myapp.prof') >>> # Cumulative time sort top 25 >>> p.sort_stats('cumulative').print_stats(25) >>> # Also may find time top 25 useful >>> p.sort_stats('time').print_stats(25)
Once you have an idea where the bottlenecks are, based on your profiler, you can now write specific performance tests to find out the latency of the bottlenecks. This is a critical part of the process because if you do not know the latency you will not be able to find out if your solution works.
Alternatively, you can use CherryPy’s built-in profiler which generates profiling data per request which may be helpful, too. For further information, please read CherryPy changelog for 2.1 (works with 2.3, too). Please note that you have to adapt the config snippet to match ConfigObject’s expectations:
[global] profiling.on = True profiling.path = "/path/to/profile/dir"
Every database has its own optimization recommendations, however, here are general considerations:
SQLObject: You can turn SQLObject’s debug mode on buy adding an additional parameter to the connection string. Example:
EXPLAIN: Execute an EXPLAIN query to see the execution plan for your query, then redefine the indices in your database or refactor your queries to optimize.
Here is the link to an article with benchmarks for Kid, Cheetah, Genshi and Jinja:
AJAX itself is a great way to improve the performance of your application, since it can avoid reloading complete pages and instead only reload the relevant parts that have changed, or only the necessary data to rebuild these parts. However, unoptimized Ajax can also cause performance problems and make your application appear as slow to the users.
You will find many articles, books and tools covering this issue. The following article will give you some clues:
TurboGears though CherryPy has the ability to compress its output. This can greatly increase the user experience and decrease bandwidth usage. There is more documentation here:
However, we do not recommend this. If one of the goals is to increase performance of the web application from the users perspective. By compressing all the text (mainly html) from you application then the user will not need to wait as long for pages to load. However, in a high traffic site the compression should be offloaded to a webproxy. Web proxies are faster at compressing and will be serving static content anyway to reduce the load on TurboGears.
A cache is a way to cut down data retrieval time. Generally caches are in memory, which saves the computer from having to retrieve them from the primary datastore, which usually resides on a hard disk and is much slower. The datastore could be a database, on the file system or accessed over the network. Caching may have drawbacks. First, you are trading off memory usage against I/O usage. Lots of caching will need lots of memory. Also, suppose you have a wiki site and cache the wiki pages, if you don’t do caching right, a page could get updated and users do not see the update.
Turbogears Caching Layers:
External Proxy: Most heavy traffic sites use an external proxy already because they are much faster and more efficient with computer resources when serving static content than TurboGears. They can also cache web content generated by a TurboGears web application. This is the easiest to setup and probably provides the most performance boost. However, it is hard to inform the external proxy dynamically about when the application’s data has changed.
CherryPy filter: This works very much like an External Proxy from the perspective of a TurboGears developer. Check the CherryPy cache filter documentation for more details on how it works.
Template Cache: TurboKid and TurboCheetah both cache compiled templates automatically.
Datastore Cache: To find out more about the SQLObject cache system, check out the SQLObject docs.
Other Caching Resources:
There some great tools available that will help you optimize your website. Read the following article to get introduced to two priminent tools of this kind: