How does caching work in Magnolia?

It simply caches what is returned to a request if caching is turned on.

Where is the cached content stored?

Although Magnolia has a pluggable cache mechanism, the default implementation stores cached content on the file system. The magnolia.cache.startdir property (in magnolia.properties) determines where the cached content is stored. Its default value is cache, and it can either be a full/absolute path, or a path relative to the root of the webapp.

Can we have real dynamic content in our pages (stuff like new Date())?

Simply turn off caching for that page. See Config: /server/cache/level1/URI/deny in adminCentral.

When will the cache expire?

Whenever you publish a page, its cached version is dumped

How do I disable Cache completely

Set the property active = false under Level1 cache configuration

When I synchronize templates from author to public, should I delete the cache/ directory after stopping public tomcat?

A change in the template itself (jsp-file) will not be noticed by the cache mechanism. Therefore you can either delete the cache, or activate a page, in which case the cache is flushed.

Is there anything else I should know about caching that would better enable me to troubleshoot any issues?

To check that the problem is really a server side caching issue, try sending request parameters
with the url, e.g. /en/site.html?test=1. Any URL with request parameters bypasses the Magnolia cache.

Additionally, the issue tracker lists issues regarding cache.