Right now we do drupal_flush_all_caches() at the end of the import process. This guarantees that things like menu system additions get picked up (we ran into this problem while testing importing views.) However I am worried this is a little too big of a hammer for all situations. For instance, this would clear the page cache which we would really only need on field system changes. We should look into what caches we realistically should be clearing on every import, and then implement other clears as needed in the module-specific import handling.

If this is the case, I think one way to solve it would be:
1. Create a new function: drupal_flush_caches($bins), that basically does exactly what the drupal_flush_all_caches() does at the moment, but flushes the caches only from the specified bins.
2. Rewrite the drupal_flush_all_caches() to be only like:

drupal_flush_caches(module_invoke_all('cache_flush'));

3. Call drupal_flush_caches() instead of drupal_flush_all_caches() in config_admin_import_form_submit(), with only a subset of bins (still have to define from where do we get them... are they hardcoded, are they got from a hook, etc).

If still to much data from the cache is flushed, then we probably have to create some specific function to clear the cache on importing the configuration, but I think that the other things that drupal_flush_all_caches() is flushing are not so heavy.

For the moment, we actually moved the drupal_flush_all_caches() call so that it only happens when you import via the UI. This is a common pattern in other areas of Drupal, and I think it will solve most of our problems. We can reopen this later if we need to.