Re: Cleanup resources on pipeline cancelation

Hi Romain,

Andrew's example actually wouldn't work for that. With Google Cloud Pub/Sub (the example source he referenced), if there is no subscription to a topic, all publishes to that topic are dropped on the floor; if you don't want to lose data, your are expected to keep the subscription around continuously. In this example, leaking a subscription is probably preferable to losing date (especially since Pub/Sub itself garbage collects subscriptions that have been inactive for a long time).

The answer might be that Beam does not have a good lifecycle story here, and something needs to be built.

Some of our IOs create external resources that need to be cleaned up when a pipeline is terminated. It looks like the org.apache.beam.sdk.io.UnboundedSource interface is called on creation, but there is no call for cleanup. For example, PubsubIO creates a Pubsub subcription in createReader()/split() and it should be deleted at shutdown. Does anyone have ideas on how I might make this happen?