ephemeral parent znodes

Details

Description

ephemeral znodes have the nice property of automatically cleaning up after themselves when the creator goes away, but since they can't have children it is hard to build subtrees that will cleanup after the clients that are using them are gone.

rather than changing the semantics of ephemeral nodes, i propose ephemeral parents: znodes that disappear when they have no more children. this cleanup would happen automatically when the last child is removed. an ephemeral parent is not tied to any particular session, so even if the creator goes away, the ephemeral parent will remain as long as there are children.

the when an ephemeral parent is created it will have an initial child, so that it doesn't get immediately removed. i think this child should be an ephemeral znode with a predefined name, "firstChild".

I implemented this feature with a small modification. The ephemeral parent is created with no children and it is deleted when its last child is deleted, so there is no need for a "firstChild".

org.apache.zookeeper.test.InvalidSnapshotTest is failing due to the change in the persisted classes. I guess this change is not as straightforward as I made it, but please comment on how to do it properly.

Daniel Gómez Ferro
added a comment - 14/Jun/11 12:33 I implemented this feature with a small modification. The ephemeral parent is created with no children and it is deleted when its last child is deleted, so there is no need for a "firstChild".
org.apache.zookeeper.test.InvalidSnapshotTest is failing due to the change in the persisted classes. I guess this change is not as straightforward as I made it, but please comment on how to do it properly.

Benjamin Reed
added a comment - 14/Jun/11 18:54 if an ephemeral parent doesn't have a child, why wouldn't it be deleted? i ask this because we lose the chance to do sanity checks if there can be an ephemeral parent with no children.

Couldn't that be checked with the cversion of the parent? I just find it a bit ugly/cumbersome having that firstChild automatically created (that you are probably going to delete right after creating any other child) but that's mostly an aesthetic reason, so if you think its more robust having the firstChild I'll provide a new version of the patch.

Daniel Gómez Ferro
added a comment - 14/Jun/11 20:02 Couldn't that be checked with the cversion of the parent? I just find it a bit ugly/cumbersome having that firstChild automatically created (that you are probably going to delete right after creating any other child) but that's mostly an aesthetic reason, so if you think its more robust having the firstChild I'll provide a new version of the patch.

good point, you can check the cversion of the parent. that was my big objection.

to be honest i can go either way. it is cumbersome to have to do the
firstChild, but i'm wondering if it is easier to explain and manage
that in the code than saying that the znode will go away if there
aren't any children left unless no child have been created. i don't
have a strong feeling one way or the other, but i do lean towards the
firstChild option.

Flavio Junqueira
added a comment - 14/Jun/11 21:52 (Ben sent by e-mail)
good point, you can check the cversion of the parent. that was my big objection.
to be honest i can go either way. it is cumbersome to have to do the
firstChild, but i'm wondering if it is easier to explain and manage
that in the code than saying that the znode will go away if there
aren't any children left unless no child have been created. i don't
have a strong feeling one way or the other, but i do lean towards the
firstChild option.
are there any others that have an opinion?

I'd prefer not have a first child, and it does sound like a good idea to use the cversion. Just to make sure we have the same understanding, the idea is that an ephemeral parent is removed once it has no more children and it has had at least one child?

Flavio Junqueira
added a comment - 14/Jun/11 22:00 I'd prefer not have a first child, and it does sound like a good idea to use the cversion. Just to make sure we have the same understanding, the idea is that an ephemeral parent is removed once it has no more children and it has had at least one child?

I agree with Flavio (no firstchild, create empty, delete when last child is deleted, using cversion we can check which of these is the case). However I have a problem calling this an "ephemeral" parent. ephemeral implies that the znode is tied to the session lifetime, which is not the case here. It's going to be confusing for users.

Can't we call this something like a "transitory" or "shortlived" parent? "container only" might work better (the others might imply non-persistent?)... I'm open to ideas.

also: this patch needs to be updated to include documentation (src/docs/... programming guide).

Patrick Hunt
added a comment - 14/Jun/11 22:27 I agree with Flavio (no firstchild, create empty, delete when last child is deleted, using cversion we can check which of these is the case). However I have a problem calling this an "ephemeral" parent. ephemeral implies that the znode is tied to the session lifetime, which is not the case here. It's going to be confusing for users.
Can't we call this something like a "transitory" or "shortlived" parent? "container only" might work better (the others might imply non-persistent?)... I'm open to ideas.
also: this patch needs to be updated to include documentation (src/docs/... programming guide).

On the one hand I really would like this feature, but on the other hand I do not like the idea of having one of these created with no children and then floating out there for some indefinite period of time until someone finally decides to create children under it. It seems confusing and hard to manage from a client perspective.

All of my use cases would be completely satisfied with the nodes as "real" ephemeral, aka, session-based and only allowing children that are ephemeral containers/nodes from the same session. I'm curious to think of a really compelling use case where I would want this to cross sessions, and the email thread did not seem to provide one. Why don't we want this to be true ephemeral?

Camille Fournier
added a comment - 15/Jun/11 16:39 On the one hand I really would like this feature, but on the other hand I do not like the idea of having one of these created with no children and then floating out there for some indefinite period of time until someone finally decides to create children under it. It seems confusing and hard to manage from a client perspective.
All of my use cases would be completely satisfied with the nodes as "real" ephemeral, aka, session-based and only allowing children that are ephemeral containers/nodes from the same session. I'm curious to think of a really compelling use case where I would want this to cross sessions, and the email thread did not seem to provide one. Why don't we want this to be true ephemeral?

yeah i don't like the floating out there indefinitely with no children part either.

one use case for allowing different session is the barrier like case in which you want to find out when everyone is done using a resource: you create a parent znode, /myresource, with a child called available. processes that use the resource will create children under /myresource. when the resource manager wants to stop providing the resource, it removes /myresource/available and then watches for /myresource to disappear.

Benjamin Reed
added a comment - 15/Jun/11 17:02 yeah i don't like the floating out there indefinitely with no children part either.
one use case for allowing different session is the barrier like case in which you want to find out when everyone is done using a resource: you create a parent znode, /myresource, with a child called available. processes that use the resource will create children under /myresource. when the resource manager wants to stop providing the resource, it removes /myresource/available and then watches for /myresource to disappear.

Flavio Junqueira
added a comment - 15/Jun/11 17:16 how is what you describe different from creating a regular node and forgetting about it there? Being able to create an ephemeral parent without a first child just gives more flexibility to the app.

I see what you mean now with a node floating out there. In that case Ben describes, we can have two conditions for the node to disappear: 1) cversion = 0 and session expired; 2) cversion != 0 and there is no more children.

Flavio Junqueira
added a comment - 15/Jun/11 18:06 I see what you mean now with a node floating out there. In that case Ben describes, we can have two conditions for the node to disappear: 1) cversion = 0 and session expired; 2) cversion != 0 and there is no more children.

Yes, iirc that's one of the issues we had touched on way back when... and one of the reasons why we kept it simply that ephemeral nodes couldn't have children. I seem to also remember another related issue, that once you start allowing arbitrarily large ephemeral trees to be built there was a concern about cleanup and it's effect on availability of the system as a whole. (still a concern I would have)

note: If this znode really is ephemeral (strongly tied to the session lifetime) I don't have a problem calling it as such. however it the znode can live beyond the session lifetime that created it then Flavio's suggestion of "solitary" sounds good to me. (was that a typo or did you really mean solidary?)

Patrick Hunt
added a comment - 15/Jun/11 19:09 Wow, my brain is in an infinite loop now.
Yes, iirc that's one of the issues we had touched on way back when... and one of the reasons why we kept it simply that ephemeral nodes couldn't have children. I seem to also remember another related issue, that once you start allowing arbitrarily large ephemeral trees to be built there was a concern about cleanup and it's effect on availability of the system as a whole. (still a concern I would have)
note: If this znode really is ephemeral (strongly tied to the session lifetime) I don't have a problem calling it as such. however it the znode can live beyond the session lifetime that created it then Flavio's suggestion of "solitary" sounds good to me. (was that a typo or did you really mean solidary?)

I seem to also remember another related issue, that once you start allowing arbitrarily large ephemeral trees to be built there was a concern about cleanup and it's effect on availability of the system as a whole. (still a concern I would have)

How is this different from having lots of ephemeral nodes that need to be cleaned up on session expiration?

I'm attaching a new version of the patch with Flavio's proposal. The ephemeral container (name change pending) will be deleted after session expiration if it doesn't have any child, or when its last child gets deleted. I've added another test for checking this behavior and some user documentation. If it is not enough let me know.

Daniel Gómez Ferro
added a comment - 17/Jun/11 09:47 I seem to also remember another related issue, that once you start allowing arbitrarily large ephemeral trees to be built there was a concern about cleanup and it's effect on availability of the system as a whole. (still a concern I would have)
How is this different from having lots of ephemeral nodes that need to be cleaned up on session expiration?
I'm attaching a new version of the patch with Flavio's proposal. The ephemeral container (name change pending) will be deleted after session expiration if it doesn't have any child, or when its last child gets deleted. I've added another test for checking this behavior and some user documentation. If it is not enough let me know.

Benjamin Reed
added a comment - 17/Jun/11 17:11 i'm afraid the semantics have gotten very complicated just to avoid creating a first child. there are now a couple of reasons for the existence or deletion of a parent:
if a session is alive and never had children -> exists
if a session is alive and doesn't have children, but did at one time -> deleted
if a session is alive and has children -> exists
if a session is dead and has children -> exists
if a session is dead and never had children -> deleted
... there are more of course ...
i'm just wondering if we are going to get a bunch of false bug reports with these semantics, and given the number of corner cases it will be hard to verify the bugs.
it is nice to have simple semantics:
if has children -> exists
if it doesn't have children -> deleted

Here is another option, which might not integrate as seamlessly. It creates a "firstChild" but the client gets to specify its name/type. It could be done with the current API: create( ".../containerName/firstChildName", rest, of, parameters), or with a new call createContainer( ".../containerName", "firstChildName", ...).

Daniel Gómez Ferro
added a comment - 17/Jun/11 20:16 Maybe it is a good idea to make a vote?
Here is another option, which might not integrate as seamlessly. It creates a "firstChild" but the client gets to specify its name/type. It could be done with the current API: create( ".../containerName/firstChildName", rest, of, parameters), or with a new call createContainer( ".../containerName", "firstChildName", ...).

How is this different from having lots of ephemeral nodes that need to be cleaned up on session expiration?

That's a good point, it's not. However the likely hood of someone "misusing" is higher imo in the case I highlighted, vs the one you detailed. But I agree, they are the same from a b/e perspective - we should really have some system tests that exercise this and monitor availability (or lack there of).

Maybe it is a good idea to make a vote?

We (zk community) typically try to stay away from voting around code changes, we work very very hard to reach consensus on the technical merits. Some good discussion/ideas still flowing, let's keep whacking on it.

I like your idea about allowing the first child to be named (I was thinking about suggesting similar) given the first thing you do after creating the parent is to create the child. What do other ppl think? createContainer would make things more explicit, increase the likelyhood the user will grok what this is...

Patrick Hunt
added a comment - 17/Jun/11 20:40 How is this different from having lots of ephemeral nodes that need to be cleaned up on session expiration?
That's a good point, it's not. However the likely hood of someone "misusing" is higher imo in the case I highlighted, vs the one you detailed. But I agree, they are the same from a b/e perspective - we should really have some system tests that exercise this and monitor availability (or lack there of).
Maybe it is a good idea to make a vote?
We (zk community) typically try to stay away from voting around code changes, we work very very hard to reach consensus on the technical merits. Some good discussion/ideas still flowing, let's keep whacking on it.
I like your idea about allowing the first child to be named (I was thinking about suggesting similar) given the first thing you do after creating the parent is to create the child. What do other ppl think? createContainer would make things more explicit, increase the likelyhood the user will grok what this is...

I thought about this better and maybe Ben is right, we should create an ephemeral child instead of having to deal with more complex logic. Do you think "firstChild" is a good name? Should we let the user choose it, or maybe use a more abstract one (some symbol, "placeholder"...) ?

Daniel Gómez Ferro
added a comment - 19/Oct/11 17:48 I thought about this better and maybe Ben is right, we should create an ephemeral child instead of having to deal with more complex logic. Do you think "firstChild" is a good name? Should we let the user choose it, or maybe use a more abstract one (some symbol, "placeholder"...) ?

Jakub Lekstan
added a comment - 18/Nov/11 12:05 Sorry but I don't like this solution. IMO ephemeral nodes have to be deleted on session expire, otherwise we lose consistency.
What I suggest is to allow ephemeral znodes to be parents.
Why would you want to have pseudo ephemeral znodes which don't get deleted after session expire?

Jordan Zimmerman
added a comment - 18/Nov/11 17:35 I suggest a compromise solution. Ephemeral parent znodes may still be needed, but my suggestion would solve many of the problems:
public boolean deleteIfNoChildren(String path, ...)
This would be an atomic operation that deletes a node iff it has no children. This pushes a lot of the burden to ZK users but it at least makes garbage collection possible.

Just my two cents, but maybe you can solve this with Multi. You can only create ephemeral parent znodes through the multi interface. That way it gives you time to make the parent node and associate a child to it before it would be garbage collected.

Ryan Shelley
added a comment - 01/Mar/12 23:52 Just my two cents, but maybe you can solve this with Multi. You can only create ephemeral parent znodes through the multi interface. That way it gives you time to make the parent node and associate a child to it before it would be garbage collected.

My 2 cents: Any ephemeral node is deleted if it's session've expired and it has no children.
Pretty simple. No two types of ephemeral nodes. Can have any children (non-ephemeral, ephemeral with same session, ephemeral with another session). Full backward compatibility.
This easily covers trees of ephemeral nodes with same session (simple case) cleaned up on session close and more complex examples like locks removed after both lock creator and all lock users die.

Vitalii Tymchyshyn
added a comment - 02/Mar/12 17:52 My 2 cents: Any ephemeral node is deleted if it's session've expired and it has no children.
Pretty simple. No two types of ephemeral nodes. Can have any children (non-ephemeral, ephemeral with same session, ephemeral with another session). Full backward compatibility.
This easily covers trees of ephemeral nodes with same session (simple case) cleaned up on session close and more complex examples like locks removed after both lock creator and all lock users die.

Both ZOOKEEPER-834 and this has similarities in the way supporting children to a znode which is owned by a session. This is really a nice feature and it would be good if we can reach to an agreement on both the cases.

After reading the comments on this JIRA I got confused a bit, but I'm also thinking similar way where Flavio Junqueira was suggesting. How about seeing a node like:-

Rakesh R
added a comment - 22/Apr/14 15:52 Hi folks,
Both ZOOKEEPER-834 and this has similarities in the way supporting children to a znode which is owned by a session. This is really a nice feature and it would be good if we can reach to an agreement on both the cases.
After reading the comments on this JIRA I got confused a bit, but I'm also thinking similar way where Flavio Junqueira was suggesting. How about seeing a node like:-
Solidary or Federation znode -
only the session owner can create children under the parent znode
allows to create my type znode as child (no persistent or ephemeral child under me)
exists if session is alive
delete if it doesn't have children and session expires
Appreciate any thoughts