Server code

Let's publish two "views" of the same server collection. One will send down a cursor with 50 records, and the other will send down a cursor with 100 records. There are over 458 million records in this fictitious server-side database, and the client does not need to know about all of those (in fact, sending them all down would probably take several hours in this example):

Client code

In order to support the latency compensation technique, we are forced to declare a single collection Items on the client. It should become apparent where the flaw is: how does one differentiate between Items for enabled_items and Items for processed_items?

var Items = new Meteor.Collection("items");
Meteor.subscribe("enabled_items", function () {
// This will output 50, fine
console.log(Items.find().count());
});
Meteor.subscribe("processed_items", function () {
// This will also output 50, since we have no choice but to use
// the same "Items" collection.
console.log(Items.find().count());
});

My current solution involves monkey-patching _publishCursor to allow the subscription name to be used instead of the collection name. But that won't do any latency compensation. Every write has to round-trip to the server:

With the monkey-patch in place, this will work. But go into Offline mode and changes won't appear on the client right away -- we'll need to be connected to the server to see changes.

What's the correct approach?

EDIT: I just revisited this thread and I realize that, as it stands, my question and answers and plethora of comments carry a lot of misinformation.

What it comes down to is that I misunderstood the publish-subscribe relationship. I thought that when you published a cursor, it would land on the client as a separate collection from other published cursors that originated from the same server collection. This is simply not how it works. The idea is that both the client and server have the same collections, but it's what is in the collections that differs. The pub-sub contracts negotiate which documents end up on the client. Tom's answer is technically correct, but was missing a few details to turn my assumptions around. I answered a similar question to mine in another SO thread based on Tom's explanation, but keeping in mind my original misunderstanding of Meteor's pub-sub: Meteor publish/subscribe strategies for unique client-side collections

Hope this helps those who run across this thread and come away more confused than anything!

NOTE

The example above is simplified. Assume the two subsets don't necessarily hold the same properties. It may be hard to visualize, but I promise you this is crucial when dealing with large amounts of data -- you do not want to send any more than what the client needs to see. I will try to think of a better example that illustrates the use cases
–
matb33Sep 28 '12 at 16:49

1

In the app I'm building with meteor I have two types of messages, basically doing the same thing (have the same "schema") but are published differently. For now I use two collections, as I didn't want to "patch" Meteor...
–
AndreasSep 28 '12 at 19:39

I'm not proposing you send all the items across; just that you use the same query on the client side (that you already are server side) to pull them back out again when you need to use them.
–
Tom ColemanSep 29 '12 at 3:04

I still plan on writing up a different scenario, but maybe a quick one here might help: Imagine the first subset consists of records 50000 to 50010 (like in pagination), and your second subset is all "processed" records across all the millions of records. Two subsets, one collection. And you need to display both of these at the same time. This type of requirement becomes more and more prevalent as an app increases beyond weekend hacks
–
matb33Sep 30 '12 at 17:27

1

OK well that completely turns my understanding of how Meteor's pubsub works upside down =) I did try it and it's working, and it appears that my Items collection contains exactly the right amount of records between the two subscriptions, no more no less. I'm going to dig a bit deeper to understand how that works. Thanks Tom
–
matb33Oct 4 '12 at 3:08

I've managed to achieve some promising preliminary results by approaching the problem with a single publish/subscribe per collection, and leveraging $or in the find query.

The idea is to provide a wrapper around Meteor.Collection that allows you to add "views", which are basically named cursors. But what's really happening is that these cursors aren't run individually... their selectors are extracted, $or'd together and run as a single query and onto a single pub-sub.

It's not perfect, in that an offset/limit won't work with this technique, but at the moment minimongo doesn't support it anyway.

But ultimately what it allows you to do is to declare what looks like different subsets of the same collection, but under the hood they are the same subset. There's just a bit of abstraction in front to make them feel cleanly separated.

In short, I take advantage of having the same code running in both server and client, and if the server isn't doing something, the client will, or vice versa.

And most importantly, only the records you are interested in are getting sent down to the client. This is all achievable without an abstraction layer by simply doing the $or yourself, but that $or will get pretty ugly as more subsets get added. This just helps manage it with minimal code.

I wrote this quickly to test it out, apologies for the length and lack of documentation:

Mat - first off, do you realise that if you publish the same collection twice (with different subscription names + filters), you end up with the "$or-ed" data in the collection on the client?
–
Tom ColemanOct 3 '12 at 3:35

Secondly, your Collection.view(..) method is cool (kind of like an ActiveRecord scope), but isn't it fundamentally just doing what I suggested in my answer (using the same query on the client + server)?
–
Tom ColemanOct 3 '12 at 3:37

Tom, yes you're right, fundamentally it is. And it isn't capable of solving the 50000 to 50010 example either. But the advantage is that by merging the results of multiple queries into a single cursor, we can have all the records involved in the various pub-subs available to the client, and no more. We then have the client refilter again (by running the same find) to re-narrow down the query. Boy that sounds confusing, but it makes sense to me! ;-)
–
matb33Oct 3 '12 at 22:49

Tom, I'm also unclear as to the first comment you made about publishing the same collection twice. I'm not quite sure what you mean. I'm guessing you've uncovered a major flaw in my experiment =)
–
matb33Oct 3 '12 at 22:50

Close, but since there's no publish on the client, the two client collections can't see each other. The latency compensation technique won't work either, which is the crux of all this
–
matb33Sep 30 '12 at 20:55

for security reasons, i don't permit any writes to any Collection at the client-side (else anyone can just open Dev Tools and modify the collection e.g. Items.insert({some:data})).. instead i do my writes server-side using Meteor.Methods and implement method stubs client-side to get Latency Compensation.
–
LloydSep 30 '12 at 22:32

Lloyd, that's exactly what I've had to resort to. We'll have to change that when the auth branch lands into master. But this is slightly different. Your stub still does a local insert so that changes occur immediately, and the server gives the final yay or nay. Try running your app in Firefox, then under the file menu, click Work Offline. Try adding a few records. Do they appear despite no connection to the server? This is a good litmus test to find out if you have latency compensation working correctly.
–
matb33Oct 1 '12 at 0:08

Latency Compensation works fine with expected behaviour - i just tested my app in Firefox as suggested.
–
LloydOct 1 '12 at 9:46

i tested both updating and inserting records in the Method stub. Changes are reflected immediately on the screen, then corrected once the server response arrives.
–
LloydOct 1 '12 at 9:55