My 2 cents: Since you specified a full partition key, it makes no sense to query all the nodes. It should clearly only query a node responsible for (35, 78005). Due to the way Cassandra is designed, I'm expecting it to consider reducing the number of involved nodes as a priority. Given that, the only involved node should probably look at how many records it has for (35, 78005) and how many it has in the index for record_link_id=9897, and use the fastest one to serve the query (which isn't necessarily the smallest one, depending on whether the index is also sorted by primary key).
– Giulio FrancoApr 17 '15 at 8:06

2 Answers
2

select * from update_audit where scopeid=35 and formid=78005 and record_link_id=9897;

How the above query will work internally in cassandra?

Essentially, all data for partition scopeid=35 and formid=78005 will be returned, and then filtered by the record_link_id index. It will look for the record_link_id entry for 9897, and attempt to match-up entries that match the rows returned where scopeid=35 and formid=78005. The intersection of the rows for the partition keys and the index keys will be returned.

How high-cardinality column (record_link_id)index will affect the query performance for the above query?

High-cardinality indexes essentially create a row for (almost) each entry in the main table. Performance is affected, because Cassandra is designed to perform sequential reads for query results. An index query essentially forces Cassandra to perform random reads. As cardinality of your indexed value increases, so does the time it takes to find the queried value.

Does cassandra will touch all nodes for the above query? WHY?

No. It should only touch a node that is responsible for the scopeid=35 and formid=78005 partition. Indexes likewise are stored locally, only contain entries that are valid for the local node.

creating index over high-cardinality columns will be the fastest and best data model

The problem here is that approach does not scale, and will be slow if update_audit is a large dataset. MVP Richard Low has a great article on secondary indexes(The Sweet Spot For Cassandra Secondary Indexing), and particularly on this point:

If your table was significantly larger than memory, a query would be very slow even to return just a few thousand results. Returning potentially millions of users would be disastrous even though it would appear to be an efficient query.

...

In practice, this means indexing is most useful for returning tens, maybe hundreds of results. Bear this in mind when you next consider using a secondary index.

Now, your approach of first restricting by a specific partition will help (as your partition should certainly fit into memory). But I feel the better-performing choice here would be to make record_link_id a clustering key, instead of relying on a secondary index.

Edit

How does having index on low cardinality index when there are millions of users scale even when we provide the primary key

It will depend on how wide your rows are. The tricky thing about extremely low cardinality indexes, is that the % of rows returned is usually greater. For instance, consider a wide-row users table. You restrict by the partition key in your query, but there are still 10,000 rows returned. If your index is on something like gender, your query will have to filter-out about half of those rows, which won't perform well.

Secondary indexes tend to work best on (for lack of a better description) "middle of the road" cardinality. Using the above example of a wide-row users table, an index on country or state should perform much better than an index on gender (assuming that most of those users don't all live in the same country or state).

Edit 20180913

For your answer to 1st question "How the above query will work internally in cassandra?", do you know what's the behavior when query with pagination?

Basically, paging will cause the query to break itself up and return to the cluster for the next iteration of results. It'd be less likely to timeout, but performance will trend downward, proportional to the size of the total result set and the number of nodes in the cluster.

TL;DR; The more requested results spread over more nodes, the longer it will take.

Thanks for the insight! How does having index on low cardinality index when there are millions of users scale even when we provide the partition key in the query like select * from users partitionkey=x and gender='M'. From the storage perspective, the hidden column family on gender, will it not overflow? will it cause problem since it needs to scan through hidden column family to filter out the result? stackoverflow.com/questions/29659564/…
– pinkpantherApr 17 '15 at 15:37

@grisaitis it depends on how you're trying to use it. By definition, the indexes are paired with the SSTables, which means that it will still exhibit poor performance when not paired with a partition key. For details, I'd suggest reading this article by Doan Duyhai, which examines the inner-workings of SASI indexes: doanduyhai.com/blog/?p=2058
– AaronSep 26 '17 at 18:12