IOW, if you can discover how to get the optimizer to help you (which may take some trickery), you can turn an O(n2) or worse query into something quite reasonable. I don’t know about MySQL – I haven’t needed to make similar queries since I’ve been using it.

Also, besides the memory issue, another potential performance problem that can be caused by clustered indices involves page splits. I’ll try to explain as briefly as possible:

The typical clustered index on a table is on the Primary Key, which is usually an INT IDENTITY column. This key is monotonically increasing, so any new rows inserted will come after all existing rows. This means that each data page will be filled before creating a new one, so the minimum number of new pages is created and the minimum number of IOs is performed.

If you put a clustered index on something else, the physical data needs to be kept in that order. So say you’ve put a clustered index on UserId.

UserId Data
1 a
1 b
1 c
2 a
2 d

If you insert (1,d), that row has to go in between (1,c) and (2,a). If the data page is full (which is optimal for minimizing IOs and space utilization), then you have to split it, move half the data to the new page, and then insert the row.

Now imagine how often this is going to happen if you’re regularly inserting rows in the middle of your clustered index. You’re either going to have significantly fragmented indices, which will be slow, or you’re going to get lots of page splits, which will slow down inserts. There ARE situations in which a clustered index on this sort of data is warranted, such as when inserts are minimal, but often the best solution for an OLTP database with a balanced workload is a clustered index on the PK and non-clustered indices on the other columns you’re interested in.

Of course, as with everything DB-related, you’ll need to apply these concepts to your specific implementation, but they should be considered.

Also, people have differing opinions about when it’s ok to use a database. I’m assuming that the reader has an interest in tuning performance, and is probably dealing with large(r) amounts of data, in which case you probably aren’t going to get better performance with the same flexibility out of some non-database-backed solution. I’d be interested to hear examples of, say, HDF5 or XML or some other file-based mechanism outperforming a database and still being able to do complex queries. I tend to find more situations where people *should* use a database and don’t than the reverse.