New blog post: "Descriptive vs. inferential community detection"
A mini-thread for those too lazy to click the link! 1/7
(Based on recent pre-print: https://arxiv.org/abs/2112.00183)
There are essentially two objectives when doing community detection (or any data analysis): To "describe" or to "infer".
Description involves finding patterns, inference involves finding explanations. 2/7
Most community detection methods out there (modularity, infomap, walktrap, etc.) are descriptive. The communities they find are there, but they cannot explain. In a very concrete sense, they *overfit* your data, confusing actual structure with random fluctuations. 3/7
Inferential approaches do not do this. They find the most parsimonious explanation for the data — according to Occam's razor — and do not confuse randomness with structure. 4/7
So, when should we "infer" or "describe"? Here's a good litmus test:
Q: "Would the usefulness of our conclusions change if we learn, after obtaining the communities, that the network being analyzed is completely random?"
5/7
If the answer is "yes", then an inferential approach is needed.
If the answer is "no", then an inferential approach is not required. 6/7
It's arguable that in most scientific contexts the answer would be "yes". I this case we need inferential approaches, and descriptive ones are just not up to the task!
There is lots more to say about this. Read it here: https://arxiv.org/abs/2112.00183 7/7
