Remove non-words from stats

[thom4parisot]

Looks like all the https://t.co/xyz tokens get caught in, as well as non-significant words (eg: ve, si, ch).

Last items have been maybe once or twice so not sure it makes them significant as well.

[DavidBruant]

The stem function I get from Natural.js does part of this work (the words to eliminate are different per language).

But I've noticed indeed that there is still quite a bit of noise. Especially the https. But not always. I follow quite a number of people who talk about web security and HTTPS and it should be counted as a word in these cases.
For now, I've decided that it was better to keep the noise rather than being strict (and potentially lose proper words). I want to reduce the noise, but it's a subtle exercise.

Last items have been maybe once or twice so not sure it makes them significant as well.

twice
You can click on the words and see the corresponding tweets ;-)

[gmarty]

How about using twttr.txt.extractUrlsWithIndices() from https://github.com/twitter/twitter-text/ and use the indices (the starting and ending positions of the urls in a tweet) to remove the urls (i.e. replace them by white spaces), then stem the processed string?

[thom4parisot]

@DavidBruant I agree if someone talks about HTTPS, you should know about it in the breakdown. But not if the token comes from a URL (https and co are predominant because of t.co URL shortener for example). I agree with @gmarty URLs (and URL parts) should be stripped out.

That's the main point I was trying to raise – the other one about displaying words mentioned twice only is a sidetrack you are right to keep open (it's not that noisy actually).